Skip to main content
SpringerLink
Log in

Deep Reinforcement Learning: An Overview

  • Conference paper
  • First Online:
Proceedings of SAI Intelligent Systems Conference (IntelliSys) 2016 (IntelliSys 2016)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 16))

Included in the following conference series:

Abstract

In recent years, a specific machine learning method called deep learning has gained huge attraction, as it has obtained astonishing results in broad applications such as pattern recognition, speech recognition, computer vision, and natural language processing. Recent research has also been shown that deep learning techniques can be combined with reinforcement learning methods to learn useful representations for the problems with high dimensional raw data input. This article reviews the recent advances in deep reinforcement learning with focus on the most used deep architectures such as autoencoders, convolutional neural networks and recurrent neural networks which have successfully been come together with the reinforcement learning framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

    Article  Google Scholar 

  2. Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32, 1238–1274 (2013)

    Article  Google Scholar 

  3. Vengerov, D.: A reinforcement learning approach to dynamic resource allocation. Sun Microsystems, Inc. (2005)

    Google Scholar 

  4. Barto, A.G., Mahadevan, S.: Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. Syst. 13, 341–379 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  5. Mousavi, S.S., Ghazanfari, B., Mozayani, N., Jahed-Motlagh, M.R.: Automatic abstraction controller in reinforcement learning agent via automata. Appl. Soft Comput. 25, 118–128 (2014)

    Article  Google Scholar 

  6. Sutton, R.S., David, A.M., Satinder, P.S., Mansour, Y.: Policy Gradient Methods for Reinforcement Learning with Function Approximation, pp. 1057–1063 (2000)

    Google Scholar 

  7. Mattner, J., Lange, S., Riedmiller, M.: Learn to swing up and balance a real pole based on raw visual input data. In: Huang, T., Zeng, Z., Li, C., Leung, C.S., (eds.) Neural Information Processing: 19th International Conference, ICONIP 2012, Doha, Qatar, 12–15 November 2012, Proceedings, Part V, pp. 126–133. Springer, Heidelberg (2012)

    Google Scholar 

  8. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., et al.: Playing atari with deep reinforcement learning. In: NIPS Deep Learning Workshop (2013)

    Google Scholar 

  9. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  10. Böhmer, W., Springenberg, J.T., Boedecker, J., Riedmiller, M., Obermayer, K.: Autonomous learning of state representations for control: an emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations. KI - Künstliche Intelligenz 29, 353–362 (2015)

    Article  Google Scholar 

  11. Levine, S., Fin, C., Darre, T., Abbee, P.: End-to-End training of deep visuomotor policies. arXiv:1504.00702 (2015)

  12. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. (JAIR) 4, 237–285 (1996)

    Google Scholar 

  13. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge (1998)

    Google Scholar 

  14. Riedmiller, M.: Neural fitted Q iteration – first experiences with a data efficient neural reinforcement learning method. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L., (eds.) Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, 3–7 October 2005, Proceedings, pp. 317–328. Springer, Heidelberg (2005)

    Google Scholar 

  15. Oh, J., Guo, X., Lee, H., Lewis, R.L., Singh, S.: Action-conditional video prediction using deep networks in Atari games, pp. 2845–2853 (2015)

    Google Scholar 

  16. Bengio, Y.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013)

    Article  Google Scholar 

  17. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2, 1–127 (2009)

    Article  MATH  Google Scholar 

  18. Bengio, Y., Pascal, L., Dan, P., Larochelle, H.: Greedy Layer-Wise Training of Deep Networks, pp. 153–160 (2007)

    Google Scholar 

  19. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. Presented at the Proceedings of the 25th International Conference on Machine learning, Helsinki, Finland (2008)

    Google Scholar 

  20. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  21. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1, 541–551 (1989)

    Article  Google Scholar 

  22. LeCun, Y., Bengio, Y.: Convolutional networks for images, speech, and time series. In: Michael, A.A., (ed.) The Handbook of Brain Theory and Neural Networks, pp. 255–258. MIT Press (1998)

    Google Scholar 

  23. Scherer, D., Müller, A., Behnke, S.: Evaluation of pooling operations in convolutional architectures for object recognition. In: Diamantaras, K., Duch, W., Iliadis, L.S., (eds.) Artificial Neural Networks – ICANN 2010: 20th International Conference, Thessaloniki, Greece, 15–18 September 2010, Proceedings, Part III, pp. 92–101. Springer, Heidelberg (2010)

    Google Scholar 

  24. Deng, J., Dong, W., Socher, R., Li, L.J., Kai, L., Li, F.-F.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009, CVPR 2009, pp. 248–255 (2009)

    Google Scholar 

  25. Deng, L., Hinton, G., Kingsbury, B.: New types of deep neural network learning for speech recognition and related applications: an overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8599–8603 (2013)

    Google Scholar 

  26. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5, 157–166 (1994)

    Article  Google Scholar 

  27. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)

    Article  Google Scholar 

  28. Tesauro, G.: TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput. 6, 215–219 (1994)

    Article  Google Scholar 

  29. Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: IEEE International Conference on Neural Networks, 1993, vol. 1, pp. 586–591 (1993)

    Google Scholar 

  30. Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Int. Res. 47, 253–279 (2013)

    Google Scholar 

  31. Lin, L.-J.: Reinforcement learning for robots using neural networks. Carnegie Mellon University (1993)

    Google Scholar 

  32. Guo, X., Singh, S., Lee, H., Lewis, R.L., Wang, X.: Deep learning for real-time atari game play using offline monte-carlo tree search planning, pp. 3338–3346 (2014)

    Google Scholar 

  33. Kocsis, L., Szepesvári, C.: Bandit based monte-carlo planning. Presented at the Proceedings of the 17th European Conference on Machine Learning, Berlin, Germany (2006)

    Google Scholar 

  34. Grüttner, M., Sehnke, F., Schaul, T., Schmidhuber, J.: Multi-dimensional deep memory Atari-Go players for parameter exploring policy gradients. In: Diamantaras, K., Duch, W., Iliadis, L.S., (eds.) Artificial Neural Networks – ICANN 2010: 20th International Conference, Thessaloniki, Greece, 15–18 September 2010, Proceedings, Part II, pp. 114–123. Springer, Heidelberg (2010)

    Google Scholar 

  35. Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: Kremer, S.C., Kolen, J.F., (eds.) A Field Guide to Dynamical Recurrent Neural Networks (2001)

    Google Scholar 

  36. Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., Schmidhuber, J.: Parameter-exploring policy gradients. Neural Netw. 23(4), 551–559 (2010)

    Article  Google Scholar 

  37. Beyer, H.-G., Schwefel, H.-P.: Evolution strategies – a comprehensive introduction. Nat. Comput. 1, 3–52 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  38. Clark, C., Storkey, A.: Teaching deep convolutional neural networks to play Go, arXiv preprint arXiv:1412.3409 (2014)

  39. Koutní, J., Cuccu, G., Schmidhuber, J., Gomez, F.: Evolving large-scale neural networks for vision-based reinforcement learning. In: Proceedings of the Genetic and Evolutionary Computation Conference, Amsterdam, pp. 1061–1068 (2013)

    Google Scholar 

  40. Lange, S., Riedmiller, M.: Deep auto-encoder neural networks in reinforcement learning. In: The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2010)

    Google Scholar 

  41. Lange, S., Riedmiller, M., Voigtlaender, A.: Autonomous reinforcement learning on raw visual input data in a real world application. In: International Joint Conference on Neural Networks, pp. 1–8 (2012)

    Google Scholar 

  42. Ormoneit, D., Sen, Ś.: Kernel-based reinforcement learning. Mach. Learn. 49, 161–178 (2002)

    Article  MATH  Google Scholar 

  43. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  44. Bakker, B., Zhumatiy, V., Gruener, G., Schmidhuber, J.: A robot that reinforcement-learns to identify and memorize important previous observations. In: 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2003, (IROS 2003), Proceedings, vol. 1, pp. 430–435 (2003)

    Google Scholar 

  45. Hausknecht, M., Stone, P.: Deep recurrent Q-learning for partially observable MDPs, arXiv preprint arXiv:1507.06527v3 (2015)

Download references

Author information

Authors and Affiliations

  1. The College of Engineering and Informatics, National University of Ireland, Galway, Republic of Ireland

    Seyed Sajad Mousavi, Michael Schukat & Enda Howley

Authors
  1. Seyed Sajad Mousavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Schukat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Enda Howley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seyed Sajad Mousavi .

Editor information

Editors and Affiliations

  1. Faculty of Computing and Engineering, School of Computing and Mathematics, University of Ulster at Jordanstown, Newtownabbey, United Kingdom

    Yaxin Bi

  2. The Science and Information (SAI) Organization, Bradford, West Yorkshire, United Kingdom

    Supriya Kapoor

  3. The Science and Information (SAI) Organization, Bradford, West Yorkshire, United Kingdom

    Rahul Bhatia

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Cite this paper

Mousavi, S.S., Schukat, M., Howley, E. (2018). Deep Reinforcement Learning: An Overview. In: Bi, Y., Kapoor, S., Bhatia, R. (eds) Proceedings of SAI Intelligent Systems Conference (IntelliSys) 2016. IntelliSys 2016. Lecture Notes in Networks and Systems, vol 16. Springer, Cham. https://doi.org/10.1007/978-3-319-56991-8_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-56991-8_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-56990-1

  • Online ISBN: 978-3-319-56991-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation