Abstract
This volume deals with the theory of Markov Decision Processes (MDPs) and their applications. Each chapter was written by a leading expert in the respective area. The papers cover major research areas and methodologies, and discuss open questions and future research directions. The papers can be read independently, with the basic notation and concepts of Section 1.2. Most chap- ters should be accessible by graduate or advanced undergraduate students in fields of operations research, electrical engineering, and computer science.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
E. Altman, Constrained Markov Decision Processes, Chapman & Hall/CRC, Boca Raton, 1999.
R. Anupindi and Y. Bassok, “Supply contracts with quantity commitments and stochastic demand,” in Quantitative Models for Supply Chain Management (S. Tayur, R. Ganeshan, M. Magazine, eds.), pp. 197–232, Kluwer, Boston, 1999.
K.J. Arrow, D. Blackwell, and M.A. Girshick, “Bayes and minimax so- lutions of sequential decision processes,” Econometrica 17, pp. 213–244, 1949.
K.J. Arrow, T. Harris, and J. Marschak, “Optimal inventory policies,” Econometrica 19, pp. 250–272, 1951.
J. Bather, “Optimal decision procedures for finite Markov chains I,” Adv. Appl. Prob. 5, pp. 328–339, 1973.
R.E. Bellman, Dynamic Programming, Princeton University Press, Princeton, 1957.
R.E. Bellman and D. Blackwell, “On a particular non-zero sum game,” RM-250, RAND Corp., Santa Monica, 1949.
R.E. Bellman and J.P. LaSalle, “On non-zero sum games and stochastic processes,” RM-212, RAND Corp., Santa Monica, 1949.
D.P. Bertsekas, Dynamic Programming and Optimal Control: Volume I, Athena Scientific, Bellmont, MA, 2000 (second edition).
D.P. Bertsekas, Dynamic Programming and Optimal Control: Volume II, Athena Scientific, Bellmont, MA, 1995.
D.P. Bertsekas and S.E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, New York, 1978 (republished by Athena Scientific, 1997).
D.P. Bertsekas and J.N. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, Bellmont, MA, 1996.
K.-J. Bierth, “An expected average reward criterion”, Stochastic Processes and Applications 26, pp. 133–140, 1987.
D. Blackwell, “Discrete dynamic programming,” Ann. Math. Stat. 33, pp. 719–726, 1962.
D. Blackwell, “The stochastic processes of Borel gambling and dynamic programming,” Annals of Statistics 4, pp. 370–374, 1976.
V.S. Borkar, Topics in Controlled Markov Chains, Pitman research Notes in Math., 240, Longman Scientific and Technical, Harlow, 1991.
R.Ya. Chitashvili, “A controlled finite Markov chain with an arbitrary set of decisions,” SIAM Theory Prob. Appl. 20, pp. 839–846, 1975.
K.L. Chung, Markov Chains with Stationary Transition Probabilities, Springer-Verlag, Berlin, 1960.
C. Derman, “On sequential decisions and Markov chains,” Man. Sci. 9, pp. 16–24, 1962.
L.E. Dubins and L.J. Savage, How to Gamble if You Must: Inequalities for Stochastic Processes, McGraw-Hill, New York, 1965.
A. Dvoretsky, J. Kiefer, and J. Wolfowitz, “The inventory problem: I. Case of known distribution of demand,” Econometrica 20, pp. 187–222, 1952.
E.B. Dynkin and A.A. Yushkevich, Controlled Markov Processes, Springer-Verlag, New York, 1979 (translation from 1975 Russian edition).
A. Federgruen, “Centralized planning models for multi-echelon inventory systems under inventory,” in Logistics of Production and Inventory, (S.C. Graves, A.H.G. Rinnooy Kan, and P.H. Zipkin, eds), Handbooks in Operations Research and Management Science, 4, pp. 133–173, North-Holland, Amsterdam, 1993.
E. A. Feinberg, “The existence of a stationary ε-optimal policy for a finite Markov chain,” SIAM Theory Prob. Appl. 23, pp. 297–313, 1978.
E.A. Feinberg, “An ε-optimal control of a finite Markov chain with an average reward criterion,” SIAM Theory Prob. Appl. 25, pp. 70–81, 1980.
E.A. Feinberg, “Controlled Markov processes with arbitrary numerical criteria,” SIAM Theory Prob. Appl. 27 pp. 486–503, 1982.
E.A. Feinberg and H. Park, “Finite state Markov decision models with average reward criteria,” Stoch. Processes Appl., 31 pp. 159–177, 1994.
J.A. Filar and D. Krass, “Hamiltonian cycles and Markov chains,” Math. Oper. Res. 19, pp. 223–237, 1994.
O. Hernández-Lerma, Adaptive Markov Control Processes, Springer, New York, 1989.
O. Hernández-Lerma and J.B. Lasserre, Further Topics in Discrete-Time Markov Control Processes, Springer, New York, 1999.
D.P. Heyman and M.J. Sobel, Stochastic Methods in Operations Research. Volume II: Stochastic Optimization, McGraw-Hill, New York, 1984.
K. Hinderer, Foundations of Non-stationary Dynamic Programming with Discrete Time Parameter, Springer-Verlag, New York, 1970.
R. A. Howard Dynamic Programming and Markov Processes, MIT Press, Cambridge, 1960.
L.C.M. Kallenberg, Linear Programming and Finite Markovian Control Problems, Mathematical Centre Tract 148, Mathematical Centre, Amsterdam, 1983.
A.S. Kechris, Classical Descriptive Set Theory, Springer-Verlag, New York, 1995.
M.Yu. Kitaev and V.V. Rykov, Controlled Queueing Systems, CRC Press, Boca Raton, 1995.
A.J. Kleywegt and J.D. Papastavrou, “Acceptance and dispatching poli- cies for a distribution problem”, Transportation Science, 32, pp. 127–141, 1998.
A.P Maitra and W.D. Sudderth, Discrete Gambling and Stochastic Games, Springer, New York, 1996.
J. Neveu, Mathematical Foundations of the Calculus of Probability, Holden-Day, San Francisco, 1965.
A.B. Piunovskiy, Optimal Control of Random Sequences in Problems with Constraints, Kluwer, Dordrecht, 1997.
A.B. Piunovskiy and X. Mao, “Constrained Markovian decision processes: the dynamic programming approach,” Operations Research Letters 27, pp. 119–126, 2000.
E.L. Porteus, “Stochastic inventory theory,” in Stochastic Models, (D.P. Heyman and M.J. Sobel, eds), Handbooks in Operations Research and Management Science, 2, pp. 605–652, North-Holland, Amsterdam, 1990.
M.L. Puterman, Markov Decision Processes, Wiley, New York, 1994.
M. Schäl, “On stochastic dynamic programming: a bridge between Markov decision processes and gambling.” Markov processes and control theory, pp. 178–216, Math. Res. 54, Akademie-Verlag, Berlin, 1989.
L. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley, New York, 1999.
S. Ross, Introduction to Stochastic Dynamic Programming, Academic Press, New York, 1983.
L.S. Shapley, “Stochastic games,” Proceedings of the National Academy of Sciences, pp. 1095–1100, 1953.
A.N. Shiryaev, “On the theory of decision functions and control by an observation process with incomplete data,” Selected Translations in Math. Statistics and Probability 6, pp. 162–188, 1966.
N.L. Stokey and R.E. Lucas, Jr. Recursive Methods in Economic Dynamics, Harvard University Press, Cambridge, 1989.
A. Wald, Sequential Analysis, Wiley, New York, 1947.
P. Whittle, Risk-Sensitive Optimal Control, Wiley, NY, 1990
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer Science+Business Media New York
About this chapter
Cite this chapter
Feinberg, E.A., Shwartz, A. (2003). Introduction. In: Feinberg, E.A., Shwartz, A. (eds) Handbook of Markov Decision Processes. International Series in Operations Research & Management Science, vol 40. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-0805-2_1
Download citation
DOI: https://doi.org/10.1007/978-1-4615-0805-2_1
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-5248-8
Online ISBN: 978-1-4615-0805-2
eBook Packages: Springer Book Archive