Skip to main content
SpringerLink
Log in

Risk Sensitive Markov Decision Processes

  • Conference paper
Systems and Control in the Twenty-First Century

Part of the book series: Systems & Control: Foundations & Applications ((PSCT,volume 22))

Abstract

Risk-sensitive control is an area of significant current interest in stochastic control theory. It is a generalization of the classical, risk-neutral approach, whereby we seek to minimize an exponential of the sum of costs that depends not only on the expected cost, but on higher order moments as well.

Research supported in part by the National Science Foundation under grant EEC 9402384.

Research supported in part by a grant from the University of Arizona Foundation and the Office of the Vice President for Research; and in part by the National Science Foundation under grant NSF-INT 9201430.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Arapostathis, V.S. Borkar, E. Fernández-Gaucherand, M.K. Ghosh and S.I. Marcus. Discrete-time controlled Markov processes with average cost criterion: a survey. SIAM J. Control and Optim. 31 (1993), 282–344.

    Article  MathSciNet  MATH  Google Scholar 

  2. J.S. Baras and M.R. James. Robust and risk-sensitive output feedback control for finite state machines and hidden Markov models. To appear J. Math. Systems, Estimation and Control.

    Google Scholar 

  3. A.Bensoussan and J.H. Van Schuppen. Optimal control of partially observable stochastic systems with exponential of integral performance index. SIAM J. Control and Optim. 23 (1985), 599–613.

    Article  MathSciNet  Google Scholar 

  4. D.P. Bertsekas. Dynamic Programming: Deterministic and Stochastic Model. Englewood Cliffs, NJ: Prentice-Hall, 1987.

    Google Scholar 

  5. V.S. Borkar. On minimum cost per unit of time control of Markov chains. SIAM J. Cont. and Optim. 22 (1984), 965–978.

    Article  MathSciNet  MATH  Google Scholar 

  6. R. Cavazos-Cadena. Weak conditions for the existence of optimal stationary policies in average Markov decision chains with unbounded costs. Kybernetika 25 (1989), 145–156.

    MathSciNet  MATH  Google Scholar 

  7. R. Cavazos-Cadena and L.I. Sennott. Comparing recent assumptions for the existence of average optimal stationary policies. Oper. Res. Lett. 11 (1992), 33–37.

    Article  MathSciNet  MATH  Google Scholar 

  8. K.-J. Chung and M.J. Sobel. Discounted MDP’s: Distribution functions and exponential utility maximization. SI AM J. Control and Optim. 25 (1987), 49–62.

    Article  MathSciNet  MATH  Google Scholar 

  9. S. Coraluppi and S.I. Marcus. Risk-sensitive control of Markov decision processes. Proc. 1996 Conf. on Information Science and Systems. 934–939.

    Google Scholar 

  10. P. Dupuis and R.S. Ellis. A Weak Convergence Approach to the Theory of Large Deviations. To be published by John Wiley & Sons.

    Google Scholar 

  11. J.N. Eagle, II. A Utility Criterion for the Markov Decision Process. Ph.D. Dissertation. Stanford University. Stanford, Caliornia. 1975.

    Google Scholar 

  12. C.-H. Fan, J.L. Speyer and C.R. Jaensch. Centralized and decentralized solutions fo the linear-exponential-Gaussian problem. IEEE Transactions on Automatic Control 39 (1994), 1986–2003.

    Article  MathSciNet  MATH  Google Scholar 

  13. E. Fernández-Gaucherand, A. Arapostathis, and S.I. Marcus. On the average cost optimality equation and the structure of optimal Policies for partially observable Markov decision processes. Annals of Operations Research 29 (1991), 439–470.

    Article  MathSciNet  MATH  Google Scholar 

  14. E. Fernández-Gaucherand, A. Arapostathis and S.I. Marcus. Analysis of an adaptive control scheme for a partially observed controlled Markov chain. IEEE Transactions on Automatic Control 38 (1993), 987–993.

    Article  MATH  Google Scholar 

  15. E. Fernández-Gaucherand and S.I. Marcus. Risk-sensitive optimal control of hidden Markov models: structural results. To appear IEEE Transactions on Automatic Control.

    Google Scholar 

  16. W.H. Fleming and D. Hernández-Hernández. Risk sensitive control of finite state machines on an infinite horizon I. To appear SIAM J. Control and Optim.

    Google Scholar 

  17. W.H. Fleming and D. Hernández-Hernández. Risk sensitive control of finite state machines on an infinite horizon II. Technical Report. Division of Applied Mathematics. Brown University.

    Google Scholar 

  18. W.H. Fleming and W.M. McEneaney. Risk-sensitive control and differential games. Springer Lecture Notes in Control and Info. Sci. 184 (1992), 185–197.

    Article  MathSciNet  Google Scholar 

  19. W.H. Fleming and W.M. McEneaney. Risk-sensitive control on an infinite horizon. SIAM J. Control and Optim. 33 (1995), 1881–1915.

    Article  MathSciNet  MATH  Google Scholar 

  20. K. Glover and J.C. Doyle, State-space formulae for all stabilizing controllers that satisfy an H∞-norm bound and relations to risk sensitivit. Systems and Control Lett. 11 (1988), 167–172.

    Article  MathSciNet  MATH  Google Scholar 

  21. L.P. Hansen and T.J. Sargent. Discounted linear exponential quadratic Gaussian control. IEEE Transactions on Automatic Control 40 (1995), 968–971.

    Article  MathSciNet  MATH  Google Scholar 

  22. D. Hernández-Hernández and S.I. Marcus. Risk-sensitive control of Markov processes in countable state space. To appear Systems and Control Lett.

    Google Scholar 

  23. D. Hernández-Hernández and S.I. Marcus. Existence of risk sensitive optimal stationary policies for controlled Markov processes. Technical Report. Institute for Systems Research. University of Maryland.

    Google Scholar 

  24. D. Hernández-Hernández, S.I. Marcus, and P. Fard. Analysis of a risk sensitive control problem for hidden Markov chains. Technical Report. Institute for Systems Research. University of Maryland.

    Google Scholar 

  25. R.A. Howard and J.E. Matheson. Risk-sensitive Markov decision processes. Management Sci. 18 (1972), 356–369.

    Article  MathSciNet  MATH  Google Scholar 

  26. D.H. Jacobson. Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games. IEEE Transactions on Automatic Control 18 (1973), 124–131.

    Article  MATH  Google Scholar 

  27. M.R. James, J.S. Baras and R.J. Elliott. Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systems. IEEE Transactions on Automatic Control 39 (1994), 780–792.

    Article  MathSciNet  MATH  Google Scholar 

  28. D.M. Kreps and E.L. Porteus. Temporal resolution of uncertainty and dynamic choice theory. Econometrica 46 (1978), 185–200.

    Article  MathSciNet  MATH  Google Scholar 

  29. P.R. Kumar and P. Varaiya. Stochastic Systems: Estimation, Identification and Adaptive Control. Englewood Cliffs, NJ: Prentice-Hall, 1986.

    MATH  Google Scholar 

  30. W.S. Lovejoy. On the convexity of policy regions in partially observed systems. Operations Research 35 (1987), 619–621.

    Article  MathSciNet  MATH  Google Scholar 

  31. T. Runolfsson. The equivalence between infinite horizon control of stochastic systems with exponential-of-integral performance index and stochastic differential games. IEEE Transactions on Automatic Control 39 (1994), 1551–1563.

    Article  MathSciNet  MATH  Google Scholar 

  32. R.D Smallwood and E.J. Sondik. The optimal control of partially observable Markov processes over a finite horizon. Operations Research 21 (1973), 1071–1088.

    Article  MATH  Google Scholar 

  33. C.C. White. A Markov quality control process subject to partial observation. Management Science 23 (1977), 843–852.

    Article  MATH  Google Scholar 

  34. P. Whittle. Risk-Sensitive Optimal Control New York: John Wiley & Sons, 1990.

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering Department and Institute for Systems Research, University of Maryland, 20742, College Park, Maryland, USA

    S. I. Marcus, S. Coraluppi & P. Fard

  2. Systems and Industrial Engineering Department, University of Arizona, 85721, Tucson, Arizona, USA

    E. Fernández-Gaucherand

  3. Department of Mathematics, CINVESTAV-IPN, Apartado postal 14-740, 07000, Mexico D.F., Mexico

    D. Hernández-Hernández

Authors
  1. S. I. Marcus
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. E. Fernández-Gaucherand
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Hernández-Hernández
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. S. Coraluppi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. P. Fard
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Engineering and Applied Science, Washington University, 63130-4899, St. Louis, MO, USA

    Christopher I. Byrnes

  2. Dept. of Mathematical Sciences, Northern Illinois University, 60115, DeKalb, IL, USA

    Biswa N. Datta

  3. Dept. of Mathematics, Texas Tech University, 79409, Lubbock, TX, USA

    Clyde F. Martin  & David S. Gilliam  & 

Rights and permissions

Reprints and permissions

Copyright information

© 1997 Springer Science+Business Media New York

About this paper

Cite this paper

Marcus, S.I., Fernández-Gaucherand, E., Hernández-Hernández, D., Coraluppi, S., Fard, P. (1997). Risk Sensitive Markov Decision Processes. In: Byrnes, C.I., Datta, B.N., Martin, C.F., Gilliam, D.S. (eds) Systems and Control in the Twenty-First Century. Systems & Control: Foundations & Applications, vol 22. Birkhäuser, Boston, MA. https://doi.org/10.1007/978-1-4612-4120-1_14

Download citation

  • DOI: https://doi.org/10.1007/978-1-4612-4120-1_14

  • Publisher Name: Birkhäuser, Boston, MA

  • Print ISBN: 978-1-4612-8662-2

  • Online ISBN: 978-1-4612-4120-1

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation