Skip to main content
SpringerLink
Log in

Algorithms for stochastic games — A survey

  • Published:
Zeitschrift für Operations Research Aims and scope Submit manuscript

Abstract

We consider finite state, finite action, stochastic games over an infinite time horizon. We survey algorithms for the computation of minimax optimal stationary strategies in the zerosum case, and of Nash equilibria in stationary strategies in the nonzerosum case. We also survey those theoretical results that pave the way towards future development of algorithms.

Zusammenfassung

In dieser Arbeit werden unendlichstufige stochastische Spiele mit endlichen ZuStands- und Aktionenräumen untersucht. Es wird ein Überblick gegeben über Algorithmen zur Berechnung von optimalen stationären Minimax-Strategien in Nullsummen-Spielen und von stationären Nash-Gleichgewichtsstrategien in Nicht-Nullsummen-Spielen. Einige theoretische Ergebnisse werden vorgestellt, die für die weitere Entwicklung von Algorithmen nützlich sind.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aumann RJ (1964) Mixed and Behaviour Strategies in Infinite Extensive Games. In: Dresher M, Shapley LS (eds) Advances in Game Theory. Princeton University Press. Annals of Mathematics Studies 52

  2. Baykal-Gursoy M, Ross KW (1989) A Sample Path Approach to Stochastic Games. Techn Rep, Rutgers University

  3. Bewley T, Kohlberg E (1976) The asymptotic theory of stochastic games. Math Oper Res 1:197–208

    Google Scholar 

  4. Bewley T, Kohlberg E (1978) On Stochastic Games with Stationary Optimal Strategies. Math Oper Res 3:104–125

    Google Scholar 

  5. Blackwell D, Ferguson T (1968) The Big Match. Ann Math Statistics 39:159–163

    Google Scholar 

  6. Breton M (1987) Equilibre pour des Jeux Sequentiel. PhD Thesis, University of Montreal

  7. Breton M, Filar JA, Haurie A, Shultz TA (1985) On the Computation of Equilibria in Discounted Stochastic Games. In: Basar T (ed) Dynamic Games and Applications in Economics. Springer, Lecture Notes in Economics and Mathematical Systems 265

  8. Brown GW (1951) Iterative Solutions of Games by Fictitious Play. In: Koopmans TC (ed) Activity Analysis of Production and Allocation. Wiley

  9. Charnes A, Schroeder R (1967) On Some Tactical Antisubmarine Games. Naval Res Log Qtly 14:291–311

    Google Scholar 

  10. Derman C (1970) Finite State Markovian Decision Processes. Academic Press, New York

    Google Scholar 

  11. Federgruen A (1980) Successive Approximation Methods in Undiscounted Stochastic Games. Oper Res 28:794–810

    Google Scholar 

  12. Federgruen A (1983) Markovian Control Problems. Mathematical Centre Tracts 97, Amsterdam

  13. Filar JA (1980) Algorithms for Solving Some Undiscounted Stochastic Games. PhD Thesis, University of Illinois at Chicago, Chicago, Illinois

    Google Scholar 

  14. Filar JA Ordered Field Property for Stochastic Games When the Player Who Controls Transitions Changes from State to State. J Optim Theory Appl 34:503–513

  15. Filar JA (1985) Player Aggregation in the Traveling Inspector Model. IEEE Trans on Aut Control AC-30:723–729

    Google Scholar 

  16. Filar JA (1986) Quadratic Programming and the Single-Controller Stochastic Game. J Math Anal Appl 113:136–147

    Google Scholar 

  17. Filar JA, Raghavan TES (1980) Two Remarks Concerning Two Undiscounted Stochastic Games. Tech Rep 392. John Hopkins University, Department of Mathematical Sciences

  18. Filar JA, Raghavan TES (1984) A Matrix Game Solution of the Single-Controller Stochastic Game. Math Oper Res 9:356–362

    Google Scholar 

  19. Filar JA, Shultz TA (1986) Nonlinear Programming and Stationary Strategies in Stochastic Games. Math Progr 35:243–247

    Google Scholar 

  20. Filar JA, Shultz TA (1987) Bilinear Programming and Structured Stochastic Games. J Optim Theory Appl 53:85–104

    Google Scholar 

  21. Filar JA, Shultz TA, Thuijsman F, Vrieze OJ (1987) Nonlinear Programming and Stationary Equilibria in Stochastic Games. Math Program (to appear)

  22. Filar JA, Tolwinski B (1988) On the Algorithm of Pollatschek and Avi-Itzhak. In: Ferguson T et al. (eds) Stochastic Games and Related Topics, Shapley Honor volume. Kluwer, Dordrecht, The Netherlands (to appear)

    Google Scholar 

  23. Fink AM (1964) Equilibrium in a Stochastic N-Person Game. J Sci in Hiroshima Univ, Series A-I. 28:89–93

    Google Scholar 

  24. Gilette D (1957) Stochastic Games with Zero Stop Probabilities. In: Dresher AWTM, Wolfe P (eds) Contributions to the Theory of Games. Princeton University Press, Annals of Mathematics Studies 39

  25. Gill PE, Murray W, Saunders MA, Wright MH (1983) User's Guide for SOL/NPSOL: A Fortran Package for Nonlinear Programming. Tech Rep SOL 83-12. Stanford University, Stanford, California

    Google Scholar 

  26. Hoffman AJ, Karp RM (1966) On Non-terminating Stochastic Games. Management Sci 12:359–370

    Google Scholar 

  27. Hordijk A, Kallenberg LGM (1981) Linear Programming and Markov Games I, II. In: Moeschlin O, Pallaschke D (eds) Game Theory and Mathematical Economics. North Holland

  28. Hordijk A, Tijms HC (1975) A Modified Form of the Iterative Method of Dynamic Programming. Annals of Stat 3:203–208

    Google Scholar 

  29. Lemke CE (1965) Bimatrix Equilibrium Points and Mathematical Programming. Management Sci 12:413–423

    Google Scholar 

  30. Lemke CE, Howson JT (1964) Equilibrium Points of Bimatrix Games. J Soc Indust Appl Math 12:413–423

    Google Scholar 

  31. Liggett T, Lipman S (1969) Stochastic Games with Perfect Information and Time Average Payoff. SIAM Rev 11:604–607

    Google Scholar 

  32. Mertens JF, Neyman A (1981) Stochastic Games. International J Game Theory 10:53–56

    Google Scholar 

  33. Mizuno N (1986) A New Algorithm for Non-Zerosum Markov Games. Tech Rep, New York University, Graduate School of Business

  34. Mizuno N (1987) A Necessary Condition for the Existence of Average Reward Equilibrium Points for Finite N-Person Markov Games. Tech Rep, New York University, Graduate School of Business

  35. Mohan SR, Raghavan TES (1987) An Algorithm for Discounted Switching Control Games. OR Spectrum 9:41–45

    Google Scholar 

  36. Monash CA (1979) Stochastic Games. The Minimax Theorem. PhD Thesis, Harvard University

  37. von Neumann J (1928) Zur Theorie der Gesellschaftsspiele. Math Annal 100:295–320

    Google Scholar 

  38. Nowak A, Raghavan TES (1989) A Finite Step Algorithm via a Bimatrix Garne to a Single-Controller Non-zerosum Stochastic Game. Tech Rep 89-1. The University of Illinois at Chicago

    Google Scholar 

  39. Parthasarathy T, Raghavan TES (1981) An Orderfield Property for Stochastic Garnes when One Player Controls Transition Probabilities. J Optim Theory Appl 33:375–392

    Google Scholar 

  40. Parthasarathy T, Stern M (1977) Markov Games: A Survey. In: Roxin PLE, Sternberg R (eds) Differential Games and Control Theory. Marcel Dekker

  41. Parthasarathy T, Tijs SH, Vrieze OJ (1984) Stochastic Games with State Independent Transitions and Separable Rewards. In: Hammer G, Pallaschke D (eds) Selected Topics in OR and Mathematical Economics. Springer, Lecture Notes Series 226

  42. Pollatschek M, Avi-Itzhak B (1969) Algorithms for Stochastic Games with Geometrical Interpretation. Management Sci 15:399–415

    Google Scholar 

  43. Raghavan TES, Tijs SH, Vrieze OJ (1985) On Stochastic Games with Additive Reward and Transition Structure. J Optim Theory Appl 47:451–464

    Google Scholar 

  44. Rao S, Chandrasekaran R, Nair K (1973) Algorithms for Discounted Stochastic Games. J Optim Theory Appl 11:627–637

    Google Scholar 

  45. Robinson J (1950) An Iterative Model of Solving a Game. Ann Math 54:296–301

    Google Scholar 

  46. Rogers PD (1969) Non-zerosum Stochastic Games. PhD Thesis, University of California at Berkeley, Berkeley, California

    Google Scholar 

  47. Rothblum UG (1978) Solving Stopping Stochastic Games by Maximizing a Linear Function Subject to Quadratic Constraints. In: Moeschlin O, Pallaschke D (eds) Game Theory and Related Topics. North Holland

  48. Schweitzer PJ (1968) Perturbation theory and finite Markov chains. J Appl Prob 5:401–413

    Google Scholar 

  49. Shapley LS (1953) Stochastic Games. Proc Nat Acad Sci USA 39:1095–1100

    Google Scholar 

  50. Shapley LS (1964) Some Topics in Two Person Games. In: Dresher LSSM, Tucker AW (eds) Advances in Game Theory. Princeton University Press

  51. Sinha S (1989) A Contribution to the Theory of Stochastic Games. PhD Thesis, Indian Statistical Institute, New Delhi

    Google Scholar 

  52. Sobel MJ (1971) Non-cooperative Stochastic Games. Ann Math Stat 42:1930–1935

    Google Scholar 

  53. Sobel MJ (1981) Myopic Solutions of Markov Decision Processes and Stochastic Games. Operat Res 29:995–1009

    Google Scholar 

  54. Sobel MJ (1981) Stochastic Fishery Games with Myotopic Equilibria. In: Mirman LJ, Spulber D (eds) The Economics of Renewable Resources. Elsevier-North-Holland

  55. Stern M (1975) On Stochastic Games with Limiting Average Payoff. PhD Thesis, University of Illinois at Chicago

    Google Scholar 

  56. Takahashi M (1964) Equilibrium Points of Stochastic Non-cooperative n-Person Games. J Sci Hiroshima University, Series A-I, 28:95–99

    Google Scholar 

  57. Thuijsman F (1989) Optimality and Equilibria in Stochastic Games. PhD Thesis, Rijksuniversiteit Limburg, Maastricht

    Google Scholar 

  58. Vrieze OJ (1981) Linear Programming and Undiscounted Stochastic Games. OR Spectrum 3:29–35

    Google Scholar 

  59. Vrieze OJ (1987) Stochastic Games with Finite State and Action Spaces. CWI Tracts 33, Amsterdam

  60. Vrieze OJ, Thuijsman F (1986) On Equilibria in Repeated Games with Absorbing States. Tech Rep 8535. Catholic University, Nijmegen, Department of Mathematics

    Google Scholar 

  61. Vrieze OJ, Tijs SH (1980) Fictitious Play Applied to Sequence of Games and Discounted Stochastic Games. Intern J Game Theory 11:71–85

    Google Scholar 

  62. Vrieze OJ, Tijs SH, Raghavan TES, Filar JA (1983) A Finite Algorithm for the Switching Controller Stochastic Game. OR Spectrum 5:15–24

    Google Scholar 

  63. van der Wal J (1981) Stochastic Dynamic Programming. Math Center Tracts 139, Amsterdam

  64. van der Wal J (1977) Discounted Markov Games: Successive Approximation and Stopping Times. Intern J Game Theory 6:11–22

    Google Scholar 

  65. Winston W (1978) A Stochastic Game Model of a Weapons Development Competition. SIAM J Control Optim 16:411–419

    Google Scholar 

  66. Winston WL, Cabot AV (1984) A Stochastic Game Model of Football Play Selection. Tech Rep, Indiana University, Paper presented at the TIMS/ORSA joint National meeting in Dallas

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Statistics and Computer Science, University of Illinois at Chicago, 60680, Chicago, IL

    T. E. S. Raghavan

  2. Department of Mathematics and Statistics, University of Maryland Baltimore County, 21288, Baltimore, MD

    J. A. Filar

Authors
  1. T. E. S. Raghavan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. A. Filar
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

This paper is based on the invited lectures given by the authors at the 12th Symposium for Operations Research in Passau, 1987. We are indebted to M. Abbad, Evangelista Fe, F. Thuijsman and O. J. Vrieze for valuable comments and discussion. Any remaining errors of either misinterpretation or of omission are the authors' alone.

Supported in part by the NSF under the grant # DMS-82601403.

Supported in part by the AFOSR and the NSF under the grant # ECS-8704954.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Raghavan, T.E.S., Filar, J.A. Algorithms for stochastic games — A survey. ZOR - Methods and Models of Operations Research 35, 437–472 (1991). https://doi.org/10.1007/BF01415989

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF01415989

Keywords

Search

Navigation