Skip to main content
Log in

Time Discretisation and Rate of Convergence for the Optimal Control of Continuous-Time Stochastic Systems with Delay

  • Published:
Applied Mathematics and Optimization Submit manuscript

Abstract

We study a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochastic delay (or functional) differential equations with bounded memory. Performance is measured in terms of expected costs. By discretising time in two steps, we construct a sequence of approximating finite-dimensional Markovian optimal control problems in discrete time. The corresponding value functions converge to the value function of the original problem, and we derive an upper bound on the discretisation error or, equivalently, a worst-case estimate for the rate of convergence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Atkinson, K., Han, W.: Theoretical Numerical Analysis. Texts in Applied Mathematics, vol. 39. Springer, New York (2001)

    MATH  Google Scholar 

  2. Barles, G., Jakobsen, E.R.: Error bounds for monotone approximation schemes for Hamilton-Jacobi-Bellman equations. SIAM J. Numer. Anal. 43(2), 540–558 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  3. Barles, G., Jakobsen, E.R.: Error bounds for monotone approximation schemes for parabolic Hamilton-Jacobi-Bellman equations. Math. Comput. 76(260), 1861–1893 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  4. Bauer, H., Rieder, U.: Stochastic control problems with delay. Math. Meth. Oper. Res. 62(3), 411–427 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  5. Buckwar, E.: Introduction to the numerical analysis of stochastic delay differential equations. J. Comput. Appl. Math. 125(1–2), 297–307 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  6. Calzolari, A., Florchinger, P., Nappo, G.: Convergence in nonlinear filtering for stochastic delay systems. SIAM J. Control Optim. (to appear)

  7. Chang, M.-H., Pang, T., Pemy, M.: Optimal control of stochastic functional differential equations with bounded memory. In: Zhang, X., Liu, D., Wu, L. (eds.) Operations Research and Its Applications, Papers from the Sixth International Symposium, ISORA’06, Xinjiang, China, August 8–12, pp. 82–94. World Publishing Corporation, Beijing (2006)

    Google Scholar 

  8. Elsanosi, I., Øksendal, B., Sulem, A.: Some solvable stochastic control problems with delay. Stoch. Stoch. Rep. 71(1–2), 69–89 (2000)

    MATH  Google Scholar 

  9. Falcone, M., Ferretti, R.: Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations. Numer. Math. 67(3), 315–344 (1994)

    Article  MATH  MathSciNet  Google Scholar 

  10. Falcone, M., Rosace, R.: Discrete-time approximation of optimal control problems for delayed equations. Control Cybern. 25(3), 665–675 (1996)

    MATH  MathSciNet  Google Scholar 

  11. Fischer, M., Reiß, M.: Discretisation of stochastic control problems for continuous time dynamics with delay. J. Comput. Appl. Math. 205(2), 969–981 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  12. Garsia, A.M., Rodemich, E., Rumsey, H. Jr.: A real variable lemma and the continuity of paths of some Gaussian processes. Indiana Math. J. 20(6), 565–578 (1970)

    Article  MATH  MathSciNet  Google Scholar 

  13. Gihman, I.I., Skorohod, A.V.: Controlled Stochastic Processes. Springer, New York (1979)

    Google Scholar 

  14. Hu, Y., Mohammed, S.-E.A., Yan, F.: Discrete-time approximations of stochastic delay equations: the Milstein scheme. Ann. Probab. 32(1A), 265–314 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  15. Karatzas, I., Shreve, S.E.: Brownian Motion and Stochastic Calculus, 2nd edn. Graduate Texts in Mathematics, vol. 113. Springer, New York (1991)

    MATH  Google Scholar 

  16. Krylov, N.V.: Controlled Diffusion Processes. Applications of Mathematics, vol. 14. Springer, New York (1980)

    MATH  Google Scholar 

  17. Krylov, N.V.: Approximating value functions for controlled degenerate diffusion processes by using piece-wise constant policies. Electron. J. Probab. 4, 1–19 (1999)

    MathSciNet  Google Scholar 

  18. Krylov, N.V.: On the rate of convergence of finite-difference approximations for Bellman’s equations with variable coefficients. Probab. Theory Relat. Fields 117(1), 1–16 (2000)

    Article  MATH  Google Scholar 

  19. Krylov, N.V.: Mean value theorems for stochastic integrals. Ann. Probab. 29(1), 385–410 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  20. Kushner, H.J.: Numerical approximations for nonlinear stochastic systems with delays. Stochastics 77(3), 211–240 (2005)

    MATH  MathSciNet  Google Scholar 

  21. Kushner, H.J.: Numerical approximations for stochastic systems with delays in the state and control. Technical Report, Lefschetz Center for Dynamical systems (2005)

  22. Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time, 2nd edn. Applications of Mathematics, vol. 24. Springer, New York (2001)

    MATH  Google Scholar 

  23. Larssen, B.: Dynamic programming in stochastic control of systems with delay. Stoch. Stoch. Rep. 74(3–4), 651–673 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  24. Larssen, B., Risebro, N.H.: When are HJB-equations for control problems with stochastic delay equations finite dimensional? Stoch. Anal. Appl. 21(3), 643–671 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  25. Mao, X.: Numerical solutions of stochastic functional differential equations. LMS J. Comput. Math. 6, 141–161 (2003)

    MATH  MathSciNet  Google Scholar 

  26. Mohammed, S.-E.A.: Stochastic Functional Differential Equations. Pitman, London (1984)

    MATH  Google Scholar 

  27. Øksendal, B., Sulem, A.: A maximum principle for optimal control of stochastic systems with delay, with applications to finance. In: Menaldi, J., Rofman, E., Sulem, A. (eds.) Optimal Control and Partial Differential Equations. In Honour of Professor Alain Bensoussan’s 60th Birthday. Proceedings of the Conference, Paris, December 4, 2000, pp. 64–79. IOS Press, Amsterdam (2001)

    Google Scholar 

  28. Rogers, L.C.G.: Pathwise stochastic optimal control. SIAM J. Control Optim. 46(3), 1116–1132 (2007)

    Article  MathSciNet  Google Scholar 

  29. Słomiński, L.: Euler’s approximations of solutions of SDEs with reflecting boundary. Stoch. Process. Appl. 94, 317–337 (2001)

    Article  Google Scholar 

  30. Stroock, D.W., Varadhan, S.R.S.: Multidimensional Diffusion Processes. Grundlehren der Mathematischen Wissenschaften, vol. 233. Springer, Berlin (1979)

    MATH  Google Scholar 

  31. Yong, J., Zhou, X.Y.: Stochastic Controls. Hamiltonian Systems and HJB Equations. Applications of Mathematics, vol. 43. Springer, New York (1999)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Markus Fischer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fischer, M., Nappo, G. Time Discretisation and Rate of Convergence for the Optimal Control of Continuous-Time Stochastic Systems with Delay. Appl Math Optim 57, 177–206 (2008). https://doi.org/10.1007/s00245-007-9019-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00245-007-9019-4

Keywords

Navigation