Elsevier

Automatica

Volume 81, July 2017, Pages 56-67
Automatica

Staticization, its dynamic program and solution propagation

https://doi.org/10.1016/j.automatica.2017.03.004Get rights and content

Abstract

Stationary-action formulations of dynamical systems are considered. Use of stationary-action formulations allows one to generate fundamental solutions for classes of two-point boundary-value problems (TPBVPs). One solves for stationary points of the payoff as a function of inputs rather than minimization/maximization, a task which is significantly different from that in optimal control problems. Both a dynamic programming principle (DPP) and a Hamilton–Jacobi partial differential equation (HJ PDE) are obtained for a class of problems subsuming the stationary-action formulation. Although convexity (or concavity) of the payoff may be lost as one propagates forward, stationary points continue to exist, and one must be able to use the DPP and/or HJ PDE to solve forward to such time horizons. In linear/quadratic models, this leads to a requirement for propagation of solutions of differential Riccati equations past finite escape times. Such propagation is also required in (nonlinear) n-body problem formulations where the potential is represented via semiconvex duality. The dynamic programming tools developed here are applicable.

Introduction

The classical approach to solution of energy-conserving dynamical systems is integration of Newton’s second law. An alternative viewpoint is that a system evolves along a path which makes the action functional stationary, i.e., such that the first-order differential around the path is the zero element. This latter viewpoint appears particularly useful in some applications in modern physics, including gravitational systems where relativistic effects are non-negligible and systems in the quantum domain (cf. Feynman, 1948; Feynman, 1964; Gray & Taylor, 2007; Padmanabhan, 2010). Our interests are more pedestrian; the stationary-action formulation has recently been found to be quite useful for generation of fundamental solutions to two-point boundary-value problems (TPBVPs) for conservative dynamical systems. For sufficiently short time horizons, stationarity of the action typically corresponds to minimization of the action. That is, the stationary point is a global minimum of that action (cf., Dower & McEneaney, 2013; McEneaney and Dower, 2013, McEneaney and Dower, 2015). For longer time horizons, the stationary point is more typically a saddle.

As our motivating interest is in solution of TPBVPs for conservative dynamical systems, we note that this specifically includes mass–spring, wave equation and n-body problems (Dower and McEneaney, 2013, McEneaney and Dower, 2013, McEneaney and Dower, 2015). By appending a min-plus delta function terminal cost to the action functional, we obtain a fundamental solution object for such TPBVPs. Min-plus convolutions of this object with functionals associated to specific terminal conditions yield the solutions of the specific TPBVPs. As a change in the boundary data only requires convolution with a different functional, our object may best be termed a fundamental solution for TPBVPs, corresponding to the given time horizon. It is worth remarking that, further, one can populate the fundamental solution semigroup by convolving the fundamental solution with itself, enabling solution of the TPBVP for all strictly positive horizons.

As noted above, for sufficiently short time horizons, one may obtain the stationary action solution by minimization of the action functional, in which case it is obvious that the fundamental solution is derived from the value function for an optimal control formulation. However, for longer horizons, we must find the stationary point, and this requires a new set of tools. We define stationarity and value for such problems. Surprisingly, for a specific class of terminal costs, one may obtain a dynamic programming principle (DPP) for stationarity, where this is directly analogous to standard DPPs (for optimization). We do not look for the absolute weakest assumptions, but only a reasonable first-foray set. We also formally write the corresponding Hamilton–Jacobi partial differential equation (HJ PDE), and then obtain a verification result, which is also quite similar to that found for classical optimal control problems. We remark that a verification result implies that any solution of the HJ PDE in the specified class must be the value function. This validates the approach of solving a stationarity problem (and hence the related TPBVP if the stationarity problem is generated by such) by solving the associated HJ PDE problem.

In the mass–spring case (which appears in Section  3 as a motivating example), the stationary-action problem is linear–quadratic, and the HJ PDE reduces to a differential Riccati equation (DRE). By the above-noted verification result, we see that solution of the DRE yields solution of the stationarity problem, and any corresponding TPBVP. The wave equation (Dower & McEneaney, 2013) also yields a DRE, albeit infinite dimensional. The n-body problem may be reduced to a parameterized set of time-dependent DREs (McEneaney and Dower, 2013, McEneaney and Dower, 2015). We see that in all cases, solutions of DREs form a critical building block. Of course, DREs can exhibit finite escape times, and do so in these cases. In classical optimal control, one is not interested in propagation of the solution past such escape times. However, in stationarity problems, these may correspond to points where one loses convexity [concavity] of the payoff. Although the minimum [maximum] may go to [+], the stationary value may be well-defined and finite past such asymptotes, and one must propagate the solution beyond them. The DPP yields a means for propagation through escape times, and this will be indicated.

Although stationary action is the motivating problem class, the theory developed below is applicable to wider classes of problems, where one is seeking a stationary point. An obvious example is that of certain differential games. Extensions to stochastic cases appear possible as well, but are not considered here.

Section  2 contains relevant definitions. Section  3 presents a simple mass–spring TPBVP motivating example. Section  4 contains the main results—the DPP and HJ PDE verification theorem. Section  5 reduces to the linear/quadratic case, and indicates a means for propagation of DREs past escape times. Section  6 very briefly indicates some application areas.

Section snippets

Stationarity definitions

Recall that we are seeking stationary points of payoffs, which is unusual in comparison to the standard classes of problems in optimization. In analogy with the language for minimization and maximization, we will refer to the search for stationary points as staticization, with these points being statica (in analogy with minima/maxima) and a single such point being a staticum (in analogy with minimum/maximum). Prior to the development, we make the following definitions. Suppose Y is a generic

Motivational examples

As indicated in the introduction, an important problem class which motivates this effort is that of TPBVPs for conservative systems.

Dynamic programming for staticization

In this section, we will obtain a dynamic programming principle as well as a verification result for the appropriate Hamilton–Jacobi PDE.

Remark 3

For clarity, we provide several definitions. Let Y,Z be Hilbert spaces, and let F:Y×ZR. We say F is strictly uniformly convex on Y with respect to zZ, if there exists CF>0 such that for all y,vY, zZ, |F(yv,z)2F(y,z)+F(y+v,z)|CF|v|2. We say F is coercive on Y if given R̄,M<, there exists Rˆ< such that F(y,z)F(0,z)+M for all |y|Rˆ, |z|R̄. We say F

Linear–quadratic example

We consider the linear–quadratic problem given by L(x,v)=12vDv12xBx,f(x,v)=v,ψ(x,z)=ψc(x,z), for all x,v,zRn, where DdI (where we write AB if AB is positive definite), d>0, B and D are symmetric, and c(0,). We look for W¯ of the form W¯(t,x,z)=12[xP(t)x+2xQ(t)z+zR(t)z]. With the above quadratic cost and given dynamics, the HJ PDE (25) takes the form 0=statvRn[12vDv12xBxW¯r(r,x,z)+W¯x(r,x,z)v]=minvRn[12vDv12xBxW¯r(r,x,z)+W¯x(r,x,z)v]=12xBxW¯r(r,x,z)12W¯x(r,x,z)D1W¯x(

Applications

The initial motivation for consideration of stationarity problems is the stationary action principle for dynamical systems. However, from the above, one sees that the main results, specifically the dynamic programming principle and the relation to the associated HJ PDE, are valid on a much larger domain. One may seek the staticum in a wide variety of problems. One obvious additional application is in the area of zero-sum games. In particular, when value exists, this value is a minimax solution

William M. McEneaney received his B.S. and M.S. in Mathematics from Rensselaer Polytechnic Institute in 1982 and 1983, respectively. He worked at PAR Technology and Jet Propulsion Laboratory, developing theory and algorithms for estimation and guidance applications. Prof. McEneaney attended Brown University from 1989 through 1993, obtaining his M.S. and Ph.D. in Applied Mathematics. His thesis research, conducted under Prof. W.H. Fleming, was on nonlinear risk-sensitive stochastic control.

References (16)

  • W.M. McEneaney

    A new fundamental solution for differential Riccati equations Arising in Control

    Automatica

    (2008)
  • R. Curtain et al.
  • Dower, P.M., & McEneaney, W.M. (2011). A max-plus based fundamental solution for a class of infinite dimensional...
  • Dower, P.M., & McEneaney, W.M. (2013). A fundamental solution for an infinite dimensional two-point boundary value...
  • Dower, P.M., & McEneaney, W.M. (2015a). A max-plus fundamental solution semigroup for a class of lossless wave...
  • P.M. Dower et al.

    A max-plus dual space fundamental solution for a class of operator differential Riccati equations

    SIAM Journal on Control and Optimization

    (2015)
  • Dower, P.M., & McEneaney, W.M. (2015c). Solving two-point boundary value problems for a wave equation via the principle...
  • R.P. Feynman

    Space–time approach to non-relativistic quantum mechanics’

    Reviews of Modern Physics

    (1948)
There are more references available in the full text version of this article.

Cited by (18)

  • Static duality and a stationary-action application

    2018, Journal of Differential Equations
    Citation Excerpt :

    This latter viewpoint appears particularly useful in some applications in modern physics, including systems where relativistic effects are non-negligible and systems in the quantum domain (cf. [7,8,10,23,26]). The stationary-action formulation has also recently been found to be quite useful for generation of fundamental solutions to two-point boundary-value problems (TPBVPs) for conservative dynamical systems (cf. [3,4,17,18,20]). It has also been demonstrated that this stationary-action approach may be applied to TPBVPs for infinite-dimensional conservative systems described by classes of lossless wave equations, see for example [3,4].

  • Min-max and stat game representations for nonlinear optimal control problems

    2023, Proceedings of the American Control Conference
  • STATICIZATION AND ITERATED STATICIZATION

    2022, SIAM Journal on Control and Optimization
  • Exploiting Characteristics in Stationary Action Problems

    2021, Applied Mathematics and Optimization
View all citing articles on Scopus

William M. McEneaney received his B.S. and M.S. in Mathematics from Rensselaer Polytechnic Institute in 1982 and 1983, respectively. He worked at PAR Technology and Jet Propulsion Laboratory, developing theory and algorithms for estimation and guidance applications. Prof. McEneaney attended Brown University from 1989 through 1993, obtaining his M.S. and Ph.D. in Applied Mathematics. His thesis research, conducted under Prof. W.H. Fleming, was on nonlinear risk-sensitive stochastic control. Prof. McEneaney has held positions at Carnegie Mellon University and North Carolina State University, prior to his current appointment in the Mechanical and Aerospace Engineering Department of University of California, San Diego. His recent interests have been in Max-Plus Algebraic Methods for Hamilton–Jacobi–Bellman Partial Differential Equations, the Principle of Stationary Action, Risk-Sensitive and Robust Control and Estimation, and Partially-Observed Stochastic Games.

Peter M. Dower is an Associate Professor with the Department of Electrical and Electronic Engineering at the University of Melbourne, Australia. Peter received a Computer Engineering degree (with Hons.) from the University of Newcastle, Australia, and a Ph.D. in Engineering from the Australian National University. He subsequently was a post-doctoral fellow in the Department of Mathematics at the University of California San Diego. His research interests are primarily in the areas of optimal control and robust stability analysis for nonlinear systems, with an emphasis on dynamic programming, Hamilton–Jacobi–Bellman PDEs, idempotent analysis, and computational methods. His research is partially supported by the Australian Research Council and AFOSR.

Research partially supported by NSF Grant DMS-1312569, AFOSR Grants FA9550-15-1-0131 and FA2386-16-1-4066 and the Australian Research Council Grant DP120101549. The material in this paper was partially presented at the SIAM Conference on Control and Its Applications, July 8–10, 2015, Paris, France. This paper was recommended for publication in revised form by Associate Editor Kok Lay Teo under the direction of Editor Ian R. Petersen.

View full text