Elsevier

Automatica

Volume 56, June 2015, Pages 44-52
Automatica

Brief paper
Extremum seeking of dynamical systems via gradient descent and stochastic approximation methods

https://doi.org/10.1016/j.automatica.2015.03.018Get rights and content

Abstract

This paper examines the use of gradient based methods for extremum seeking control of possibly infinite-dimensional dynamic nonlinear systems with general attractors within a periodic sampled-data framework. First, discrete-time gradient descent method is considered and semi-global practical asymptotic stability with respect to an ultimate bound is shown. Next, under the more complicated setting where the sampled measurements of the plant’s output are corrupted by an additive noise, three basic stochastic approximation methods are analysed; namely finite-difference, random directions, and simultaneous perturbation. Semi-global convergence to an optimum with probability one is established. A tuning parameter within the sampled-data framework is the period of the synchronised sampler and hold device, which is also the waiting time during which the system dynamics settle to within a controllable neighbourhood of the steady-state input–output behaviour.

Introduction

Extremum seeking locates via online computations an optimal operating regime of the steady-state input–output map of a dynamical system without explicit knowledge of a model (Ariyur and Krstić, 2003, Zhang and Ordóñez, 2011). Two categories of extremum seeking controllers can be found in the literature. The first of which is continuous-time controllers which exploit dither/excitation signals to probe the local behaviour of the system to be optimised and continuously transition the system input to one that results in an optimum. See Ariyur and Krstić (2003), Krstić and Wang (2000) and Tan, Nešić, and Mareels (2006) for such methods that utilise periodic dithers and Liu and Krstić (2012), Manzie and Krstić (2009) for stochastic dithers. The convergence proofs of the former rely on averaging and singular perturbation techniques (Khalil, 2002, Teel et al., 2003), while the latter on stochastic averaging (Liu & Krstić, 2012). On the contrary, discrete-time extremum seeking controllers based on nonlinear programming methods are examined in Teel and Popović (2001) within a sampled-data framework. The convergence proof therein is established using Lyapunov arguments.

An alternative and more direct proof for convergence to an extremum in a sampled-data framework is given in Khong, Nešić, Tan, and Manzie (2013) using trajectory-based techniques. In the same paper, the sampled-data framework of extremum seeking is further examined to accommodate global nonconvex optimisation methods, such as those described in Strongin and Sergeyev (2000). These results demonstrate that a wide range of optimisation algorithms in the literature can be applied to extremum seeking of dynamic plants. Making use of the results in Khong, Nešić, Tan et al. (2013), deterministic gradient descent based extremum seeking control is reviewed in this paper. Furthermore, stochastic gradient descent (a.k.a. stochastic approximation) methods are accommodated for extremum seeking in a way that is robust against measurement errors.

Stochastic approximation methods (Kushner and Clark, 1978, Kushner and Yin, 2003, Spall, 2003) are a family of well-studied iterative gradient-based optimisation algorithms that find applications in a broad range of areas, such as adaptive control and neural networks (Bertsekas & Tsitsiklis, 1996). In contrast to the standard optimisation algorithms such as the steepest descent or Newton methods (Boyd & Vandenberghe, 2004) which exploit direct gradient information, stochastic approximation methods operate based on approximation to the gradient constructed from noisy measurements of the objective/cost function. For the former, knowledge of the underlying system input–output relationships are often needed to calculate the gradient using for example, the chain rule. This is not necessary for stochastic approximation, making it well-suited for non-model based extremum seeking control.

This paper adapts within a periodic sampled-data framework three discrete-time multivariate stochastic approximation algorithms for extremum seeking control of dynamical systems which can be of infinite dimension and contain general attractors. Namely, Kiefer–Wolfowitz–Blum’s Finite Difference Stochastic Approximation (FDSA) (Blum, 1954, Kiefer and Wolfowitz, 1952), Random Directions Stochastic Approximation (RDSA) (Kushner & Clark, 1978), and Simultaneous Perturbation Stochastic Approximation (SPSA) (Spall, 1992, Spall, 2003). It is shown that there exists a sufficiently long sampling period under which semi-global convergence with probability one to an extremum of the steady-state input–output relation can be achieved. This stands in comparison with the gradient descent method based extremum seeking control under ideal noise-free sample measurements, for which semi-global practical ultimately bounded asymptotic stability can be established. Note that the existence of Lyapunov functions satisfying the conditions in Teel and Popović (2001) is not known for the stochastic approximation methods, and hence the convergence results therein do not directly generalise to these methods.

A related work (Nusawardhana & Żak, 2004) considers an extremum seeking method based on the SPSA within a different setup (i.e. not sampled-data and has continuous plant output measurements). There, the steady-state input–output objective function is assumed to evaluate to a constant after some waiting time with respect to a constant input, and the output measurements are corrupted by noise. By contrast, this paper exploits the fact that the state trajectory of an asymptotically stable dynamical system converges to a neighbourhood of its steady-state value after the system’s input is held constant for a pre-selected waiting time. Furthermore, the sampled output value is assumed to be corrupted by measurement noise. The SPSA method has also been applied to optimisation of variable cam timing engine operation in Popović, Janković, Magner, and Teel (2006), alongside several other optimisation algorithms. Azuma, Sakar, and Pappas (2012) adapts the SPSA method for extreme source seeking of randomly switching static distribution fields using a nonholonomic mobile robot. On a different note, Stanković and Stipanović (2010) considers a related problem of extremum seeking of static functions under noisy measurements using a discrete-time controller with sinusoidal dither signals. These works differ from the setting of the paper, where stochastic approximation methods based extremum seeking of the steady-state input–output maps of dynamical systems is analysed within a sampled-data framework.

The paper has the following structure. First, the next section states the properties of the nonlinear dynamical systems to which gradient descent and stochastic optimisation methods are applied. Section  3 depicts the sampled-data framework in which extremum seeking control is analysed. Subsequently, Section  4 examines the gradient descent method for extremum seeking. Stochastic optimisation methods are considered in Section  5. Illustrative simulation examples are provided in Section  6, followed by some concluding remarks in Section  7.

Section snippets

Dynamical systems

The class of nonlinear, possibly infinite-dimensional, systems with general attractors considered in this paper is introduced in this section. A function γ:R0R0 is of class-K (denoted γK) if it is continuous, strictly increasing, and γ(0)=0. If γ is also unbounded, then γK. A continuous function β:R0×R0R0 is of class-KL if for each fixed t, β(,t)K and for each fixed s, β(s,) is decreasing to zero (Khalil, 2002). The Euclidean norm is denoted 2.

Let X be a Banach space whose norm

Sampled-data extremum seeking framework

The sampled-data extremum seeking framework of Khong, Nešić, Tan et al. (2013), Teel and Popović (2001) is detailed in this section. The gradient based methods in the succeeding sections can be applied to extremum seeking of dynamical systems defined in Section  2 within this framework.

Let {uk}k=0 be a sequence of vectors in Rm and define the zero-order hold (ZOH) operation u(t)ukfor all  t[kT,(k+1)T) and k=0,1,2,, where T>0 denotes the sampling period or waiting time. Furthermore, let the

Gradient descent method

This section adapts the gradient descent method for extremum seeking control within the sampled-data setting introduced in the previous section. The method may be considered as a special case under the unified framework proposed in the paper (Khong, Nešić, Tan et al., 2013), of which several results are utilised here.

Consider the following optimisation problem: yminuΩQ(u), where Q:RmR and ΩRm. Assuming that Q is differentiable, one of the most used methods in operations research is the

Stochastic approximation

This section adapts three stochastic approximation algorithms for extremum seeking control and establishes semi-global convergence with probability one. It is divided into three subsections, respectively dedicated to the Finite Difference Stochastic Approximation (FDSA), Random Directions Stochastic Approximation (RDSA), and Simultaneous Perturbation Stochastic Approximation (SPSA). Every subsection begins by reviewing the respective stochastic approximation (SA) algorithm. The review material

Simulation examples

Consider the following one-dimensional nonlinear system with a single input: ẋ=x3+u2,x(0)=2;y=x3. Note that for any fixed uR, x=u23 is a globally asymptotically stable equilibrium; see Section  2. It is apparent that the steady-state input–output map is Q(u)=u2 with uΩ(200,200), of which its unique global minimum is 0. The input is started at u=100 and the FDSA is employed for minimum-seeking of the plant using the sampled-data control law detailed in Section  3. The sampled output

Conclusions

This paper applies stochastic optimisation methods to extremum seeking control of possibly infinite-dimensional time-invariant nonlinear systems with noisy output measurements and establishes semi-global convergence results. These contrast the standard gradient descent method based extremum seeking control scheme under noise-free measurements, where semi-global practical asymptotic stability with respect to an ultimate bound can be shown. Future research directions involve investigating

Sei Zhen Khong received the Bachelor of Electrical Engineering degree (with honours) and the Ph.D. degree from The University of Melbourne, Australia, in 2008 and 2012, respectively. He held a postdoctoral research fellowship in the Department of Electrical and Electronic Engineering, The University of Melbourne and is currently a postdoctoral researcher in the Department of Automatic Control, Lund University, Sweden. His research interests include distributed analysis of heterogeneous

References (31)

  • J. Blum

    Multidimensional stochastic approximation methods

    The Annals of Mathematical Statistics

    (1954)
  • S. Boyd et al.

    Convex optimization

    (2004)
  • D.C. Chin

    Comparative study of stochastic algorithms for system optimization based on gradient approximations

    IEEE Transactions on Systems, Man and Cybernetics, Part B

    (1997)
  • C. Cruz-Hernández et al.

    Stability of discrete nonlinear systems under nonvanishing perturbations: application to a nonlinear model-matching problem

    IMA Journal of Mathematical Control and Information

    (1999)
  • H.K. Khalil

    Nonlinear systems

    (2002)
  • Cited by (32)

    • Combustion variation control strategy with thermal efficiency optimization for lean combustion in spark-ignition engines

      2019, Applied Energy
      Citation Excerpt :

      ES is regarded as a valid on-line optimization method, especially when limited information is known about the control plant. Its gradient-based iterative optimization approach has been proven effective and robust in determining the extremum of static maps and dynamic systems [34,35]. More specifically, ES is performed under the assumption that certain mappings exist between system inputs and outputs (e.g., linear relationships).

    • Stochastic Approximation in Online Steady State Optimization Under Noisy Measurements

      2017, Computer Aided Chemical Engineering
      Citation Excerpt :

      This is the case for the family of techniques known as Stochastic Approximations (SA). Despite the fact that important progress has been achieved in this field including applications in model-free approaches as extremum seeking control (Khong et al., 2015); there is a lack of literature regarding the potential applications of these methods in RTO schemes. This work attempts to explore the use of SA in online steady state optimization and proposes a framework that is particularly suitable for RTO under noise.

    View all citing articles on Scopus

    Sei Zhen Khong received the Bachelor of Electrical Engineering degree (with honours) and the Ph.D. degree from The University of Melbourne, Australia, in 2008 and 2012, respectively. He held a postdoctoral research fellowship in the Department of Electrical and Electronic Engineering, The University of Melbourne and is currently a postdoctoral researcher in the Department of Automatic Control, Lund University, Sweden. His research interests include distributed analysis of heterogeneous networks, robust control, linear systems theory, extremum seeking control, and sampled-data control.

    Ying Tan received her Bachelor from Tianjin University, China in 1995. In 1998, she joined the National University of Singapore and obtained her Ph.D. in 2002. She joined McMaster University in 2002 as a postdoctoral fellow in the Department of Chemical Engineering. She has started her work in the Department of Electrical and Electronic Engineering, The University of Melbourne since 2004. Currently she is Future Fellow (2010–2013), which is a research position funded by the Australian Research Council. Her research interests are in intelligent systems, nonlinear control systems, real time optimisation, sampled-data distributed parameter systems and formation control.

    Chris Manzie received the B.S. degree in Physics and the B.E. degree (with honours) in Electrical and Electronic Engineering and the Ph.D. degree from The University of Melbourne, Melbourne, Australia, in 1996 and 2001, respectively. Since 2003, he has been affiliated to the Department of Mechanical Engineering, The University of Melbourne, where he is currently an Associate Professor and an Australian Research Council Future Fellow. He was a Visiting Scholar with the University of California, San Diego in 2007, and a Visiteur Scientifique at IFP Energies Nouvelles, Paris in 2012. He has industry collaborations with companies including Ford Australia, BAE Systems, ANCA Motion and Virtual Sailing. His research interests lie in applications of model-based and extremum-seeking control in fields including mechatronics and energy systems. He is a member of the IEEE and IFAC Technical Committees on Automotive Control.

    Dragan Nešić is a Professor in the Department of Electrical and Electronic Engineering (DEEE) at The University of Melbourne, Australia. He received his B.E. degree in Mechanical Engineering from The University of Belgrade, Yugoslavia in 1990, and his Ph.D. degree from Systems Engineering, RSISE, Australian National University, Canberra, Australia in 1997. Since February 1999 he has been with The University of Melbourne. His research interests include networked control systems, discrete-time, sampled-data and continuous-time nonlinear control systems, input-to-state stability, extremum seeking control, applications of symbolic computation in control theory, hybrid control systems, and so on. He was awarded a Humboldt Research Fellowship (2003) by the Alexander von Humboldt Foundation, an Australian Professorial Fellowship (2004–2009) and Future Fellowship (2010–2014) by the Australian Research Council. He is a Fellow of IEEE and a Fellow of IEAust. He is currently a Distinguished Lecturer of CSS, IEEE (2008-). He served as an Associate Editor for the journals Automatica, IEEE Transactions on Automatic Control, Systems and Control Letters and European Journal of Control.

    This work was supported by the Swedish Research Council through the LCCC Linnaeus centre and the Australian Research Council (DP120101144). The material in this paper was presented at the 9th Asian Control Conference, June 23–26, 2013, Istanbul, Turkey. This paper was recommended for publication in revised form by Associate Editor Raul Ordóñez under the direction of Editor Miroslav Krstic.

    1

    Tel.: +46 462220362; fax: +46 46138118.

    View full text