Brief paperExtremum seeking of dynamical systems via gradient descent and stochastic approximation methods☆
Introduction
Extremum seeking locates via online computations an optimal operating regime of the steady-state input–output map of a dynamical system without explicit knowledge of a model (Ariyur and Krstić, 2003, Zhang and Ordóñez, 2011). Two categories of extremum seeking controllers can be found in the literature. The first of which is continuous-time controllers which exploit dither/excitation signals to probe the local behaviour of the system to be optimised and continuously transition the system input to one that results in an optimum. See Ariyur and Krstić (2003), Krstić and Wang (2000) and Tan, Nešić, and Mareels (2006) for such methods that utilise periodic dithers and Liu and Krstić (2012), Manzie and Krstić (2009) for stochastic dithers. The convergence proofs of the former rely on averaging and singular perturbation techniques (Khalil, 2002, Teel et al., 2003), while the latter on stochastic averaging (Liu & Krstić, 2012). On the contrary, discrete-time extremum seeking controllers based on nonlinear programming methods are examined in Teel and Popović (2001) within a sampled-data framework. The convergence proof therein is established using Lyapunov arguments.
An alternative and more direct proof for convergence to an extremum in a sampled-data framework is given in Khong, Nešić, Tan, and Manzie (2013) using trajectory-based techniques. In the same paper, the sampled-data framework of extremum seeking is further examined to accommodate global nonconvex optimisation methods, such as those described in Strongin and Sergeyev (2000). These results demonstrate that a wide range of optimisation algorithms in the literature can be applied to extremum seeking of dynamic plants. Making use of the results in Khong, Nešić, Tan et al. (2013), deterministic gradient descent based extremum seeking control is reviewed in this paper. Furthermore, stochastic gradient descent (a.k.a. stochastic approximation) methods are accommodated for extremum seeking in a way that is robust against measurement errors.
Stochastic approximation methods (Kushner and Clark, 1978, Kushner and Yin, 2003, Spall, 2003) are a family of well-studied iterative gradient-based optimisation algorithms that find applications in a broad range of areas, such as adaptive control and neural networks (Bertsekas & Tsitsiklis, 1996). In contrast to the standard optimisation algorithms such as the steepest descent or Newton methods (Boyd & Vandenberghe, 2004) which exploit direct gradient information, stochastic approximation methods operate based on approximation to the gradient constructed from noisy measurements of the objective/cost function. For the former, knowledge of the underlying system input–output relationships are often needed to calculate the gradient using for example, the chain rule. This is not necessary for stochastic approximation, making it well-suited for non-model based extremum seeking control.
This paper adapts within a periodic sampled-data framework three discrete-time multivariate stochastic approximation algorithms for extremum seeking control of dynamical systems which can be of infinite dimension and contain general attractors. Namely, Kiefer–Wolfowitz–Blum’s Finite Difference Stochastic Approximation (FDSA) (Blum, 1954, Kiefer and Wolfowitz, 1952), Random Directions Stochastic Approximation (RDSA) (Kushner & Clark, 1978), and Simultaneous Perturbation Stochastic Approximation (SPSA) (Spall, 1992, Spall, 2003). It is shown that there exists a sufficiently long sampling period under which semi-global convergence with probability one to an extremum of the steady-state input–output relation can be achieved. This stands in comparison with the gradient descent method based extremum seeking control under ideal noise-free sample measurements, for which semi-global practical ultimately bounded asymptotic stability can be established. Note that the existence of Lyapunov functions satisfying the conditions in Teel and Popović (2001) is not known for the stochastic approximation methods, and hence the convergence results therein do not directly generalise to these methods.
A related work (Nusawardhana & Żak, 2004) considers an extremum seeking method based on the SPSA within a different setup (i.e. not sampled-data and has continuous plant output measurements). There, the steady-state input–output objective function is assumed to evaluate to a constant after some waiting time with respect to a constant input, and the output measurements are corrupted by noise. By contrast, this paper exploits the fact that the state trajectory of an asymptotically stable dynamical system converges to a neighbourhood of its steady-state value after the system’s input is held constant for a pre-selected waiting time. Furthermore, the sampled output value is assumed to be corrupted by measurement noise. The SPSA method has also been applied to optimisation of variable cam timing engine operation in Popović, Janković, Magner, and Teel (2006), alongside several other optimisation algorithms. Azuma, Sakar, and Pappas (2012) adapts the SPSA method for extreme source seeking of randomly switching static distribution fields using a nonholonomic mobile robot. On a different note, Stanković and Stipanović (2010) considers a related problem of extremum seeking of static functions under noisy measurements using a discrete-time controller with sinusoidal dither signals. These works differ from the setting of the paper, where stochastic approximation methods based extremum seeking of the steady-state input–output maps of dynamical systems is analysed within a sampled-data framework.
The paper has the following structure. First, the next section states the properties of the nonlinear dynamical systems to which gradient descent and stochastic optimisation methods are applied. Section 3 depicts the sampled-data framework in which extremum seeking control is analysed. Subsequently, Section 4 examines the gradient descent method for extremum seeking. Stochastic optimisation methods are considered in Section 5. Illustrative simulation examples are provided in Section 6, followed by some concluding remarks in Section 7.
Section snippets
Dynamical systems
The class of nonlinear, possibly infinite-dimensional, systems with general attractors considered in this paper is introduced in this section. A function is of class- (denoted ) if it is continuous, strictly increasing, and . If is also unbounded, then . A continuous function is of class- if for each fixed , and for each fixed , is decreasing to zero (Khalil, 2002). The Euclidean norm is denoted .
Let be a Banach space whose norm
Sampled-data extremum seeking framework
The sampled-data extremum seeking framework of Khong, Nešić, Tan et al. (2013), Teel and Popović (2001) is detailed in this section. The gradient based methods in the succeeding sections can be applied to extremum seeking of dynamical systems defined in Section 2 within this framework.
Let be a sequence of vectors in and define the zero-order hold (ZOH) operation and , where denotes the sampling period or waiting time. Furthermore, let the
Gradient descent method
This section adapts the gradient descent method for extremum seeking control within the sampled-data setting introduced in the previous section. The method may be considered as a special case under the unified framework proposed in the paper (Khong, Nešić, Tan et al., 2013), of which several results are utilised here.
Consider the following optimisation problem: where and . Assuming that is differentiable, one of the most used methods in operations research is the
Stochastic approximation
This section adapts three stochastic approximation algorithms for extremum seeking control and establishes semi-global convergence with probability one. It is divided into three subsections, respectively dedicated to the Finite Difference Stochastic Approximation (FDSA), Random Directions Stochastic Approximation (RDSA), and Simultaneous Perturbation Stochastic Approximation (SPSA). Every subsection begins by reviewing the respective stochastic approximation (SA) algorithm. The review material
Simulation examples
Consider the following one-dimensional nonlinear system with a single input: Note that for any fixed , is a globally asymptotically stable equilibrium; see Section 2. It is apparent that the steady-state input–output map is with , of which its unique global minimum is 0. The input is started at and the FDSA is employed for minimum-seeking of the plant using the sampled-data control law detailed in Section 3. The sampled output
Conclusions
This paper applies stochastic optimisation methods to extremum seeking control of possibly infinite-dimensional time-invariant nonlinear systems with noisy output measurements and establishes semi-global convergence results. These contrast the standard gradient descent method based extremum seeking control scheme under noise-free measurements, where semi-global practical asymptotic stability with respect to an ultimate bound can be shown. Future research directions involve investigating
Sei Zhen Khong received the Bachelor of Electrical Engineering degree (with honours) and the Ph.D. degree from The University of Melbourne, Australia, in 2008 and 2012, respectively. He held a postdoctoral research fellowship in the Department of Electrical and Electronic Engineering, The University of Melbourne and is currently a postdoctoral researcher in the Department of Automatic Control, Lund University, Sweden. His research interests include distributed analysis of heterogeneous
References (31)
- et al.
Multidimensional global extremum seeking via the DIRECT optimisation algorithms
Automatica
(2013) - et al.
Unified frameworks for sampled-data extremum seeking control: global optimisation and multi-unit systems
Automatica
(2013) - et al.
Stability of extremum seeking feedback for general nonlinear dynamic systems
Automatica
(2000) - et al.
A non-gradient approach to global extremum seeking: an adaptation of the Shubert algorithm
Automatica
(2013) - et al.
Sufficient conditions for stabilization of sampled-data nonlinear systems via discrete-time approximations
Systems & Control Letters
(1999) - et al.
Extremum seeking under stochastic noise and applications to mobile sensors
Automatica
(2010) - et al.
On non-local stability properties of extremum seeking controllers
Automatica
(2006) - et al.
Real-time optimization by extremum seeking control
(2003) - et al.
Stochastic source seeking by mobile robots
IEEE Transactions on Automatic Control
(2012) - et al.
Neuro-dynamic programming
(1996)
Multidimensional stochastic approximation methods
The Annals of Mathematical Statistics
Convex optimization
Comparative study of stochastic algorithms for system optimization based on gradient approximations
IEEE Transactions on Systems, Man and Cybernetics, Part B
Stability of discrete nonlinear systems under nonvanishing perturbations: application to a nonlinear model-matching problem
IMA Journal of Mathematical Control and Information
Nonlinear systems
Cited by (32)
Nonsmooth optimization by Lie bracket approximations into random directions
2023, Systems and Control LettersAutomatic crossbow control in industrial hot-dip galvanizing lines
2023, Journal of Process ControlCombustion variation control strategy with thermal efficiency optimization for lean combustion in spark-ignition engines
2019, Applied EnergyCitation Excerpt :ES is regarded as a valid on-line optimization method, especially when limited information is known about the control plant. Its gradient-based iterative optimization approach has been proven effective and robust in determining the extremum of static maps and dynamic systems [34,35]. More specifically, ES is performed under the assumption that certain mappings exist between system inputs and outputs (e.g., linear relationships).
Stochastic Approximation in Online Steady State Optimization Under Noisy Measurements
2017, Computer Aided Chemical EngineeringCitation Excerpt :This is the case for the family of techniques known as Stochastic Approximations (SA). Despite the fact that important progress has been achieved in this field including applications in model-free approaches as extremum seeking control (Khong et al., 2015); there is a lack of literature regarding the potential applications of these methods in RTO schemes. This work attempts to explore the use of SA in online steady state optimization and proposes a framework that is particularly suitable for RTO under noise.
Almost sure convergence of extremum seeking algorithm using stochastic perturbation
2016, Systems and Control LettersMULTI-POINT SEARCH BASED OPTIMIZATION ALGORITHM AND APPLICATION IN SYSTEM IDENTIFICATION
2023, ICIC Express Letters
Sei Zhen Khong received the Bachelor of Electrical Engineering degree (with honours) and the Ph.D. degree from The University of Melbourne, Australia, in 2008 and 2012, respectively. He held a postdoctoral research fellowship in the Department of Electrical and Electronic Engineering, The University of Melbourne and is currently a postdoctoral researcher in the Department of Automatic Control, Lund University, Sweden. His research interests include distributed analysis of heterogeneous networks, robust control, linear systems theory, extremum seeking control, and sampled-data control.
Ying Tan received her Bachelor from Tianjin University, China in 1995. In 1998, she joined the National University of Singapore and obtained her Ph.D. in 2002. She joined McMaster University in 2002 as a postdoctoral fellow in the Department of Chemical Engineering. She has started her work in the Department of Electrical and Electronic Engineering, The University of Melbourne since 2004. Currently she is Future Fellow (2010–2013), which is a research position funded by the Australian Research Council. Her research interests are in intelligent systems, nonlinear control systems, real time optimisation, sampled-data distributed parameter systems and formation control.
Chris Manzie received the B.S. degree in Physics and the B.E. degree (with honours) in Electrical and Electronic Engineering and the Ph.D. degree from The University of Melbourne, Melbourne, Australia, in 1996 and 2001, respectively. Since 2003, he has been affiliated to the Department of Mechanical Engineering, The University of Melbourne, where he is currently an Associate Professor and an Australian Research Council Future Fellow. He was a Visiting Scholar with the University of California, San Diego in 2007, and a Visiteur Scientifique at IFP Energies Nouvelles, Paris in 2012. He has industry collaborations with companies including Ford Australia, BAE Systems, ANCA Motion and Virtual Sailing. His research interests lie in applications of model-based and extremum-seeking control in fields including mechatronics and energy systems. He is a member of the IEEE and IFAC Technical Committees on Automotive Control.
Dragan Nešić is a Professor in the Department of Electrical and Electronic Engineering (DEEE) at The University of Melbourne, Australia. He received his B.E. degree in Mechanical Engineering from The University of Belgrade, Yugoslavia in 1990, and his Ph.D. degree from Systems Engineering, RSISE, Australian National University, Canberra, Australia in 1997. Since February 1999 he has been with The University of Melbourne. His research interests include networked control systems, discrete-time, sampled-data and continuous-time nonlinear control systems, input-to-state stability, extremum seeking control, applications of symbolic computation in control theory, hybrid control systems, and so on. He was awarded a Humboldt Research Fellowship (2003) by the Alexander von Humboldt Foundation, an Australian Professorial Fellowship (2004–2009) and Future Fellowship (2010–2014) by the Australian Research Council. He is a Fellow of IEEE and a Fellow of IEAust. He is currently a Distinguished Lecturer of CSS, IEEE (2008-). He served as an Associate Editor for the journals Automatica, IEEE Transactions on Automatic Control, Systems and Control Letters and European Journal of Control.
- ☆
This work was supported by the Swedish Research Council through the LCCC Linnaeus centre and the Australian Research Council (DP120101144). The material in this paper was presented at the 9th Asian Control Conference, June 23–26, 2013, Istanbul, Turkey. This paper was recommended for publication in revised form by Associate Editor Raul Ordóñez under the direction of Editor Miroslav Krstic.
- 1
Tel.: +46 462220362; fax: +46 46138118.