Non-asymptotic model quality assessment of transfer functions at multiple frequency points☆
Introduction
In system identification, providing a description of the uncertainties associated with the nominal system model is as important as obtaining the nominal model itself, especially for the synthesis of robust controllers. A popular technique for evaluating the model quality is based on constructing confidence regions using asymptotic system identification theory. This is a mature approach and the confidence regions can be computed relatively easily (see Ljung, 1999). However, in some cases using asymptotic theory may lead to unreliable results (see Garatti, Campi, & Bittanti, 2004) when applied to a finite number of data points.
In this paper, we consider a method for constructing confidence regions based on finitely many data points as, e.g., considered in Bayard (1993), Campi and Weyer (2005), den Dekker, Bombois, and Van den Hof (2008), Goodwin, Gevers, and Ninnes (1992) and Hjalmarsson and Ninness (2006). Unlike methods based on asymptotic theory, the developed method generates guaranteed confidence regions for a finite number of data points. The developed approach is based on the LSCR method introduced in Campi and Weyer (2005) (see also Campi, Ko, & Weyer, 2009 and Campi & Weyer, 2010), and it is extended to produce guaranteed confidence regions for the frequency response of a dynamical system. As a finite number of data points does not provide any information about the tail of the impulse response, prior information, such as exponentially decaying bounds, is introduced and incorporated in the algorithm in order to deal with tail effects. Moreover, an experimental scheme is derived that allows the confidence regions to be constructed separately frequency by frequency. This reduces the computational burden significantly.
In the next subsection we give simple preview examples that illustrate the main ideas of the proposed approach. Then, in Section 2, the procedure used in the preview examples is generalized to construct simultaneous confidence regions when the system is excited by a multi-sine input signal. In Section 3 an experimental scheme and an algorithm that allow the confidence regions to be constructed at low computational costs are introduced. Two simulation examples demonstrating the usefulness of the proposed approach are given in Section 4.
In this section we first introduce a simple example illustrating the main ideas of LSCR by generating a confidence interval for the amplitude of a sinusoid, before moving on to the construction of a confidence set for the frequency response of a dynamical system at a given frequency. For further descriptions of the main ideas in the LSCR algorithm, the reader is referred to Campi and Weyer (2006) and Section 1.2 of Campi et al. (2009).
The signal of interest is a sinusoid observed in noise We have observations , . is a sequence of zero mean independent random variables, symmetrically distributed about zero. The frequency is known, but the amplitude is unknown. The observed signal is shown in Fig. 1. We wish to construct a confidence interval for . Given the signal model we compute the observation error and correlate it with , which gives We note that for , and is different from zero for . The idea is now to use random subsamples of to form empirical estimates of the correlation between the observation error and . To this end we compute empirical subsample estimates where are independent and identically distributed (i.i.d.) random variables taking on the values 0 and 1 with probability 1/2 each. The exception is which is equal to zero for all , and hence . This means that determines whether sample is used when is computed.
The non-zero functions are shown in Fig. 2. Corresponding to the true amplitude , is a sum of zero mean random variables. It is therefore unlikely that nearly all of the functions are positive or negative for , and hence we exclude those values of where all the functions take on positive or negative values. Thus, the confidence interval marked with a thick line in Fig. 2 is obtained by keeping those values of for at which at least of the functions are positive and at least are negative. It is shown in Theorem 1 that the constructed confidence interval contains the true amplitude () with probability .
Next we move onto a more realistic situation where also the phase is unknown and transient effects need to be taken into account.
Suppose that the true continuous-time system is given by where is the impulse response function, and is additive noise. The transfer function of the system (1) is the Laplace transform of given by and in this example it is given by This information about the true system is given for completeness of description but is unknown to the user.
The input to the system is a sinusoid The output is given by where , and represents the transient effects due to initial conditions.
Our task is to construct a confidence region for and , the frequency response parameters at 1 rad/s based on a finite number of data points obtained by sampling the input and output at time instants for with . We assume that the sampled noise , , is a sequence of independent (but not necessarily identically distributed) random variables with symmetric distributions around zero and all admit densities.
One way to estimate the frequency response parameters and to construct a confidence region is to measure the output once the transients have died out. In order to avoid the transient phase of the response we wait samples before starting the measurements of the output (see Fig. 3). We take 315 samples of the input and output. This corresponds to approximately one cycle of the input signal.
From finite-length input and output data we cannot obtain full information about the frequency response of the system since the data do not carry any information about the tail of the impulse response. The only way we can bound the uncertainty due to the tail is via a priori knowledge and assumptions. Here we assume that a bound on the impulse response is available, i.e., parameters and are known such that For this example, we use the following prior information which is shown in Fig. 4. Using this information, we can bound the unknown value as follows We can compute the predictions of the output and the prediction error for as follows where , and denotes the parameter vector. Using random subsamples of the data set we calculate the following scaled empirical correlation functions and between the prediction error and sines and cosines of the same frequency as the input signal where for and are i.i.d. with distribution and are independent of the noise sequence . The first string is given by for . Note that, using (6), we have and, hence, at the true parameter, , for all we obtain Since is zero-mean, it is unlikely that nearly all of the sums , , take on positive values or that nearly all of them take on negative values, hence, it is unlikely that nearly all take on negative values or that nearly all take on positive values, and the same holds for and . Based on this observation, in order to construct a confidence region we discard those regions in the parameter space where or take on negative value too many times and also the regions where or take on positive value too many times, and hence the name of the algorithm “Leave-out Sign-dominant Correlation Regions”.
Therefore, in order to find a confidence set, we excluded the regions in the parameter space where less than empirical correlation function satisfies either , , or . The obtained confidence set is shown as the blank area in Fig. 5, and according to Theorem 2 it contains the true parameters with probability at least . In the figure, the region where at most 9 of the functions were negative is marked with , and the region where at most 9 of the were positive is marked with . Likewise, and represents the regions where at most 9 values of and were negative and positive, respectively. As we can see, each correlation excludes a particular region of the parameter space. Note that in order to construct the confidence region we have not made any assumptions on the noise other than it should be symmetrically distributed around 0. Still the algorithm constructs a confidence region with guaranteed probability with a finite number of data points.
Section snippets
Main algorithm
Here we extend the approach in the preview example to a multi sine input signal.
Computational aspects
Using the procedure in the previous section, we can construct non-asymptotic confidence regions for the frequency response at multiple frequencies. However, each of the empirical correlation functions (20) depends on the whole set of parameters , and thus the resulting confidence regions and are not only dependent on and , but also on all other parameters.
In this section we develop an experiment procedure and a method for the generation of decoupling binary strings
Simulation example
In this section, we present two simulation examples to illustrate the procedures for constructing confidence regions developed in the previous sections. We consider the same first-order system as described by (1), (3) in the preview example in Section 1.1 and construct simultaneous confidence regions for a two-frequency and a ten-frequency case.
Conclusion
In this paper, we have extended the LSCR algorithm introduced in Campi and Weyer (2005) to the problem of constructing confidence regions for the frequency response at multiple frequencies using a finite number of input–output data points. No information about the tail of the impulse response can be obtained from a finite number of data points, and hence a priori information has been used to bound the effects of the tail. Three theorems have been established describing the probabilistic
Acknowledgments
The research of S. Ko and E. Weyer was partly supported by the Australian Research Council under the Discovery Grant Scheme, Project DOP0558579 and the National Research Foundation of Korea (NRF) grant funded by the Korean Government (MSIP) (NRF-2013M1A3A3A02042434). The research of M.C. Campi was supported by MIUR under the project “Identification and Adaptive Control for Industrial Systems”.
Sangho Ko is an associate professor at the School of Aerospace and Mechanical Engineering, Korea Aerospace University, South Korea. He received his Ph.D. degree from the Department of Mechanical and Aerospace Engineering of the University of California, San Diego (UCSD) in 2005. From 2005 to 2006, he was a post-doctoral scholar in UCSD to research system identification techniques. From March 2006 to February 2008 he was a research fellow in the Department of Electrical and Electronic
References (17)
Statistical plant set estimation using Schroeder-phased multisinusoidal input design
Applied Mathematics and Computation
(1993)- et al.
Guaranteed non-asymptotic confidence regions in system identification
Automatica
(2005) - et al.
Identification with finitely many data points: the LSCR approach
- et al.
Non-asymptotic confidence regions for model parameters in the presence of unmodelled dynamics
Automatica
(2009) - et al.
Quantification of uncertainty in transfer function estimation: a mixed probabilistic-worst-case approach
Automatica
(1995) - et al.
Assessing the quality of identified models through the asymptotic theory - When is the result reliable?
Automatica
(2004) - et al.
Least-squares estimation of a class of frequency functions: a finite sample variance expression
Automatica
(2006) - et al.
Non-asymptotic confidence sets for the parameters of linear transfer functions
IEEE Transactions on Automatic Control
(2010)
Cited by (3)
Non-asymptotic Confidence Regions for the Transfer Functions of Errors-in-Variables Systems
2022, IEEE Transactions on Automatic ControlMoving data window based multi-innovation identification stochastic gradient parameter identification method for transfer functions
2017, Kongzhi yu Juece/Control and Decision
Sangho Ko is an associate professor at the School of Aerospace and Mechanical Engineering, Korea Aerospace University, South Korea. He received his Ph.D. degree from the Department of Mechanical and Aerospace Engineering of the University of California, San Diego (UCSD) in 2005. From 2005 to 2006, he was a post-doctoral scholar in UCSD to research system identification techniques. From March 2006 to February 2008 he was a research fellow in the Department of Electrical and Electronic Engineering of the University of Melbourne, Australia, where he worked on finite sample quality assessment of system identification. From 1992 to 1999, he was with Samsung Aerospace Industries, Ltd., Kyungnam, Korea, where he was a research and development engineer for digital flight control systems. His research interests include: state estimation and control, system identification, flight control, propulsion system control.
Erik Weyer is a professor in the Department of Electrical and Electronic Engineering at the University of Melbourne. He received the Siv. Ing. degree in 1988 and the Ph.D. in 1993, both from the Norwegian Institute of Technology, Trondheim, Norway. From 1994 to 1996 he was a Research Fellow at the University of Queensland, and since 1997 he has been with the Department of Electrical and Electronic Engineering at the University of Melbourne. He has held visiting positions at the University of Brescia, Italy, the Technical University of Vienna, Austria, and Politecnico di Milano, Italy. From 2010 to 2012 he was an associate editor of IEEE Transactions of Automatic Control, and he is currently an associate editor of Automatica. His research interests are in the areas of system identification and control, with particular emphasis on finite sample properties of system identification methods, and modeling and control of irrigation channels and rivers. He was a co-recipient of the IEEE CSS Control System Technology Award in 2014.
Marco Claudio Campi is Professor of Automatic Control at the University of Brescia, Italy.
In 1988, he received the Doctor degree in electronic engineering from the Politecnico di Milano, Milano, Italy. From 1988 to 1989, he was a Lecturer at the Department of Electrical Engineering of the Politecnico di Milano. From 1989 to 1992, he was a Research Fellow at the Centro di Teoria dei Sistemi of the National Research Council (CNR) in Milano and, in 1992, he joined the University of Brescia, Brescia, Italy. He has held visiting and teaching appointments at the Australian National University, Canberra, Australia; the University of Illinois at Urbana-Champaign, USA; the Centre for Artificial Intelligence and Robotics, Bangalore, India; the University of Melbourne, Australia; the Kyoto University, Japan.
Marco Campi is the chair of the Technical Committee IFAC on Modeling, Identification and Signal Processing (MISP). He has been in various capacities on the Editorial Board of Automatica, Systems and Control Letters and the European Journal of Control. Marco Campi is a recipient of the “Giorgio Quazza” prize, and, in 2008, he received the IEEE CSS George S. Axelby outstanding paper award for the article The Scenario Approach to Robust Control Design. He has delivered plenary and semi-plenary addresses at major conferences including SYSID, MTNS, and CDC, and has been a distinguished lecturer of the Control Systems Society. Marco Campi is a Fellow of IEEE, a member of IFAC, and a member of SIDRA.
The research interests of Marco Campi include: system identification, stochastic systems, randomized methods, adaptive and data-based control, robust optimization, and learning theory.
- ☆
The material in this paper was partially presented at the 2007 International Conference on Control, Automation and Systems, October 17–20, 2007, Seoul, Korea and at the 17th IFAC World Congress, July 6–11, 2008, Seoul, Korea. This paper was recommended for publication in revised form by Associate Editor Alessandro Chiuso under the direction of Editor Torsten Söderström.
- 1
Tel.: +82 2 300 0119; fax: +82 2 3158 2191.