Van Luong Le
ABSTRACT
This paper deals with the identification of hybrid systems switching between nonlinear subsystems of unknown structure and focuses on the connections with a family of machine learning algorithms known as support vector machines. In particular, we consider a recent approach to nonlinear hybrid system identification based on a convex relaxation of a sparse optimization problem. In this approach, the submodels are iteratively estimated one by one by maximizing the sparsity of the corresponding error vector. We extend this approach in several ways. First, we relax the sparsity condition by introducing robust sparsity, which can be optimized through the minimization of a modified l1-norm or, equivalently, of the ε-insensitive loss function. Then, we show that, depending on the choice of regularizer, the method is equivalent to different forms of support vector regression. More precisely, the submodels can be estimated by iteratively solving a classical support vector regression problem, in which the sparsity of support vectors relates to the sparsity of the error vector in the considered hybrid system identification framework. This allows us to extend theoretical results as well as efficient optimization algorithms from the field of machine learning to the hybrid system framework.
- L. Bako. Identification of switched linear systems via sparse optimization. Automatica, 47(4):668--677, 2011. Google Scholar
Digital Library
- L. Bako, K. Boukharouba, and S. Lecoeuche. An l_0-l_1 norm based optimization procedure for the identification of switched nonlinear systems. In Proc. of the 49th IEEE Int. Conf. on Decision and Control (CDC), pages 4467--4472, 2010.Google ScholarCross Ref
- L. Bako, V. L. Le, F. Lauer, and G. Bloch. Identification of MIMO switched state-space models. In Proc. of the American Control Conference (ACC), Washington, DC, USA, 2013.Google ScholarCross Ref
- A. Bemporad, A. Garulli, S. Paoletti, and A. Vicino. A bounded-error approach to piecewise affine system identification. IEEE Transactions on Automatic Control, 50(10):1567--1580, 2005.Google ScholarCross Ref
- A. Berlinet and C. Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic Publishers, Boston, 2004.Google ScholarCross Ref
- L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, editors. Large Scale Kernel Machines. MIT Press, Cambridge, MA, 2007. Google ScholarDigital Library
- E. Candès, M. Wakin, and S. Boyd. Enhancing sparsity by reweighted l_1 minimization. Journal of Fourier Analysis and Applications, 14(5):877--905, 2008.Google ScholarCross Ref
- C. Chang and C. Lin. LibSVM: a library for support vector machines, 2001. Available at http://www.csie.ntu.edu.tw/$\sim$cjlin/libsvm/. Google ScholarDigital Library
- G. Ferrari-Trecate, M. Muselli, D. Liberati, and M. Morari. A clustering technique for the identification of piecewise affine systems. Automatica, 39(2):205--217, 2003. Google ScholarDigital Library
- L. Gunter and J. Zhu. Computing the solution path for the regularized support vector regression. In Advances in Neural Information Processing Systems 18, pages 483--490, 2006.Google Scholar
- T. Hastie, S. Rosset, R. Tibshirani, and J. Zhu. The entire regularization path for the support vector machine. Journal of Machine Learning Research, 5:1391--1415, 2004. Google ScholarDigital Library
- T. Joachims. SVM light, 1998. Available at http://svmlight.joachims.org/.Google Scholar
- A. L. Juloski, S. Weiland, and W. Heemels. A Bayesian approach to identification of hybrid systems. IEEE Transactions on Automatic Control, 50(10):1520--1533, 2005.Google ScholarCross Ref
- J. Kwok and I. Tsang. Linear dependency between $\varepsilon$ and the input noise in ε-support vector regression. IEEE Transactions on Neural Networks, 14(3):544--553, 2003. Google ScholarDigital Library
- F. Lauer. Estimating the probability of success of a simple algorithm for switched linear regression. Nonlinear Analysis: Hybrid Systems, 8:31--47, 2013. Supplementary material available at http://www.loria.fr/ lauer/klinreg/.Google ScholarCross Ref
- F. Lauer and G. Bloch. Switched and piecewise nonlinear hybrid system identification. In Proc. of the 11th Int. Conf. on Hybrid Systems: Computation and Control (HSCC), volume 4981 of LNCS, pages 330--343, 2008. Google ScholarDigital Library
- F. Lauer, G. Bloch, and R. Vidal. A continuous optimization framework for hybrid system identification. Automatica, 47(3):608--613, 2011. Google ScholarDigital Library
- F. Lauer, V. L. Le, and G. Bloch. Learning smooth models of nonsmooth functions via convex optimization. In Proc. of the IEEE Int. Workshop on Machine Learning for Signal Processing (MLSP), Santander, Spain, 2012.Google ScholarCross Ref
- V. L. Le, G. Bloch, and F. Lauer. Reduced-size kernel models for nonlinear hybrid system identification. IEEE Transactions on Neural Networks, 22(12):2398--2405, 2011. Google ScholarDigital Library
- G. Loosli, G. Gasso, and S. Canu. Regularization paths for ν-SVM and ν-SVR. In Advances in Neural Networks -- ISNN, volume 4493 of LNCS, pages 486--496, 2007. Google ScholarDigital Library
- O. Mangasarian and D. Musicant. Large scale kernel regression via linear programming. Machine Learning, 46(1):255--269, 2002. Google ScholarDigital Library
- H. Ohlsson and L. Ljung. Identification of piecewise affine systems using sum-of-norms regularization. In Proc. of the 18th IFAC World Congress, pages 6640--6645, 2011.Google ScholarCross Ref
- N. Ozay, M. Sznaier, C. Lagoa, and O. Camps. A sparsification approach to set membership identification of switched affine systems. IEEE Transactions on Automatic Control, 57(3):634--648, 2012.Google ScholarCross Ref
- S. Paoletti, A. L. Juloski, G. Ferrari-Trecate, and R. Vidal. Identification of hybrid systems: a tutorial. European Journal of Control, 13(2--3):242--262, 2007.Google ScholarCross Ref
- J. Platt. Fast training of support vector machines using sequential minimal optimization. In B. Schölkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods: Support Vector Learning, pages 185--208. MIT Press, 1999. Google ScholarDigital Library
- J. Roll, A. Bemporad, and L. Ljung. Identification of piecewise affine systems via mixed-integer programming. Automatica, 40(1):37--50, 2004. Google ScholarDigital Library
- B. Schölkopf, R. Herbrich, and A. Smola. A generalized representer theorem. In Proc. of COLT/EuroCOLT, volume 2111 of LNAI, pages 416--426, 2001. Google ScholarDigital Library
- B. Schölkopf, A. Smola, R. Williamson, and P. Bartlett. New support vector algorithms. Neural computation, 12(5):1207--1245, 2000. Google ScholarDigital Library
- S. Shevade, S. Keerthi, C. Bhattacharyya, and K. Murthy. Improvements to the SMO algorithm for SVM regression. IEEE Transactions on Neural Networks, 11(5):1188--1193, 2000. Google ScholarDigital Library
- A. Smola, N. Murata, B. Schölkopf, and K. Müller. Asymptotically optimal choice of ε-loss for support vector machines. In Proc. of the 8th Int. Conf. on Artificial Neural Networks (ICANN), pages 105--110, 1998.Google ScholarCross Ref
- A. Smola, B. Schölkopf, and G. Ratsch. Linear programs for automatic accuracy control in regression. In Proc. of the 9th Int. Conf. on Artificial Neural Networks (ICANN), pages 575--580, 1999.Google ScholarCross Ref
- A. J. Smola and B. Schölkopf. A tutorial on support vector regression. Statistics and Computing, 14(3):199--222, 2004. Google ScholarDigital Library
- I. Steinwart and A. Christmann. Support Vector Machines. Springer, 2008. Google ScholarDigital Library
- F. Tay and L. Cao. Modified support vector machines in financial time series forecasting. Neurocomputing, 48(1):847--861, 2002.Google ScholarCross Ref
- V. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, 1995. Google ScholarDigital Library
- R. Vidal, S. Soatto, Y. Ma, and S. Sastry. An algebraic geometric approach to the identification of a class of linear hybrid systems. In Proc. of the 42nd IEEE Conf. on Decision and Control (CDC), pages 167--172, 2003.Google ScholarCross Ref
- G. Wang, D. Yeung, and F. Lochovsky. Two-dimensional solution path for support vector regression. In Proc. of the 23rd Int. Conf. on Machine Learning (ICML), pages 993--1000, 2006. Google ScholarDigital Library
Index Terms
- Learning nonlinear hybrid systems: from sparse optimization to support vector regression
Recommendations
Sparse discriminant twin support vector machine for binary classification
AbstractFor a binary classification problem, twin support vector machine (TSVM) has a faster learning speed than support vector machine (SVM) by seeking a pair of nonparallel hyperplanes. However, TSVM has two deficiencies: poor discriminant ability and ...
Support vector machine classifier with truncated pinball loss
A new loss function and corresponding support vector machine are proposed.The model can handle feature noise.The model keeps sparsity to a certain extent.The problem is solved by concave-convex procedure and decomposition method. Feature noise, namely ...
Manifold-preserving graph reduction for sparse semi-supervised learning
Representing manifolds using fewer examples has the advantages of eliminating the influence of outliers and noisy points and simultaneously accelerating the evaluation of predictors learned from the manifolds. In this paper, we give the definition of ...
Comments