Abstract
Least Squares Support Vector Machines (LS-SVM) is aproven method for classification and function approximation. In comparison to the standard Support Vector Machines (SVM) it only requires solving a linear system, but it lacks sparseness in the number of solution terms. Pruning can therefore be applied. Standard ways of pruning the LS-SVM consist of recursively solving the approximation problem and subsequently omitting data that have a small error in the previous pass and are based on support values. We suggest a slightly adapted variant that improves the performance significantly. We assess the relative regression performance of these pruning schemes in a comparison with two (for pruning adapted) subset selection schemes, -one based on the QR decomposition (supervised), one that searches the most representative feature vector span (unsupervised)-, random omission and backward selection on independent test sets in some benchmark experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Vapnik, V.N.: Statistical Learning Theory. John Wiley & Sons, Chichester (1998)
Suykens, J.A.K., Vandewalle, J.: Least squares support vector machine classifiers. Neural Processing Letters 9(3), 293–300 (1999)
Suykens, J.A.K., De Brabanter, J., Lukas, L., Vandewalle, J.: Weighted least squares support vector machines: Robustness and sparse approximation. Neurocomputing, Special issue on fundamental and information processing aspects of neurocomputing 48(1-4), 85–105 (2002)
de Kruif, B.J., de Vries, T.J.A.: Pruning error minimization in least squares support vector machines. IEEE Transactions on Neural Networks 14(3), 696–702 (2003)
Suykens, J.A.K., Van Gestel, T., De Brabanter, J., DeMoor, B., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific, Singapore (2002)
Hoegaerts, L., Suykens, J.A.K., Vandewalle, J., De Moor, B.: Subset based least squares subspace regression in RKHS. Accepted for publication in Neurocomputing, Special issue (2004)
Reed, R.: Pruning algorithms – A survey. IEEE Transactions on Neural Networks 4(5), 740–747 (1993)
Smola, A.J., Schölkopf, B.: Sparse greedy matrix approximation for machine learning. In: Proc. 17th International Conf. on Machine Learning, pp. 911–918. Morgan Kaufmann, San Francisco (2000)
Miller, A.J.: Subset Selection in Regression. Chapman & Hall, Boca Raton (1990)
Chen, S., Cowan, C., Grant, P.: Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans. Neural Networks 2, 302–309 (1991)
Baudat, G., Anouar, F.: Kernel based methods and function approximation. In: IJCN, Washington DC, July 2001, pp. 1244–1249 (2001)
Aronszajn, N.: Theory of reproducing kernels. Transactions of the American mathematical society 686, 337–404 (1950)
Schölkopf, B., Smola, A.: Learning with kernels. MIT Press, Cambridge (2002)
Hassibi, B., Stork, D.G., Wolf, G.: Optimal brain surgeon and general network pruning. In: Proceedings of the 1993 IEEE International Conference on Neural Networks, San Francisco, CA, April 1993, pp. 293–300 (1993)
Blake, C., Merz, C.: UCI repository of machine learning databases (1998), Online Available, http://www.ics.uci.edu/~mlearn/MLRepository.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hoegaerts, L., Suykens, J.A.K., Vandewalle, J., De Moor, B. (2004). A Comparison of Pruning Algorithms for Sparse Least Squares Support Vector Machines. In: Pal, N.R., Kasabov, N., Mudi, R.K., Pal, S., Parui, S.K. (eds) Neural Information Processing. ICONIP 2004. Lecture Notes in Computer Science, vol 3316. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30499-9_194
Download citation
DOI: https://doi.org/10.1007/978-3-540-30499-9_194
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23931-4
Online ISBN: 978-3-540-30499-9
eBook Packages: Springer Book Archive