Skip to main content
SpringerLink
Log in

Sparse Linear Least Squares Problems in Optimization

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Numerical and computational aspects of direct methods for largeand sparseleast squares problems are considered. After a brief survey of the most oftenused methods, we summarize the important conclusions made from anumerical comparison in matlab. Significantly improved algorithms haveduring the last 10-15 years made sparse QR factorization attractive, andcompetitive to previously recommended alternatives. Of particular importanceis the multifrontal approach, characterized by low fill-in, dense subproblemsand naturally implemented parallelism. We describe a Householder multifrontalscheme and its implementation on sequential and parallel computers. Availablesoftware has in practice a great influence on the choice of numericalalgorithms. Less appropriate algorithms are thus often used solely because ofexisting software packages. We briefly survey softwarepackages for the solution of sparse linear least squares problems. Finally,we focus on various applications from optimization, leading to the solution oflarge and sparse linear least squares problems. In particular, we concentrateon the important case where the coefficient matrix is a fixed general sparsematrix with a variable diagonal matrix below. Inner point methods forconstrained linear least squares problems give, for example, rise to suchsubproblems. Important gains can be made by taking advantage of structure.Closely related is also the choice of numerical method for these subproblems.We discuss why the less accurate normal equations tend to be sufficient inmany applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. M. Arioli, I. S. Duff, and P. de Rijk, On the augmented system approach to sparse least-squares problems, Numer. Math., 55 (1989), pp. 667–684.

    Google Scholar 

  2. Å. Björck, Stability analysis of the method of semi-normal equations for least squares problems, Linear Algebra Appl., 88/89 (1987), pp. 31–48.

    Google Scholar 

  3. Å. Björck, A direct method for sparse least squares problems with lower and upper bounds, Numer. Math., 54 (1988), pp. 19–32.

    Google Scholar 

  4. Å. Björck, Pivoting and stability in the augmented system method, Technical Report LiTH-MAT-R-1991-30, Department of Mathematics, Linköping University, June 1991.

  5. Å. Björck, Numerical Methods for Least Squares Problems, SIAM, Philadelphia, 1996.

  6. J. R. Bunch and L. Kaufman, Some stable methods for calculating inertia and solving symmetric linear systems, Mathematics of Computation, 31 (1977), pp. 162–179.

    Google Scholar 

  7. E. C. H. Chu, J. A. George, J. Liu, and E. Ng, SPARSPAK: Waterloo sparse matrix package user's guide for SPARSPAK-A, Research Report CS-84-36, Dept. of Computer Science, University of Waterloo, 1984.

  8. T. F. Coleman, A. Edenbrandt, and J. R. Gilbert, Predicting fill for sparse orthogonal factorization, J. ACM, 33 (1986), pp. 517–532.

    Google Scholar 

  9. J. Dennis and R. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall, Englewood Cliffs, N.J., 1983.

    Google Scholar 

  10. I. S. Duff, Pivot selection and row orderings in Givens reduction on sparse matrices, Computing, 13 (1974), pp. 239–248.

    Google Scholar 

  11. I. S. Duff, N. I. M. Gould, J. K. Reid, J. A. Scott, and K. Turner, The factorization of sparse symmetric indefinite matrices, IMA J. Numer. Anal., 11 (1991), pp. 181–204.

    Google Scholar 

  12. I. S. Duff, R. G. Grimes, and J. G. Lewis, Sparse matrix test problems, ACMTrans. Math. Softw., 15 (1989), pp. 1–14.

    Google Scholar 

  13. I. S. Duff and J. K. Reid, A comparison of some methods for the solution of sparse overdetermined systems of linear equations, J. Inst. Maths. Applics., 17 (1976), pp. 267–280.

    Google Scholar 

  14. I. S. Duff and J. K. Reid, The multifrontal solution of indefinite sparse symmetric linear systems, ACM Trans. Math. Software, 9 (1983), pp. 302–325.

    Google Scholar 

  15. S. C. Eisenstat, M. H. Schultz, and A. H. Sherman, Algorithms and data structures for sparse symmetric Gaussian elimination, SIAM J. Sci. Statist. Comput., 2 (1981), pp. 225–237.

    Google Scholar 

  16. L. EldÉn, Algorithms for the regularization of ill-conditioned least squares problems, BIT, 17 (1977), pp. 134–145.

    Google Scholar 

  17. L. V. Foster, Modifications of the normal equations method that are numerically stable, in Numerical Linear Algebra, Digital Signal Processing and Parallel Algorithms, G. H. Golub and </del> P. V. Dooren, eds., NATO ASI Series, Berlin, 1991, Springer-Verlag, pp. 501–512.

  18. C. F. Gauss, Theory of the Motion of the Heavenly Bodies Moving about the Sun in Conic Sections, Dover, New York (1963), 1809. C. H. Davis, Trans.

  19. W. M. Gentleman, Basic procedures for large, sparse, or weighted linear least squares problems, Research report CSRR 2068, University of Waterloo, Waterloo, Ontario, Canada, July 1972.

    Google Scholar 

  20. W. M. Gentleman, Least squares computations by Givens transformations without square roots, J. Inst. Maths. Applics., 12 (1973), pp. 329–336.

    Google Scholar 

  21. W. M. Gentleman, Error analysis of QR decompositions by Givens transformations, Linear Algebra Appl., 10 (1975), pp. 189–197.

    Google Scholar 

  22. W. M. Gentleman, Row elimination for solving sparse linear systems and least squares problems, in Proceedings the 6th Dundee Conference on Numerical Analysis, G. A. Watson, ed., Springer Verlag, 1976, pp. 122–133.

  23. J. A. George and M. T. Heath, Solution of sparse linear least squares problems using Givens rotations, Linear Algebra Appl., 34 (1980), pp. 69–83.

    Google Scholar 

  24. J. A. George and J. W.-H. Liu, Householder reflections versus Givens rotations in sparse orthogonal decomposition, Linear Algebra Appl., 88/89 (1987), pp. 223–238.

    Google Scholar 

  25. J. A. George and E. G. Ng, On row and column orderings for sparse least squares problems, SIAM J. Numer. Anal., 20 (1981), pp. 326–344.

    Google Scholar 

  26. J. A. George and E. G. Ng, SPARSPAK:Waterloo sparse matrix package user's guide for SPARSPAKB, Research Report CS-84-37, Dept. of Computer Science, University of Waterloo, 1984.

  27. J. R. Gilbert, C. Moler, and R. Schreiber, Sparse matrices in MATLAB: Design and implementation, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 333–356.

    Google Scholar 

  28. P. E. Gill and W. Murray, Nonlinear least squares and nonlinearly constrained Optimization, in In Proceedings Dundee Conference on Numerical Analysis 1975, Lecture Notes in Mathematics No. 506, Springer Verlag, 1976.

  29. P. E. Gill and W. Murray, The orthogonal factorization of a large sparse matrix, in Sparse Matrix Computations, J. R. Bunch and D. J. Rose, eds., New York, 1976, Academic Press, pp. 201–212.

  30. P. E. Gill, W. Murray, and M. H. Wright, Practical Optimization, Academic Press, London and New York, 1981.

    Google Scholar 

  31. G. H. Golub, Numerical methods for solving least squares problems, Numer. Math., 7 (1965), pp. 206–216.

    Google Scholar 

  32. C. C. Gonzaga, Path-following methods for linear programming, SIAM Review, 34 (1992), pp. 167–224.

    Google Scholar 

  33. M. T. Heath, Numerical methods for large sparse linear least squares problems, SIAM J. Sci. Statist. Comput.,, 5 (1984), pp. 497–513.

    Google Scholar 

  34. G. B. Kolata, Geodesy: Dealing with an enormous computer task, Science, 200 (1978), pp. 421–422.

    Google Scholar 

  35. C. L. Lawson and R. J. Hanson, Solving Least Squares Problems, Prentice Hall, Englewood Cliffs, New Jersey, 1974.

    Google Scholar 

  36. A. M. Legendre, Nouvelle méthodes pour la détermination des orbites des comètes, Courcier, Paris, 1805.

  37. K. Levenberg, A method for the solution of certain non-linear problems in least squares, Quart. Appl. Math., 2 (1944), pp. 164–168.

    Google Scholar 

  38. J. W.-H. Liu, On general row merging schemes for sparse Givens transformations, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 1190–1211.

    Google Scholar 

  39. J. W.-H. Liu, The role of elimination trees in sparse factorization, SIAM J. Matrix Anal. Appl., 11 (1990), pp. 134–172.

    Google Scholar 

  40. S. Lu and J. L. Barlow, Multifrontal computation with the orthogonal factors of sparse matrices, SIAM J. Matr. Anal. 3(1996), pp. 658–679.

    Google Scholar 

  41. I. Lustig, R. Marsten, and D. Shanno, Computational experience with a primal-dual interior point method for linear programming, Linear Algebra Appl., (1991), pp. 191–222.

  42. D. W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters, Journal of the Society for Industrial and Applied Mathematics, 11 (1963), pp. 431–441.

    Google Scholar 

  43. P. Matstoms, The multifrontal solution of sparse linear least squares problems, Licentiat thesis, Linköping University, 1991.

  44. P. Matstoms, QR27-Specification sheet, Tech. Report March 1992, Department of Mathematics, 1992.

  45. P. Matstoms, SparseQRfactorization inMATLAB, ACMTrans. Math. Software, 20 (1994), pp. 136–159.

    Google Scholar 

  46. P. Matstoms, Sparse QR Factorization with Applications to Linear Least Squares Problems, PhD thesis, Linköping University, 1994.

  47. P. Matstoms, Parallel sparse QR factorization on shared memory architectures, Parallel Computing, 21 (1995), pp. 473–486.

    Google Scholar 

  48. J. J. MorÉ, The Levenberg-Marquardt algorithm: Implementation and theory, in G.A. Watson, Lecture Notes in Math. 630, Berlin, 1978, Springer Verlag, pp. 105–116.

  49. U. Oreborn, A Direct Method for Sparse Nonnegative Least Squares Problems, licentiat thesis, Linköping University, 1986.

  50. C. C. Paige and M. A. Saunders, LSQR. an algorithm for sparse linear equations and sparse least squares, ACM Trans. Math. Software, 8 (1982), pp. 43–71.

    Google Scholar 

  51. D. J. Pierce and J. G. Lewis, Sparse multifrontal rank revealing QR factorization, Technical Report MEA-TR-193-Revised, Boeing Information and Support Services, 1995.

  52. L. F. Portugal, J. J. JÚdice, and L. N. Vicente, Solution of large scale linear least-squares problems with nonnegativ variables, technical report, Departamento de ciêcias da terra, Universidade de Coimbra, 3000 Coimbra, Portugal, 1993.

    Google Scholar 

  53. C. Puglisi, QR factorization of large sparse overdetermined and square matrices with the multifrontal method in a multiprocessor environment, PhD thesis, CERFACS, 42 av. G. Coriolis, 31057 Toulouse Cedex, France.

  54. J. K. Reid, A note on the least squares solution of a band system of linear equations by Householder reductions, Comput J., 10 (1967), pp. 188–189.

    Google Scholar 

  55. C. Sun, Parallel sparse orthogonal factorization on distributed-memory multiprocessors, SIAM J. Sci. Comput., 17 (1996), p. to appear.

  56. M. H. Wright, Interior methods for constrained Optimization, Acta Numerica 1992, Cambridge University Press, 1992, pp. 341–407.

  57. Z. Zlatev and H. Nielsen, LLSS01-a Fortran subroutine for solving least squares problems ( User's guide ), Technical Report 79-07, Institute of Numerical Analysis, Technical University of Denmark, Lyngby, Denmark, 1979.

    Google Scholar 

  58. Z. Zlatev and H. Nielsen, Solving large and sparse linear least-squares problems by conjugate gradient algorithms, Comput. Math. Applic., 15 (1988), pp. 185–202.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Linköping University, S-581 82, Linköping, Sweden

    Pontus Matstoms

Authors
  1. Pontus Matstoms
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Matstoms, P. Sparse Linear Least Squares Problems in Optimization. Computational Optimization and Applications 7, 89–110 (1997). https://doi.org/10.1023/A:1008680131271

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008680131271

Search

Navigation