Skip to main content
SpringerLink
Log in

An algorithm for solving sparse Nonlinear Least Squares problems

Ein Algorithmus zur Lösung dünnbesetzter nichtlinearer Ausgleichsprobleme

  • Contributed Papers
  • Published:
Computing Aims and scope Submit manuscript

Abstract

We introduce a new method for solving Nonlinear Least Squares problems when the Jacobian matrix of the system is large and sparse.

The main features of the new method are the following:

  1. a)

    The Gauss-Newton equation is “partially” solved at each iteration using a preconditioned Conjugate Gradient algorithm.

  2. b)

    The new point is obtained using a two-dimensional trust region scheme, similar to the one introduced by Bulteau and Vial.

We prove global and local convergence results and we present some numerical experiments.

Zusammenfassung

Eine neue Methode zur Lösung nichtlinearer Least-Squares-Probleme bei hochdimensionaler dünnbesetzter Jakobimatrix wird vorgestellt.

Die wichtigsten Charakteristika sind:

  1. a)

    Die Gauß-Newton-Gleichung wird „teilweise” bei jeder Iteration gelöst, wobei eine präkonditionierte konjugierte Gradientenmethode verwendet wird.

  2. b)

    Die neue Lösung wird über eine zweidimensionale trust-region Technik erreicht, ähnlich der von Bulteau und Vial vorgeschlagenen Variante.

Globale und lokale Konvergenzaussagen werden bewiesen und anhand einiger numerischer Beispiele demonstriert.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aziz, M. A., Jennings, A.: A robust incomplete Cholesky-conjugate gradient algorithm. Internat. J. Numer. Methods Engin.20, 949–966 (1984).

    Google Scholar 

  2. Axelsson, O.: Incomplete block matrix factorizations preconditioning methods: The ultimate answer? J. Comput. Appl. Math.12/13, 3–18 (1985).

    Google Scholar 

  3. Axelsson, O., Brinkkemper, S., Il'in, V. P.: On some versions of incomplete block-matrix factorizations iterative methods. Linear Alg. Applics.58, 3–15 (1984).

    Google Scholar 

  4. Axelsson, O., Lindskog, G.: On the eigenvalue distribution of a class of preconditioning methods. Numer. Math.48, 479–498 (1986).

    Google Scholar 

  5. Axelsson, O., Lindskog, G.: On the rate of convergence of the preconditioned conjugate gradient method. Numer. Math.48, 499–524 (1986).

    Google Scholar 

  6. Bishop, T. N. et al.: Tomographic determination of velocity and depth in laterally varying media. Geophysics50, 903–923 (1985).

    Google Scholar 

  7. Boggs, P. T.: The convergence of the Ben Israel iteration for nonlinear least squares problems. Math. Comput.30, 512–522 (1976).

    Google Scholar 

  8. Bulteau, J. P., Vial, J. P.: A restricted trust region algorithm for unconstrained optimization. JOTA47, 413–435 (1985).

    Google Scholar 

  9. Coleman, T. F.: Large sparse numerical optimization. Lectures Notes in Computer Science No. 165. Springer-Verlag, 1984.

  10. Dembo, R. S., Eisenstat, S. C., Steihaug, T.: Inexact Newton methods. SIAM J. of Numer. Anal.19, 400–408 (1982).

    Google Scholar 

  11. Dembo, R. S., Steihaug, T.: Truncated Newton methods. Math. Programming26, 190–212 (1983).

    Google Scholar 

  12. Dennis, J. E.: Some computational techniques for the nonlinear least-squares problems. In: Numerical Solution of Systems of Nonlinear Algebraic Equations (Byrne, G. D., Hall, C. A., eds.), pp. 157–183. New York: Academic Press 1973.

    Google Scholar 

  13. Dennis, J. E., Gay, D. M., Welsch, R. E.: An Adaptive Nonlinear Least-Squares Algorithm. Report TR 77-321. Department of Computer Sciences, Cornell University, 1977.

  14. Dennis, J. E., Marwil, E. S.: Direct secant updates of matrix factorizations. Math. Comput.38, 459–476 (1982).

    Google Scholar 

  15. Deuflhard, P., Apostolescu, V.: A study of the Gauss-Newton method for the solution of nonlinear least squares problems. In: Special Topics of Applied Mathematics (Frehse/Pallaschke/Trottenberg, eds.), pp. 129–150. North-Holland Publ. Co., 1980.

  16. Dennis, J., Schnabel, R.: Numerical methods for unconstrained optimization and nonlinear equations. Prentice-Hall Series in Comp. Math. N. J.: Prentice Hall 1983.

    Google Scholar 

  17. Dongarra, J. J., Leaf, G. K., Minkoff, M.: A Preconditioned Conjugate Gradient Method for Solving a Class of Non-Symmetric Linear Systems ANL-81-71. Argonne, Il. Argonne National Laboratory 1981.

    Google Scholar 

  18. Fletcher, R.: A Modified Marquardt Subroutine for Nonlinear Least Squares. Harwell: AERE Report 6799, 1971.

  19. George, A., Heath, M. T.: Solution of sparse linear least squares using Givens rotations. Linear Algebra and Applics.34, 69–83 (1980).

    Google Scholar 

  20. George, A., Heath, M. T., Ng, E.: A comparison of some methods for solving sparse least-squares problems. SIAM J. Sci. St. Comp.4, 177–187 (1983).

    Google Scholar 

  21. Gill, P., Murray, W., Wright, M. H.: Practical Optimization. N. Y.: Academic Press 1981.

    Google Scholar 

  22. Gill, P., Murray, W., Saunders, M., Wright, M.: Sparse methods in optimization. SIAM J. Sci. St. Comp.5, 562–589 (1984).

    Google Scholar 

  23. Harwell Subroutine Library: Harwell: Subroutine NS03A, 1971.

  24. Heath, M. T.: Numerical methods for large sparse linear least squares problems. SIAM J. Sci. St. Comp.5, 497–513 (1984).

    Google Scholar 

  25. Hestenes, M. R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Nat. Bur. Standards49, 409–436 (1952).

    Google Scholar 

  26. Hiebert, K. L.: An evaluation of mathematical software that solves nonlinear least squares problems. ACM TOMS7, 1–16 (1981).

    Google Scholar 

  27. Le, D.: A fast and robust unconstrained optimization method requiring minimum storage. Math. Prog.32, 41–48 (1985).

    Google Scholar 

  28. Leite, L. W. B., Leão, J. W. D.: Ridge regression applied to the inversion of two-dimensional aeromagnetic anomalies. Geophysics50, 1294–1306 (1985).

    Google Scholar 

  29. Levenberg, K.: A method for the solution of certain nonlinear problems in least squares. Quarterly J. of Appl. Math..2, 164–168 (1944).

    Google Scholar 

  30. Luenberger, D. G.: Linear and Nonlinear Programming (2nd edition). Addison Wesley 1984.

  31. Manteuffel, T. A.: Shifted incomplete factorization. In: Sparse Matrix Proceedings. (Duff, I., Stewart, G., eds.), Philadelphia: SIAM 1979.

    Google Scholar 

  32. Marquardt, D. W.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math.11, 431–441 (1963).

    Google Scholar 

  33. Martínez, J. M.: A quasi-Newton method with a new updating for the LDU factorization of the approximate Jacobian. Mat. Aplic. e Comput.2, 131–142 (1983).

    Google Scholar 

  34. Martínez, J. M.: A quasi-Newton method with modification of one column per iteration. Computing33, 353–362 (1984).

    Google Scholar 

  35. Martínez, J. M.: Solving systems of nonlinear equations by means of an accelerated successive orthogonal projections method. J. Comput. Appl. Math.16, 169–179 (1986).

    Google Scholar 

  36. Martínez, J. M., Sampaio, R. J. B.: Parallel and sequential Kaczmarz methods for solving underdetermined nonlinear equations. J. Comput. Appl. Math.15, 311–321 (1986).

    Google Scholar 

  37. Menzel, R.: On solving nonlinear least-squares problems in case of Rank deficient Jacobians. Computing34, 63–72 (1985).

    Google Scholar 

  38. Meyn, K. H.: Solution of underdetermined nonlinear equations by stationary iteration methods. Numer. Math.42, 161–172 (1983).

    Google Scholar 

  39. Moré, J. J.: The Levenberg-Marquardt algorithm: Implementation and theory. Proc. Biennial Conf. Num. An., Dundee 1977 (Watson, G. A., ed.). Lecture Notes in Math. Springer-Verlag (1978).

  40. Moré, J. J.: Recent developments in algorithms and software for trust region methods. TR ANL/MCS-TM-2. Argonne: Argonne National Laboratory 1982.

    Google Scholar 

  41. Moré, J. J., Garbow, B. S., Hillstrom, K. E.: Testing unconstrained minimization software. ACM TOMS7, 17–41 (1981).

    Google Scholar 

  42. Nazareth, L.: Some recent approaches to solving large residual nonlinear least-squares problems. SIAM Review22, 1–11 (1980).

    Google Scholar 

  43. Ortega, J. M., Rheinboldt, W. C.: Iterative Solution of Nonlinear Equations in Several Variables. New York: Academic Press 1970.

    Google Scholar 

  44. Paige, C. C., Saunders, M. A., LSQR: An algorithm for sparse linear equations and sparse least squares. ACM TOMS8, 43–71 (1982).

    Google Scholar 

  45. Pedersen, L. B.: Interpretation of potential field data. A generalized inverse approach. Geophys. Prosp.25, 199–230 (1977).

    Google Scholar 

  46. Saunders, M. A.: Sparse least squares by conjugate gradients: A comparison of preconditioning methods. Proc. Computer Science and Statistics: 12th Annual Symposium on the Interface (Gentleman, J. F., ed.), pp. 15–20. Ontario: Univ. of Waterloo 1979.

    Google Scholar 

  47. Schabak, R.: Convergence analysis of the general Gauss-Newton algorithm. Numer. Math.46, 281–309 (1985).

    Google Scholar 

  48. Schubert, L. K.: Modification of a quasi-Newton method for nonlinear equations with a sparse Jacobian. Math. Comput.24, 27–30 (1970).

    Google Scholar 

  49. Schwetlick, H.: Numerische Lösung nichtlinearer Gleichungen. Berlin: Deutscher Verlag der Wissenschaften 1978.

    Google Scholar 

  50. Shanno, D. F.: Globally convergent conjugate gradient algorithms. Math. Prog.33, 61–67 (1985).

    Google Scholar 

  51. Steihaug, T.: The conjugate gradient method and trust regions in large scale optimization. SIAM J. Num. Anal.20, 626–637 (1983).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Applied Mathematics Laboratory, IMECC-UNICAMP, CP. 6065-13081, Campinas, S.P., Brazil

    J. M. Martínez

Authors
  1. J. M. Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Martínez, J.M. An algorithm for solving sparse Nonlinear Least Squares problems. Computing 39, 307–325 (1987). https://doi.org/10.1007/BF02239974

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02239974

Keywords

Search

Navigation