Skip to main content
SpringerLink
Log in

An Overview of Unconstrained Optimization

  • Chapter
Algorithms for Continuous Optimization

Part of the book series: NATO ASI Series ((ASIC,volume 434))

Abstract

Developments in the theory and practice of unconstrained optimization are described. Both line search and trust region prototypes are explained and various algorithms based on the use of a quadratic model are outlined. The properties and implementation of the BFGS method are described in some detail and the current preference for this approach is discussed. Various conjugate gradient methods for large unstructured systems are given, but it is argued that limited memory methods are considerably more effective without a great increase in storage or time. Further developments in this area are described. For structured problems the possibilities for updates that retain sparsity are described, including a recent proposal which maintains positive definite matrices and reduces to the BFGS update in the dense case. The alternative use of structure in partially separable optimization is also discussed

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Al-Baali M. (1985) Descent property and global convergence of the Fletcher-Reeves method with inexact line search, J. Inst. Maths. Applns., 5, 121–124

    MathSciNet  MATH  Google Scholar 

  2. Allwright J.C. (1988) Positive semi-definite matrices: Characterization via conical hulls and least-squares solution of a matrix equation, SIAM J. Control Optim., 26, 537–556 (see also errata in SIAM J. Control Optim., 28, (1990), 250–251)

    Google Scholar 

  3. Barzilai J. and Borwein J.M. (1988) Two-point step size gradient methods, IMA J. Numer. Anal., 8, 141–148

    Article  MathSciNet  MATH  Google Scholar 

  4. Beale E.M.L. (1972) A derivation of conjugate gradients, in Numerical Methods for Nonlinear Optimization, ed. F.A.Lootsma, Academic Press, London

    Google Scholar 

  5. Broyden C.G. (1967) Quasi-Newton methods and their application to function minimization, Maths, of Computation, 21, 368–381

    Article  MathSciNet  MATH  Google Scholar 

  6. Buckley A. and LeNir A. (1983) QN-like variable storage conjugate gradients, Math. Programming, 27, 155–175

    Article  MathSciNet  MATH  Google Scholar 

  7. Byrd R.H., Liu D.C. and Nocedal J. (1990) On the behavior of Broyden’s class of quasi-Newton methods, Report NAM-01, Northwestern Univ., Dept. EECS

    Google Scholar 

  8. Byrd R.H. and Nocedal J. (1989) A tool for the analysis of quasi-Newton methods with application to unconstrained minimization, SIAM J. Numer. Anal., 26, 727–739

    Article  MathSciNet  MATH  Google Scholar 

  9. Byrd R.H., Nocedal J. and Yuan Y. (1987) Global convergence of a class of quasi-Newton methods on convex problems, SIAM J. Numer. Anal., 24, 1171–1190

    Article  MathSciNet  MATH  Google Scholar 

  10. Conn A.R., Gould N.I.M. and Toint Ph.L. (1988) Testing a class of methods for solving minimization problems with simple bounds on the variables, Maths, of Computation, 50, 399–430

    Article  MathSciNet  MATH  Google Scholar 

  11. Conn A.R., Gould N.I.M. and Toint Ph.L. (1991) Convergence of quasi-Newton matrices generated by the symmetric rank one update, Math. Programming, 50, 177–195

    Article  MathSciNet  MATH  Google Scholar 

  12. Conn A.R., Gould N.I.M. and Toint Ph.L. (1992) LANCELOT: a Fortran package for large-scale nonlinear optimization (Release A), Springer Series in Computational Mathematics 17, Springer-Verlag, Heidelberg

    MATH  Google Scholar 

  13. Crowder H.P. and Wolfe P. (1972) Linear convergence of the conjugate gradient method, IBM J. Res. and Dev., 16, 431–433

    Article  MathSciNet  MATH  Google Scholar 

  14. Davidon W.C. (1959) Variable metric methods for minimization, Argonne Nat. Lab. Report ANL-5990 (rev.), and in SIAM J. Optimization, 1, (1991), 1–17

    Article  MathSciNet  Google Scholar 

  15. Davidon W.C. (1975) Optimally conditioned optimization algorithms without line searches, Math. Programming, 9, 1–30

    Article  MathSciNet  MATH  Google Scholar 

  16. Dembo R.S., Eisenstat S.C. and Steihaug T. (1982) Inexact Newton methods, SIAM J. Numer. Anal., 19, 400–408

    Article  MathSciNet  MATH  Google Scholar 

  17. Dennis J.E. and Moré J.J. (1974) A characterization of superlinear convergence and its application to quasi-Newton methods, Maths, of Computation, 28, 549–560

    Article  MATH  Google Scholar 

  18. Dennis J.E. and Schnabel R.B. (1983) Numerical Methods for Unconstrained Optimization ana Nonlinear Equations, Prentice-Hall Inc., Englewood Cliffs, NJ

    Google Scholar 

  19. Dixon L.C.W. (1972) Quasi-Newton algorithms generate identical points, Math. Programming, 2, 383–387

    Article  MathSciNet  MATH  Google Scholar 

  20. Fletcher R. (1970) A new approach for variable metric algorithms, Computer J., 13, 317–322

    Article  MATH  Google Scholar 

  21. Fletcher R. (1987) Practical Methods of Optimization, (2nd. Edition), John Wiley, Chichester

    MATH  Google Scholar 

  22. Fletcher R. (1990) Low storage methods for unconstrained optimization, in Computational Solution of Nonlinear Systems of Equations, eds. E.L.Allgower and K.Georg, Lectures in Applied Mathematics Vol. 26, AMS Publications, Providence, RI

    Google Scholar 

  23. Fletcher R. (1991) A new result for quasi-Newton formulae, SIAM J. Optimization, 1, 18–21

    Article  MathSciNet  MATH  Google Scholar 

  24. Fletcher R. (1992) An optimal positive definite update for sparse Hessian matrices, Report NA/145, Univ. of Dundee, to appear in SIAM J. Optimization

    Google Scholar 

  25. Fletcher R. and Powell M.J.D. (1963) A rapidly convergent descent method for minimization, Computer J., 3, 163–168

    MathSciNet  Google Scholar 

  26. Fletcher R. and Reeves C.M. (1964) Function minimization by conjugate gradients, Computer J., 7, 149–154

    Article  MathSciNet  MATH  Google Scholar 

  27. Forsgren A., Gill P.E. and Murray W. (1993) Computing modified Newton directions using a partial Cholesky factorization, Royal Inst, of Tech., Sweden, Dept. of Math. Report TRITA/MAT-93–09

    Google Scholar 

  28. Gilbert J.C. and Nocedal J. (1990) Global convergence properties of conjugate gradient methods for optimization, Report 1268, INRIA

    Google Scholar 

  29. Gill P.E. and Murray W. (1972) Quasi-Newton methods for unconstrained optimization, J. Inst. Maths. Applns., 9, 91–108

    Article  MathSciNet  MATH  Google Scholar 

  30. Goldfarb D. (1970) A family of variable metric methods derived by variational means, Maths. of Computation, 24, 23–26

    Article  MathSciNet  MATH  Google Scholar 

  31. Goldfarb D. (1976) Factorized variable metric methods for unconstrained optimization, Maths. of Computation, 30, 796–811

    Article  MathSciNet  MATH  Google Scholar 

  32. Grandinetti (1979) Factorization versus nonfactorization in quasi-Newtonian algorithms for differentiable optimization, in Methods for Operations Research, Hain Verlay

    Google Scholar 

  33. Greenstadt J.L. (1970) Variations of variable metric methods, Maths. of Computation, 24, 1–22

    MathSciNet  MATH  Google Scholar 

  34. Griewank A. and Toint Ph.L. (1982) Partitioned variable metric updates for large structured optimization problems, Numerische Math., 39, 429–448

    Article  MathSciNet  MATH  Google Scholar 

  35. Grippo L., Lampariello F. and Lucidi S. (1990) A quasi-discrete Newton algorithm with a nonmonotone stabilization technique, J. Optim. Theo. Applns., 64, 485–500

    MathSciNet  Google Scholar 

  36. Hestenes M.R. and Stiefel E. (1952) Methods of conjugate gradients for solving linear systems, J. Res. NBS, 49, 409–436

    MathSciNet  MATH  Google Scholar 

  37. Khalfan H., Byrd R.H. and Schnabel R.B. (1990) A theoretical and experimental study of the symmetric rank one update, Report CU-Cs-489–90, Univ. of Colorado at Boulder, Dept. CS

    Google Scholar 

  38. Lalee M. and Nocedal J. (1991) Automatic scaling strategies for quasi-Newton updates, Report NAM-04, Northwestern Univ., Dept. EECS

    Google Scholar 

  39. Lefebure B. (1993) Investigating new methods for large scale unconstrained optimization. M.Sc. Thesis, Univ. of Dundee, Dept. of Maths, and CS

    Google Scholar 

  40. Liu D.C. and Nocedal J. (1989) On the limited memory BFGS method for large scale optimization, Math. Programming, 45, 503–528

    Article  MathSciNet  MATH  Google Scholar 

  41. Luksan L. (1993) Computational experience with known variable metric updates, Working paper

    Google Scholar 

  42. Moré J.J. (1983) Recent developments in algorithms and software for trust region methods, in Mathematical Programming, The State of the Art, eds. A.Bachem, M Grotschel and G.Korte, Springer-Verlag, Berlin

    Google Scholar 

  43. Moré J.J., Garbow B.S. and Hillstrom K.E. (1981) Testing unconstrained optimization software, ACM Trans. Math. Softw., 7, 17–41

    Article  MATH  Google Scholar 

  44. Moré J.J. and Sorensen D.C. (1979) On the use of directions of negative curvature in a modified Newton method, Math. Programming, 16, 1–20

    Article  MathSciNet  MATH  Google Scholar 

  45. Nash S.C. (1985) Preconditioning of truncated-Newton methods, SIAM J. S.S.C., 6, 599–618

    Article  MathSciNet  MATH  Google Scholar 

  46. Nash S.C. and Nocedal J. (1989) A numerical study of the limited memory BFGS method and the truncated-Newton method for large scale optimization, Report NAM-02, Northwestern Univ., Dept. EECS

    Google Scholar 

  47. Nocedal J. (1980) Updating quasi-Newton matrices with limited storage, Maths. of Computation, 35, 773–782

    Article  MathSciNet  MATH  Google Scholar 

  48. Nocedal J. and Yuan Y. (1991) Analysis of a self-scaling quasi-Newton method, Report NAM-02, Northwestern Univ., Dept. EECS

    Google Scholar 

  49. O’Leary D.P. (1982) A discrete Newton algorithm for minimizing a function of many variables, Math. Programming, 23, 20–33

    Article  MathSciNet  MATH  Google Scholar 

  50. Oren S.S. (1974) On the selection of parameters in self-scaling variable metric algorithms, Math. Programming, 7, 351–367

    Article  MathSciNet  MATH  Google Scholar 

  51. Oren S.S. and Luenberger D.G (1974) Self-scaling variable metric (SSVM) algorithms: I Criteria and sufficient conditions for scaling a class of algorithms, Management Sci., 20, 845–862

    Article  MathSciNet  MATH  Google Scholar 

  52. Oren S.S. and Spedicato E. (1976) Optimal conditioning of self-scaling variable metric algorithms, Math. Programming, 10, 70–90

    Article  MathSciNet  MATH  Google Scholar 

  53. Polak E. and Ribière G. (1969) Note sur la convergence de methodes de directions conjugées, Rev. Française Informat Recherche Opérationnelle, 3e Année, 16, 35–43

    Google Scholar 

  54. Powell M.J.D. (1970) A new algorithm for unconstrained optimization, in Nonlinear Programming, eds. J.B.Rosen, O.L.Mangasarian and K.Ritter, Academic Press, New York

    Google Scholar 

  55. Powell M.J.D. (1971) On the convergence of the variable metric algorithm, J. Inst. Maths. Applns., 7, 21–36

    Article  MATH  Google Scholar 

  56. Powell M.J.D. (1976) Some global convergence properties of a variable metric algorithm for minimization without exact line searches, in Nonlinear Programming, SIAM-AMS Proceedings Vol. IX, eds. R.W.Cottle and C. E. Lemke, SIAM Publications, Philadelphia

    Google Scholar 

  57. Powell M.J.D. (1977) Restart procedures for the conjugate gradient method, Math. Programming, 12, 241–254

    Article  MathSciNet  MATH  Google Scholar 

  58. Powell M.J.D. (1984) Nonconvex minimization calculations and the conjugate gradient method, in Numerical Analysis, Dundee 1983, ed. D.F.Griffiths, Lecture Notes in Mathematics 1066, Springer Verlag, Berlin

    Google Scholar 

  59. Powell M.J.D. (1986) How bad are the BFGS and DFP methods when the objective function is quadratic?, Math. Programming, 34, 34–47

    Article  MathSciNet  MATH  Google Scholar 

  60. Powell M.J.D. (1987) Updating conjugate directions by the BFGS formula, Math. Programming, 38, 29–46

    Article  MathSciNet  MATH  Google Scholar 

  61. Powell M.J.D. and Toint Ph. L. (1979) On the estimation of sparse Hessian matrices, SIAM J. Numer. Anal., 16, 1060–1074

    Article  MathSciNet  MATH  Google Scholar 

  62. Schnabel R.B. (1983) Quasi-Newton methods using multiple secant equations, Report CU-CS-247–83, Univ. of Colorado at Boulder, Dept. CS

    Google Scholar 

  63. Schnabel R.B. and Eskow E. (1990) A new modified Cholesky factorization, SIAM J. Sci. Stat. Comp., 11, 1136–1158

    Article  MathSciNet  MATH  Google Scholar 

  64. Shanno D.F. (1970) Conditioning of quasi-Newton methods for function minimization, Maths. of Computation, 24, 647–656

    Article  MathSciNet  Google Scholar 

  65. Shanno D.F. and Phua K.H. (1980) Remark on algorithm 500, minimization of unconstrained multivariate functions, ACM Trans. Math. Softw., 6, 618–622

    Article  Google Scholar 

  66. Siegel D. (1991) Updating of conjugate direction matrices using members of Broyden’s family, Rep. DAMTP 1991/NA4, Univ. of Cambridge, Dept. AMTP

    Google Scholar 

  67. Siegel D. (1991) Modifying the BFGS update by a new column scaling technique, Report DAMTP 1991/NA5, Univ. of Cambridge, Dept. AMTP

    Google Scholar 

  68. Siegel D. (1992) Implementing and modifying Broyden class updates for large scale optimization, Report DAMTP 1992/NA12, Univ. of Cambridge, Dept. AMTP

    Google Scholar 

  69. Sorensen D.C. (1982) Collinear scaling and sequential estimation in sparse optimization algorithms, Math. Programming Study, 18, 135–159

    MathSciNet  MATH  Google Scholar 

  70. Steihaug T. (1983) The conjugate gradient method and trust regions in large scale optimization, SIAM J. Numer. Anal., 20, 626–637

    Article  MathSciNet  MATH  Google Scholar 

  71. Toint Ph. L. (1977) On sparse and symmetric updating subject to a linear equation. Maths, of Computation, 31, 954–961

    Article  MathSciNet  MATH  Google Scholar 

  72. Toint Ph. L. (1981) A note on sparsity exploiting quasi-Newton methods. Math. Programming, 21, 172–181

    Article  MathSciNet  MATH  Google Scholar 

  73. Toint Ph. L. (1981) Towards an efficient sparsity exploiting Newton method for minimization, in Sparse Matrices and Their Uses, ed. I.S.Duff, Academic Press, London

    Google Scholar 

  74. Torczon V. (1991) On the convergence of the multidimensional search algorithm, SIAM J. Optimization, 1, 123–145

    Article  MathSciNet  MATH  Google Scholar 

  75. Wood G.R. (1992) The bisection method in higher dimensions, Math. Programming, 55, 319–338

    Article  MathSciNet  MATH  Google Scholar 

  76. Zhang Y. and Tewarson R.P. (1988) Quasi-Newton algorithms with updates from the pre-convex part of Broyden’s family, IMA J. Numer. Anal., 8, 487–509

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dept. of Mathematics and Computer Science, University of undee, Dundee, DDI 4HN, Scotland, UK

    R. Fletcher

Authors
  1. R. Fletcher
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Mathematics, University of Bergamo, Bergamo, Italy

    Emilio Spedicato

Rights and permissions

Reprints and permissions

Copyright information

© 1994 Kluwer Academic Publishers

About this chapter

Cite this chapter

Fletcher, R. (1994). An Overview of Unconstrained Optimization. In: Spedicato, E. (eds) Algorithms for Continuous Optimization. NATO ASI Series, vol 434. Springer, Dordrecht. https://doi.org/10.1007/978-94-009-0369-2_5

Download citation

  • DOI: https://doi.org/10.1007/978-94-009-0369-2_5

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-010-6652-5

  • Online ISBN: 978-94-009-0369-2

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation