Abstract
Motivated by the superlinear behavior of the Barzilai-Borwein (BB) method for two-dimensional quadratics, we propose two gradient methods which adaptively choose a small step-size or a large step-size at each iteration. The small step-size is primarily used to induce a favorable descent direction for the next iteration, while the large step-size is primarily used to produce a sufficient reduction. Although the new algorithms are still linearly convergent in the quadratic case, numerical experiments on some typical test problems indicate that they compare favorably with the BB method and some other efficient gradient methods.
Similar content being viewed by others
References
H.Akaike, “On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method,” Ann. Inst. Statist. Math., Tokyo, vol. 11, pp. 1–17, 1959.
R.Andreani, E.G.Birgin, J.M.Martínez, and J.Yuan, “Spectral projected gradient and variable metric methods for optimization with linear inequalities,” IMA J.Numer. Anal., vol. 25, pp. 221–252, 2005.
J.Barzilai and J.M.Borwein, “Two-point step size gradient methods,” IMA J. Numer. Anal., vol. 8, pp. 141–148, 1988.
L.Bello and M.Raydan, “Preconditioned spectral projected gradient methods on convex sets,” Journal of Computational Mathematics, vol. 23, pp. 225–232, 2005.
E.G.Birgin, J.M.Martínez, and M.Raydan, “Nonmonotone spectral projected gradient methods on convex sets,” SIAM J. Optim., vol. 10, pp. 1196–1211, 2000.
E.G.Birgin, J.M.Martínez, and M.Raydan, “Algorithm 813: SPG—software for convex-constrained optimization,” ACM Trans. Math. Software, vol. 27, pp. 340–349, 2001.
E.G.Birgin, J.M.Martínez, and M.Raydan, “Inexact spectral projected gradient methods on convex sets,” IMA J. Numer. Anal., vol. 23, pp. 539–559, 2003.
A.Cauchy, “Méthode générale pour la résolution des syst è ms d’équations simultanées,” Comp. Rend. Sci. Paris, vol. 25, pp. 536–538, 1847.
Y.H.Dai, “Alternate step gradient method,” Optimization, vol. 52, pp. 395–415, 2003.
Y.H.Dai and L.Z.Liao, “R-linear convergence of the Barzilai and Borwein gradient method,” IMA J. Numer. Anal., vol. 22, pp. 1–10, 2002.
Y.H.Dai and Y.Yuan, “Alternate minimization gradient method,” IMA J. Numer. Anal., vol. 23, pp. 377–393, 2003.
R.Fletcher, “Low storage methods for unconstrained optimization,” Lectures in Applied Mathematics (AMS), vol. 26, pp. 165–179, 1990.
R.Fletcher, “On the Barzilai-Borwein method,” Numerical Analysis Report NA/207, Department of Mathematics, University of Dundee, 2001.
A.Friedlander, J.M. Martínez, B.Molina, and M.Raydan, “Gradient method with retards and generalizations,” SIAM J. Numer. Anal., vol. 36, pp. 275–289, 1999.
L.Grippo, F.Lampariello, and S.Lucidi, “A nonmonotone line search technique for Newton’s method,” SIAM J. Numer. Anal., vol. 23, pp. 707–716, 1986.
M.R.Hestenes and E.L.Stiefel, “Methods of conjugate gradients for solving linear systems,” J. Research National Bureau of Standards, vol. B49, pp. 409–436, 1952.
W.LaCruz, J.M.Martínez, and M.Raydan, “Spectral residual method without gradient information for solving large-scale nonlinear systems of equations,” Mathematics of Computation, to appear.
W.LaCruz and M.Raydan, “Nonmonotone spectral methods for large-scale nonlinear systems,” Optimization Methods and Software, vol. 18, pp. 583–599, 2003.
J.-L.Lamotte, B.Molina, and M.Raydan, “Smooth and adaptive gradient method with retards,” Mathematical and Computer Modelling, vol. 36, pp. 1161–1168, 2002.
F.Luengo and M.Raydan, “Gradient method with dynamical retards for large-scale optimization problems,” Electronic Transactions on Numerical Analysis, vol. 16, pp. 186–193, 2003.
M.Raydan, “On the Barzilai and Borwein choice of steplength for the gradient method,” IMA J. Numer. Anal., vol. 13, pp. 321–326, 1993.
M.Raydan, “The Barzilai and Borwein method for the large scale unconstrained minimization problem,” SIAM J. Optim., vol. 7, pp. 26–33, 1997.
M.Raydan and B.F.Svaiter, “Relaxed steepest descent and Cauchy-Barzilai-Borwein method,” Computational Optimization and Applications, vol. 21, pp. 155–167, 2002.
T.Serafini, G.Zanghirati, and L.Zanni, “Gradient projection methods for quadratic programs and applications in training support vector machines,” Tech. Rep. 48, University of Modena and Reggio Emilia, Italy, 2003.
H.Zhang and W.W. Hager, “A nonmonotone line search technique and its application to unconstrained optimization,” SIAM J. Optim., vol. 14, pp. 1043–1056, 2004.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhou, B., Gao, L. & Dai, YH. Gradient Methods with Adaptive Step-Sizes. Comput Optim Applic 35, 69–86 (2006). https://doi.org/10.1007/s10589-006-6446-0
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-006-6446-0