Nezam Mahdavi-Amiri
Abstract
This paper is concerned with the development, numerical implementation, and testing of an algorithm for solving constrained nonlinear least squares problems. The algorithm is an adaptation of the least squares case of an exact penalty method for solving nonlinearly constrained optimization problems due to Coleman and Conn. It also uses the ideas of Nocedal and Overton for handling quasi-Newton updates of projected Hessians, those of Dennis, Gay, and Welsch for approaching the structure of nonlinear least squares Hessians, and those of Murray and Overton for performing line searches. This method has been tested on a selection of problems listed in the collection of Hock and Schittkowski. The results indicate that the approach taken here is a viable alternative for least squares problems to the general nonlinear methods studied by Hock and Schittkowski.
- 1 BOGGS, P. T., AND DENNIS, J. E., JR. A stability analysis for perturbed nonlinear iterative methods. Math. Comput. 30 (1976), 1-17.Google Scholar
Cross Ref
- 2 BROYDEN, C.G. The convergence of a class of double-rank minimization algorithms. J. Inst. Math. Appl. 6 (1970), 76-90.Google ScholarCross Ref
- 3 BUSOVACA, S. Handling degeneracy in a nonlinear L-1 algorithm. Ph.D. dissertation, University of Waterloo, Waterloo, Ontario, Canada, 1985. Google ScholarDigital Library
- 4 BYRD, R. H., AND SCHNABEL, R. B. Continuity of the null space basis and constrained optimization. Math. Program. 35, 1 (1985), 32-41. Google ScholarDigital Library
- 5 CHAMBERLAIN, R. M., POWELL, M. J. D., LEMARI~CHAL, C., AND PEDERSEN, H. C. The watchdog technique for forcing convergence in algorithms for constrained optimization. Math. Program. Stud. 16 (1982), 1-17.Google ScholarCross Ref
- 6 COLEMAN, T. F., AND CONN, A. R. Second-order conditions for an exact penalty function. Math. Program. 19 (1980), 155-177.Google ScholarDigital Library
- 7 COLEMAN, T. F., AND CONN, A. R. Nonlinear programming via an exact penalty function: Global analysis. Math. Program. 24 (1982), 137-161.Google ScholarDigital Library
- 8 COLEMAN, T. F., AND CONN, A. R. Nonlinear programming via an exact penalty function: Asymptotic analsyis. Math. Program. 24 (1982), 123-136.Google ScholarDigital Library
- 9 COLEMAN, T. F., AND CONN, A.R. On the local convergence of a quasi-Newton method for the nonlinear programming problem. SIAM J. Numer. Anal. 21 (1984), 755-769.Google ScholarDigital Library
- 10 COLEMAN, T. F., AND SORENSEN, D.C. A note on the computation of an orthogonal basis for the null space of a matrix. Math. Program. 29, 2 (1984), 234-242.Google ScholarDigital Library
- 11 DAVIDON, W.C. Variable metric method for minimization. ANL-5990 Review, Argonne National Laboratory, Argonne, Ill., 1959.Google Scholar
- 12 DENNIS, J. E., JR. Nonlinear Least Squares and Equations. In The State of the Art of Numerical Analysis, D. Jacobs, Ed. Academic Press, Orlando, Fla., 1977.Google Scholar
- 13 DENNIS, J. E., JR. Techniques for nonlinear least squares and robust regression. Commun. Stat.-Simul. Comput. B 7, 4 (1978), 345-359.Google Scholar
- 14 DENNIS, J. E., JR., AND Moal~, J.J. Quasi-Newton methods: Motivation and theory. SIAM Rev. 19 (1977), 46-89.Google ScholarCross Ref
- 15 DENNIS, J. E., JR., GAY, D. M., AND WELSCH, R. E. An adaptive nonlinear least-squares algorithm. ACM Trans. Math. Softw. 7, 3 (Sept. 1981), 348-368. Google ScholarDigital Library
- 16 DENNIS, J. E., JR:, MARTINEZ, H. J., AND TAPIA, R.A. A convergence theory for the structured BFGS secant method with an application to nonlinear least squares. Tech. Rep. 87-15, Rice University, Department of Mathematical Sciences, Houston, Tex., 1987.Google Scholar
- 17 FLETCHER, R. A new approach to variable metric algorithms. Comput. J. 13 (1970), 317-322.Google ScholarCross Ref
- 18 FLETCHER, R., AND POWELL, M. J.D. A rapidly convergent descent method for minimization. Comput. J. 6 (1963), 163-168.Google ScholarCross Ref
- 19 FRALEY, C. Solution of nonlinear least-squares probolems. Ph.D. dissertation, Stanford University, Stanford, Calif., 1987. Google ScholarDigital Library
- 20 GILL, P. E., AND MURRAY, W. Algorithms for the solution of the nonlinear least-squares problem. SIAM J. Numer. Anal. 15~ 5 (1978), 977-992.Google Scholar
- 21 GILL, P. E., MURRAY, W., AND WRIGHT, M.H. Practical Optimization. Academic Press, Orlando, Fla., 1981.Google Scholar
- 22 GILL, P. E., MURRAY, W., SAUNDERS, M. A., STEWART, G. W., AND WRIGHT, M.H. Properties of a representation of a basis for the null space. Math. Program. 33, 2 (1985), 172-186.Google ScholarDigital Library
- 23 GOLDFARB, D. A family of variable metric updates derived by variational means. Math. Com put. 24 (1970), 23-26.Google Scholar
- 24 HAN, S.-P. Variable metric methods for minimizing a class of nondifferentiable functions. Math. Program. 20 (1981), 1-13.Google ScholarCross Ref
- 25 HOCK, W., AND SCHITTKOWSKI, K. Test Examples for Nonlinear Programming Codes. Lecture Notes in Economic and Mathematical Systems # 187. Springer-Verlag, New York, 1981. Google ScholarDigital Library
- 26 MURRAY, W., AND OVERTON, M.L. Steplength algorithms for minimizing a class of nondifferentiable functions. STAN-CS-78-679, Stanford University, Stanford, Calif., 1978. Google ScholarDigital Library
- 27 NOCEDAL, J., AND OVERTON, M. L. Projected Hessian updating algorithms for nonlinearly constrained optimization. SIAM J. Numer. Anal. 22, 5 (1985), 821-850.Google ScholarCross Ref
- 28 PIETRZYKOWSKI, T. An exact potential method for constrained maxima. SIAM J. Anal. 6 (1969), 299-304.Google ScholarCross Ref
- 29 POWELL, M. J.D. How bad are the BFGS and DFP methods when the objective function is quadratic? Math. Program. 27 (1986), 34-47. Google ScholarDigital Library
- 30 SHANNO, D.F. Conditioning of quasi-Newton methods for function minimization. Mathematics of Computation 24 (1970), 647-656.Google ScholarCross Ref
- 31 WEDIN, P-A. The nonlinear least squares problem from a numerical point of view. Technical Memoranda I and II, Lund University, Lund, Sweden, 1972.Google Scholar
- 32 WEDIN, P-~k. On the Gauss-Newton method for the non-linear least squares problem. ITM Arbetsrapport No. 24, Institute for Tellampad Matematik, Box 5073, Stockholm 5, Sweden, 1974.Google Scholar
- 33 WEDIN, P-~k. Oil surface dependent properties of methods for separable nonlinear least squares problems. ITM Arbetsrapport NO. 23, Institute for Tellampad Matematik, Box 5073, Stockholm, 5, Sweden, 1974.Google Scholar
Index Terms
- Constrained nonlinear least squares: an exact penalty approach with projected structured quasi-Newton updates
Recommendations
Superlinearly Convergent Exact Penalty Methods with Projected Structured Secant Updates for Constrained Nonlinear Least Squares
We present a superlinearly convergent exact penalty method for solving constrained nonlinear least squares problems, in which the projected exact penalty Hessian is approximated by using a structured secant updating scheme. We give general conditions ...
Penalty Lagrangian Methods Via a Quasi-Newton Approach
For solving nonlinear programming problems we iteratively minimize the penalty Lagrangian developed by Hestenes, Powell, and Rockafellar with the multipliers estimated by solving nonnegatively constrained quadratic programming subproblems and a penalty ...
Tackling Box-Constrained Optimization via a New Projected Quasi-Newton Approach
Numerous scientific applications across a variety of fields depend on box-constrained convex optimization. Box-constrained problems therefore continue to attract research interest. We address box-constrained (strictly convex) problems by deriving two ...
Reviews
Benjamin L. Schwartz
This paper gives theory, implementation, and testing of an algorithm for nonlinear constrained least squares problems. The algorithm is adapted from an exact penalty method due to Coleman and Conn [1–3]. The word “exact” has a technical meaning relating to linear independence of the constraints at the isolated stationary solution point. The adaptation includes a more formal structuring of the formulation to include both equality and inequality constraints. Solution methods require setting convergence bounds (“epsilons”) for several functions under several conditions, such as global or local. Step size and direction are revised at each iteration, and the switch from global to local solution procedure is made based on the epsilon for that iteration. The bounds are all chosen empirically. While the bounds selected by the authors work well for a large sample set of problems, the paper offers no theory to guide the choices. Some theoretical questions could create difficulties, such as testing whether a stationary point is a true minimum. The implementation does not address this point or some other similar issues. In the sample problems, these omissions are not important, but a user of the code should be aware of the possible difficulties. On the other hand, the code explicitly accommodates other pathological conditions, such as a vacuous solution set for Z, one of the intermediate computation products used in the algorithm. From the sample problems, I conclude that for least squares problems, this approach is competitive with the general methods studies by Hock and Schittkowski [4].
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments