Abstract
We propose a first order interior point algorithm for a class of non-Lipschitz and nonconvex minimization problems with box constraints, which arise from applications in variable selection and regularized optimization. The objective functions of these problems are continuously differentiable typically at interior points of the feasible set. Our first order algorithm is easy to implement and the objective function value is reduced monotonically along the iteration points. We show that the worst-case iteration complexity for finding an \(\epsilon \) scaled first order stationary point is \(O(\epsilon ^{-2})\). Furthermore, we develop a second order interior point algorithm using the Hessian matrix, and solve a quadratic program with a ball constraint at each iteration. Although the second order interior point algorithm costs more computational time than that of the first order algorithm in each iteration, its worst-case iteration complexity for finding an \(\epsilon \) scaled second order stationary point is reduced to \(O(\epsilon ^{-3/2})\). Note that an \(\epsilon \) scaled second order stationary point must also be an \(\epsilon \) scaled first order stationary point.
Similar content being viewed by others
References
Bian, W., Chen, X.: Worst-case complexity of smoothing quadratic regularization methods for non-Lipschitzian optimization. SIAM J. Optim. 28, 1718–1741 (2013)
Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)
Cartis, C., Gould, N.I.M., Toint, PhL: On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. SIAM J. Optim. 21, 1721–1739 (2011)
Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part I: motivation, convergence and numerical results. Math. Program. 127, 245–295 (2011)
Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part II: worst-case function- and derivative-evaluation complexity. Math. Program. 130, 295–319 (2011)
Cartis, C., Gould, N.I.M., Toint, PhL: An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity. IMA J. Numer. Anal. 32, 1661–1695 (2012)
Cartis, C., Gould, N.I.M., Toint, PhL: Complexity bounds for second-order optimality in unconstrained optimization. J. Complex. 28, 93–108 (2012)
Chen, X.: Smoothing methods for nonsmooth, novonvex minimization. Math. Program. 134, 71–99 (2012)
Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(L_2\)-\(L_p\) minimization. Math. Program. 143, 371–383 (2014)
Chen, X., Ng, M., Zhang, C.: Nonconvex \(l_p\) regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21, 4709–4721 (2012)
Chen, X., Niu, L., Yuan, Y.: Optimality conditions and smoothing trust region Newton method for non-Lipschitz optimization. SIAM J. Optim. 23, 1528–1552 (2013)
Chen, X., Xu, F., Ye, Y.: Lower bound theory of nonzero entries in solutions of \(l_2\)-\(l_{p}\) minimization. SIAM J. Sci. Comput. 32, 2832–2852 (2010)
Chen, X., Zhou, W.: Smoothing nonlinear conjugate gradient method for image restoration using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 3, 765–790 (2010)
Fan, J.: Comments on ’Wavelets in stastics: a review’ by A. Antoniadis. J. Ital. Stat. Soc. 6, 131–138 (1997)
Garmanjani, R., Vicente, L.N.: Smoothing and worst case complexity for direct-search methods in nonsmooth optimization. IMA J. Numer. Anal. 33, 1008–1028 (2013)
Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(L_p\) minimization. Math. Program. 21, 1721–1739 (2011)
Gratton, S., Sartenaer, A., Toint, PhL: Recursive trust-region methods for multiscale nonlinear optimization. SIAM J. Optim. 19, 414–444 (2008)
Griewank, A.: The Modification of Newton’s Method for Unconstrained Optimization by Bounding Cubic Terms. Technical report NA/12 (1981). University of Cambridge, UK Department of Applied Mathematics and Theoretical Physics (1981)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning Data Mining, Inference, and Prediction. Springer, New York (2009)
Huang, J., Horowitz, J.L., Ma, S.: Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Stat. 36, 587–613 (2008)
Nesterov, Y.: Introductory Lectures on Convex Optimization, Applied Optimization. Kluwer, Dordrecht (2004)
Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton’s method and its global performance. Math. Program. 108, 177–205 (2006)
Nesterov, Y.: Accelerating the cubic regularization of Newton’s method on convex problems. Math. Program. 112, 159–181 (2008)
Nikolova, M., Ng, M.K., Zhang, S., Ching, W.-K.: Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 1, 2–25 (2008)
Tibshirani, R.: Shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996)
Vavasis, S.A., Zippel, R.: Proving Polynomial Time for Sphere-Constrained Quadratic Programming, Technical Report 90-1182, Department of Computer Science. Cornell University, Ithaca, NY (1990)
Vavasis, S.A.: Nonlinear Optimization: Complexity Issues. Oxford Sciences, New York (1991)
Ye, Y.: Interior Point Algorithms: Theory and Analysis. Wiley, New York (1997)
Ye, Y.: On the complexity of approximating a KKT point of quadratic programming. Math. Program. 80, 195–211 (1998)
Ye, Y.: A new complexity result on minimization of a quadratic function with a sphere constraint. In: Floudas, C., Pardalos, P.M. (eds.) Recent Advances in Global Optimization. Princeton University Press, Princeton (1992)
Zhang, C.-H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38, 894–942 (2010)
Acknowledgments
We would like to thank Prof. Sven Leyffer, the anonymous Associate Editor and referees for their insightful and constructive comments, which help us to enrich the content and improve the presentation of the results in this paper. The authors acknowledge the support of the Department of Applied Mathematics, the Hong Kong Polytechnic University for the visit of Yinyu Ye.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported partly by Hong Kong Research Council Grant PolyU5003/10p, The Hong Kong Polytechnic University Postdoctoral Fellowship Scheme, the NSF foundation (11101107, 11271099) of China and US AFOSR Grant FA9550-12-1-0396.
Rights and permissions
About this article
Cite this article
Bian, W., Chen, X. & Ye, Y. Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization. Math. Program. 149, 301–327 (2015). https://doi.org/10.1007/s10107-014-0753-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-014-0753-5
Keywords
- Constrained non-Lipschitz optimization
- Complexity analysis
- Interior point method
- First order algorithm
- Second order algorithm