Robert Mansel Gower
Abstract
We compare two methods that calculate the sparsity pattern of Hessian matrices using the computational framework of automatic differentiation. The first method is a forward-mode algorithm by Andrea Walther in 2008 which has been implemented as the driver called hess_pat in the automatic differentiation package ADOL-C. The second is edge_push_sp, a new reverse mode algorithm descended from the edge_pushing algorithm for calculating Hessians by Gower and Mello in 2012. We present complexity analysis and perform numerical tests for both algorithms. The results show that the new reverse algorithm is very promising.
- Jason Abate, Christian Bischof, Lucas Roh, and Alan Carle. 1997. Algorithms and design for a second-order automatic differentiation module. In Proceedings of the International Symposium on Symbolic and Algebraic Computation. ACM, New York, 149--155. DOI:http://dx.doi.org/10.1145/258726.258770 Google Scholar
Digital Library
- C. Bischof, G. Corliss, and A. Griewank. 1993. Structured second- and higher-order derivatives through univariate Taylor series. Optimization Methods Softw. 2, 3, 211--232.Google ScholarCross Ref
- I. Bongartz, A. R. Conn, Nick Gould, and Ph. L. Toint. 1995. CUTE: Constrained and unconstrained testing environment. ACM Trans. Math. Softw. 21, 1, 123--160. DOI:http://dx.doi.org/10.1145/200979.201043 Google ScholarDigital Library
- Bruce Christianson. 1992. Automatic Hessians by reverse accumulation. IMA J. Numer. Anal. 12, 2, 135--150. DOI:http://dx.doi.org/10.1093/imanum/12.2.135Google ScholarCross Ref
- D. M. Gay. 1996. More AD of nonlinear AMPL models: Computing Hessian information and exploiting partial separability. In Computational Differentiation: Techniques, Applications, and Tools, M. Berz, C. Bischof, G. Corliss and A. Griewank, Eds., SIAM, Philadelphia, 173--184.Google Scholar
- Assefaw Hadish Gebremedhin, Fredrik Manne, and Alex Pothen. 2005. What color is your Jacobian? Graph coloring for computing derivatives. SIAM Rev. 47, 4, 629--705. DOI:http://dx.doi.org/10.1137/S0036144504444711 Google ScholarDigital Library
- Assefaw H. Gebremedhin, Arijit Tarafdar, Alex Pothen, and Andrea Walther. 2009. E_cient computation of sparse Hessians using coloring and automatic differentiation. INFORMS J. Comput. 21, 2, 209--223. Google ScholarDigital Library
- R. M. Gower and M. P. Mello. 2012. A new framework for the computation of Hessians. Optimization Methods Softw. 27, 2, 251--273. DOI:http://dx.doi.org/10.1080/10556788.2011.580098Google ScholarCross Ref
- Andreas Griewank. 1993. Some bounds on the complexity of gradients, Jacobians, and Hessians. In Complexity in Numerical Optimization, World Science Publications, River Edge, NJ, 128--162.Google Scholar
- Andreas Griewank, David Juedes, and Jean Utke. 1996. Algorithm 755: ADOL-C: a package for the automatic differentiation of algorithms written in C/C++. ACM Trans. Math. Softw. 22, 2, 131--167. DOI:http://dx.doi.org/10.1145/229473.229474 Google ScholarDigital Library
- Andreas Griewank and Andrea Walther. 2000. Algorithm 799: revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Trans. Math. Softw. 26, 1, 19--45. Google ScholarDigital Library
- Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: Principles and Techniques of Algorithmic Differentiation. 2nd Ed. SIAM, Philadelphia, PA. Google ScholarDigital Library
- W. Hock and K. Schittkowski. 1980. Test examples for nonlinear programming codes. J. Optimization Theory Appl. 30, 1, 127--129.Google ScholarCross Ref
- Ladislav Luksan and Jan Vlcek. 2003. Test problems for unconstrained optimization. Tech. Rep. 897. Academy of Sciences of the Czech Republic.Google Scholar
- Ph. L. Toint and A. Griewank. 1982. On the unconstrained optimization of partially separable objective functions. In Nonlinear Optimization, Academic Press, London, 301--312.Google Scholar
- E. Varnik. 2011. Exploitation of structural sparsity in algorithmic differentiation. Ph.D. dissertation, RWTH Aachen.Google Scholar
- Andrea Walther. 2008. Computing sparse Hessians with automatic differentiation. ACM Trans. Math. Softw. 34, 1, Art. 3. DOI:http://dx.doi.org/10.1145/1322436.1322439 Google ScholarDigital Library
- A. Walther. 2012. On the efficient computation of sparsity patterns for Hessians. In Recent Advances in Algorithmic Differentiation, Lecture Notes in Computational Science and Engineering, vol. 87, 139--149.Google ScholarCross Ref
Index Terms
- Computing the sparsity pattern of Hessians using automatic differentiation
Recommendations
Computing sparse Hessians with automatic differentiation
A new approach for computing a sparsity pattern for a Hessian is presented: nonlinearity information is propagated through the function evaluation yielding the nonzero structure. A complexity analysis of the proposed algorithm is given. Once the ...
Truncated Newton method for sparse unconstrained optimization using automatic differentiation
When solving large complex optimization problems, the user is faced with three major problems. These are (i) the cost in human time in obtaining accurate expressions for the derivatives involved; (ii) the need to store second derivative information; and ...
Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning
AbstractModern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions ...
Comments