A proximal method for composite minimization

Abstract

We consider minimization of functions that are compositions of convex or prox-regular functions (possibly extended-valued) with smooth vector functions. A wide variety of important optimization problems fall into this framework. We describe an algorithmic framework based on a subproblem constructed from a linearized approximation to the objective and a regularization term. Properties of local solutions of this subproblem underlie both a global convergence result and an identification property of the active manifold containing the solution of the original problem. Preliminary computational results on both convex and nonconvex examples are promising.

Appendix

The basic building block for variational analysis (see Rockafellar and Wets [40] or Mordukhovich [35]) is the normal cone to a (locally) closed set S at a point $s \in S$, denoted by $N_S(s)$. It consists of all normal vectors: limits of sequences of vectors of the form $\lambda (u-v)$ for points $u,v \in \mathfrak {R}^m$ approaching s such that v is a closest point to u in S, and scalars $\lambda > 0$. On the other hand, tangent vectors are limits of sequences of vectors of the form $\lambda (u-s)$ for points $u \in S$ approaching s and scalars $\lambda > 0$. The set S is Clarke regular at s when the inner product of any normal vector with any tangent vector is always nonpositive. Closed convex sets and smooth manifolds are everywhere Clarke regular.

The epigraph of a function $h:\mathfrak {R}^m \rightarrow {\bar{\mathfrak {R}}}$ is the set

$$\begin{aligned} \text{ epi }\,h=\{(c,r)\in \mathfrak {R}^m\times \mathfrak {R}:r\ge h(c)\}. \end{aligned}$$

If the value of h is finite at some point $\bar{c} \in \mathfrak {R}^m$, then h is lower semicontinuous nearby if and only if its epigraph is locally closed around the point $\big (\bar{c}, h(\bar{c})\big )$. Henceforth we focus on that case.

The subdifferential of h at $\bar{c}$ is the set

$$\begin{aligned} \partial h(\bar{c})=\big \{v\in \mathfrak {R}^m\,:\,(v,-1) \in N_{\mathrm{epi}\,h}(\bar{c},h\big (\bar{c})\big ) \big \} \end{aligned}$$

and the horizon subdifferential is

$$\begin{aligned} \partial ^{\infty } h(\bar{c})=\big \{v\in \mathfrak {R}^m:(v,0) \in N_{\mathrm{epi}\,h}\big (\bar{c},h(\bar{c})\big ) \big \} \end{aligned}$$

(6.5)

(see [40, Theorem 8.9]). The function h is subdifferentially regular at $\bar{c}$ if its epigraph is Clarke regular at $\big (\bar{c}, h(\bar{c})\big )$ (as holds in particular if h is convex lower semicontinuous, or smooth). Subdifferential regularity implies that $\partial h(\bar{c})$ is a closed and convex set in $\mathfrak {R}^m$, and its recession cone is exactly $\partial ^{\infty } h(\bar{c})$ (see [40, Corollary 8.11]). In the case when h is locally Lipschitz, it is almost everywhere differentiable: h is then subdifferentially regular at $\bar{c}$ if and only if its directional derivative for every direction $d \in \mathfrak {R}^m$ equals

$$\begin{aligned} \limsup _{c \rightarrow \bar{c}} \langle \nabla h(c),d \rangle , \end{aligned}$$

where the $\limsup$ is taken over points c where h is differentiable.

Consider a subgradient $\bar{v} \in \partial h(\bar{c})$, and a localization of the subdifferential mapping $\partial h$ around the point $(\bar{c},\bar{v})$, by which we mean a set-valued mapping $T:\mathfrak {R}^m \rightrightarrows \mathfrak {R}^m$ defined by

$$\begin{aligned} T(y)=\left\{ \begin{array}{ll} \partial h(y) \cap B_{\epsilon }(\bar{v}) &{} (|y - \bar{c}| \le \epsilon ,\,|h(y) - h(\bar{c})| \le \epsilon ) \\ \emptyset &{}(\text{ otherwise }) \end{array}\right. \end{aligned}$$

for some constant $\epsilon >0$. The function h is prox-regular at $\bar{c}$ for $\bar{v}$ if some such localization is hypomonotone: that is, for some constant $\rho > 0$, we have

$$\begin{aligned} z \in T(y)\quad \text{ and }\quad z'\in T(y')\Rightarrow \langle z'-z,y'-y \rangle \ge -\rho |y'-y|^2. \end{aligned}$$

This definition is equivalent to Definition 1.1 (with the same constant $\rho$) [40, Example 12.28 and Theorem 13.36]. Prox-regularity at $\bar{c}$ (for all subgradients v) implies subdifferential regularity.

A general class of prox-regular functions common in engineering applications is “lower ${\mathcal {C}}^2$” functions [40, Definition 10.29]. A function $h:\mathfrak {R}^m \rightarrow \mathfrak {R}$ is lower ${\mathcal {C}}^2$ around a point $\bar{c} \in \mathfrak {R}^m$ if h has the local representation

$$\begin{aligned} h(c)=\max _{t\in T}f(c,t)\quad \text{ for }\,c\in \mathfrak {R}^m\,\text{ near }\,{\bar{c}}, \end{aligned}$$

for some function $f:\mathfrak {R}^m \times T \rightarrow \mathfrak {R}$, where the space T is compact and the quantities f(c, t), $\nabla _c f(c,t)$, and $\nabla ^2_{cc} f(c,t)$ all depend continuously on (c, t). All lower ${\mathcal {C}}^2$ functions are prox-regular [40, Proposition 13.3]. A simple equivalent property, useful in theory though harder to check in practice, is that h has the form $g-\kappa |\cdot |^2$ around the point $\bar{c}$ for some continuous convex function g and some constant $\kappa$.

The normal cone is crucial to the definition of another central variational-analytic tool. Given a set-valued mapping $F : \mathfrak {R}^p \rightrightarrows \mathfrak {R}^q$ with closed graph,

$$\begin{aligned} \text{ gph }\,F=\{(u,v):v\in F(v)\}, \end{aligned}$$

at any point $(\bar{u},\bar{v}) \in \text{ gph }\,F$, the coderivative $D^*F(\bar{u}|\bar{v}):\mathfrak {R}^q \rightrightarrows \mathfrak {R}^p$ is defined by

$$\begin{aligned} w \in D^* F(\bar{u} | \bar{v})(y)\Leftrightarrow (w,-y) \in N_{\mathrm{gph}\,F}(\bar{u},\bar{v}). \end{aligned}$$

The coderivative generalizes the adjoint of the derivative of smooth vector function: for smooth $c : \mathfrak {R}^n \rightarrow \mathfrak {R}^m$, the set-valued mapping $x \mapsto F(x) := \{c(x)\}$ has coderivative given by $D^*F(x|c(x))(y) = \{\nabla c(x)^* y\}$ for all $x \in \mathfrak {R}^n$ and $y\in \mathfrak {R}^m$. As we see next, coderivative calculations drive two of the arguments in Sect. 4.1.

Proof of Corollary 4.3

Corresponding to any linear map $A :\mathfrak {R}^p \rightarrow \mathfrak {R}^q$, define a set-valued mapping $F_A :\mathfrak {R}^p \rightrightarrows \mathfrak {R}^q$ by $F_A(u) = Au-S$. A coderivative calculation shows, for vectors $v \in \mathfrak {R}^p$,

$$\begin{aligned} D^* F_A(0|0)(v) =\left\{ \begin{array}{ll} \{A^*v\} &{} \quad \big (v \in N_S(0)\big ) \\ \emptyset &{} \quad (\text{ otherwise }). \end{array}\right. \end{aligned}$$

Hence, by assumption, the only vector $v \in \mathfrak {R}^p$ satisfying $0 \in D^* F_{\bar{A}}(0|0)(v)$ is zero, so by [40, Thm 9.43], the mapping $F_{\bar{A}}$ is metrically regular at zero for zero. Applying Theorem 4.2 shows that there exist constants $\delta ,\gamma > 0$ such that, if $\Vert A-\bar{A}\Vert < \delta$ and $|v| < \delta$, then we have

$$\begin{aligned} \mathrm{dist}\,\!\big (0,F_A^{-1}(-v)\big ) \le \gamma \, \mathrm{dist}\,\!\big (-v,F_A(0)\big ), \end{aligned}$$

or equivalently,

$$\begin{aligned} \mathrm{dist}\,\!\big (0,A^{-1}(S-v)\big ) \le \gamma \, \mathrm{dist}\,(v,S). \end{aligned}$$

Since $0 \in S$, the right-hand side is bounded above by $\gamma |v|$, so the result follows. $\square$

Proof of Theorem 4.4

We simply need to check that the set-valued mapping $G :\mathfrak {R}^p \!\rightrightarrows \mathfrak {R}^q$ defined by $G(z) = F(z) - S$ is metrically regular at $\bar{z}$ for zero. Much the same coderivative calculation as in the proof of Corollary 4.3 shows, for vectors $v \in \mathfrak {R}^p$, the formula

$$\begin{aligned} D^* G(\bar{z}|0)(v) =\left\{ \begin{array}{ll} \{\nabla F(\bar{z})^*v\} &{} \big (v \in N_S(\bar{z})\big ) \\ \emptyset &{} (\text{ otherwise }). \end{array}\right. \end{aligned}$$

Hence, by assumption, the only vector $v \in \mathfrak {R}^p$ satisfying $0 \in D^* G(\bar{z}|0)(v)$ is zero, so metric regularity follows by [40, Thm 9.43]. $\square$

Alternative proof of Theorem 4.2

In the text we gave a short ad hoc proof of Theorem 4.2. Here we present a more formal approach. Denote the space of linear maps from $\mathfrak {R}^p$ to $\mathfrak {R}^q$ by $L(\mathfrak {R}^p,\mathfrak {R}^q)$, and define a mapping $g :L(\mathfrak {R}^p,\mathfrak {R}^q) \times \mathfrak {R}^p \rightarrow \mathfrak {R}^q$ and a parametric mapping $g_H :\mathfrak {R}^p \rightarrow \mathfrak {R}^q$ by $g(H,u)= g_H(u) = Hu$ for maps $H \in L(\mathfrak {R}^p,\mathfrak {R}^q)$ and points $u \in \mathfrak {R}^p$. Using the notation of [14, Section 3], the Lipschitz constant $l[g](0;\bar{u},0)$, is by definition the infimum of the constants $\rho$ for which the inequality

$$\begin{aligned} d\big (w,g_H(u)\big ) \le \rho d\big (u,g_H^{-1}(w)\big ) \end{aligned}$$

(6.6)

holds for all triples (u, w, H) sufficiently near the triple $(\bar{u}, 0, 0)$. Inequality (6.6) says simply

$$\begin{aligned} |w-Hu| \le \rho |u-z|\quad \text{ for } \text{ all }\,z \in \mathfrak {R}^p\,\hbox {satisfying}\, Hz=w, \end{aligned}$$

a property that holds providing $\rho \ge \Vert H\Vert$. We deduce

$$\begin{aligned} l[g](0;\bar{u},0) = 0. \end{aligned}$$

(6.7)

We can also consider $F+g$ as a set-valued mapping from $L(\mathfrak {R}^p,\mathfrak {R}^q) \times \mathfrak {R}^p$ to $\mathfrak {R}^q$, defined by $(F+g)(H,u) = F(u) + Hu$, and then the parametric mapping $(F+g)_H :\mathfrak {R}^p \rightrightarrows \mathfrak {R}^q$ is defined in the obvious way: in other words, $(F+g)_H(u) = F(u) + Hu$. According to [14, Theorem 2], Equation (6.7) implies the following relationship between the “covering rates” for F and $F+g$:

$$\begin{aligned} r[F+g](0;\bar{u},\bar{v}) = r[F](\bar{u}, \bar{v}). \end{aligned}$$

The reciprocal of the right-hand side is, by definition, the infimum of the constants $\kappa > 0$ such that inequality (4.1) holds for all pairs (u, v) sufficiently near the pair $(\bar{u}, \bar{v})$. By metric regularity, this number is strictly positive. On the other hand, the reciprocal of the left-hand side is, by definition, the infimum of the constants $\gamma > 0$ such that inequality (4.2) holds for all triples (u, v, H) sufficiently near the pair $(\bar{u}, \bar{v},0)$.