Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization

Abstract

The main goal of this paper is to develop uniformly optimal first-order methods for convex programming (CP). By uniform optimality we mean that the first-order methods themselves do not require the input of any problem parameters, but can still achieve the best possible iteration complexity bounds. By incorporating a multi-step acceleration scheme into the well-known bundle-level method, we develop an accelerated bundle-level method, and show that it can achieve the optimal complexity for solving a general class of black-box CP problems without requiring the input of any smoothness information, such as, whether the problem is smooth, nonsmooth or weakly smooth, as well as the specific values of Lipschitz constant and smoothness level. We then develop a more practical, restricted memory version of this method, namely the accelerated prox-level (APL) method. We investigate the generalization of the APL method for solving certain composite CP problems and an important class of saddle-point problems recently studied by Nesterov (Math Program 103:127–152, 2005). We present promising numerical results for these new bundle-level methods applied to solve certain classes of semidefinite programming and stochastic programming problems.

Appendix

In this section, we provide the proof of Lemma 8.

Let $F$ and $F_\eta$ be defined in (4.11) and (4.13), respectively. Also let us denote, for any $\eta > 0$ and $x \in X$,

$$\begin{aligned} \psi _x(z) := F_\eta (x) + \langle \nabla F_\eta (x), z - x \rangle + \frac{{{\mathcal {L}}}_\eta }{2} \Vert z - x\Vert ^2 + \eta {{\mathcal {D}}}_v, \end{aligned}$$

(7.1)

where ${{\mathcal {D}}}_v$ and ${{\mathcal {L}}}_\eta$ are defined in (3.7) and (4.15), respectively. Clearly, in view of (1.2) and (4.16), $\psi _x$ is a majorant of both $F_\eta$ and $f$. Also let us define

$$\begin{aligned} Z_x {:=} \left\{ z \in {\mathbb {R}}^n: \Vert z - x\Vert ^2 = \frac{2}{{{\mathcal {L}}}_\eta } \left[ \eta {{\mathcal {D}}}_v + F_\eta (x) - F(x) \right] \right\} \!. \end{aligned}$$

(7.2)

Clearly, by the first relation in (4.16), we have

$$\begin{aligned} \Vert z-x\Vert ^2 \le \frac{2 \eta {{\mathcal {D}}}_v}{{{\mathcal {L}}}_\eta }, \ \ \forall \, z \in Z_x. \end{aligned}$$

(7.3)

Moreover, we can easily check that, for any $z \in Z_x$,

$$\begin{aligned} \psi _x(z) + \langle \nabla \psi _x(z), x - z \rangle = F(x), \end{aligned}$$

(7.4)

where $\nabla \psi _x(z) = \nabla F_\eta (x) + {{\mathcal {L}}}_\eta (z-x)$.

The following results provides the characterization of a subgradient direction of $F$.

Lemma 9

Let $x \in {\mathbb {R}}^n$ and $p \in {\mathbb {R}}^n$ be given. Then, $\exists z \in Z_x$ such that

$$\begin{aligned} \langle F'(x), p \rangle \le \langle \nabla \psi _x(z), p\rangle = \langle \nabla F_\eta (x) + {{\mathcal {L}}}_\eta (z-x), p \rangle . \end{aligned}$$

where $F'(x) \in \partial F(x)$.

Proof

Let us denote

$$\begin{aligned} t = \frac{1}{\Vert p\Vert }\left\{ \frac{2}{{{\mathcal {L}}}_\eta } \left[ \eta {{\mathcal {D}}}_v + F_\eta (x) - F(x) \right] \right\} ^\frac{1}{2} \end{aligned}$$

and $z_0 = x + t p$. Clearly, in view of (7.2), we have $z_0 \in Z_x$. By convexity of $F$ and (7.4), we have

$$\begin{aligned} F(x) + \langle F'(x), tp \rangle \le F(x + tp)&= \psi _x(z_0) = F(x) + \langle \nabla \psi _x(z_0), z_0 - x\rangle \\&= F(x) + t \langle \nabla \psi _x(z_0), p\rangle , \end{aligned}$$

which clearly implies the result.$\square$

We are now ready to prove Lemma 8.

Proof of Lemma 8

First note that by the convexity of $F$, we have

$$\begin{aligned} F(x_0) - \left[ F(x_1) + \langle F'(x_1), x_0 - x_1\right] \rangle \le \langle F'(x_0), x_0 - x_1 \rangle + \langle F'(x_1), x_1 - x_0 \rangle . \end{aligned}$$

Moreover, by Lemma 9, $\exists z_0 \in Z_{x_0}$ and $z_1 \in Z_{x_1}$ s.t.

$$\begin{aligned}&\langle F'(x_0), x_0 - x_1 \rangle + \langle F'(x_1), x_1 - x_0 \rangle \\&\quad \le \langle \nabla F_\eta (x_0) - \nabla F_\eta (x_1), x_0 - x_1 \rangle + {{\mathcal {L}}}_\eta \langle z_0 - x_0 - (z_1 - x_1), x_0 - x_1\rangle \\&\quad \le {{\mathcal {L}}}_\eta \Vert x_0 - x_1\Vert ^2 + {{\mathcal {L}}}_\eta (\Vert z_0 - x_0\Vert + \Vert z_1-x_1\Vert ) \Vert x_0 - x_1\Vert \\&\quad \le {{\mathcal {L}}}_\eta \Vert x_0 - x_1\Vert ^2 + 2 {{\mathcal {L}}}_\eta \left( \frac{2 \eta {{\mathcal {D}}}_v}{{{\mathcal {L}}}_\eta } \right) ^\frac{1}{2} \Vert x_0 - x_1\Vert \\&\quad = \frac{\Vert A\Vert ^2}{\sigma _v \eta } \Vert x_0 - x_1\Vert ^2 + 2 \left( \frac{2 \Vert A\Vert ^2 {{\mathcal {D}}}_v}{\sigma _v}\right) ^\frac{1}{2} \Vert x_0 - x_1\Vert , \end{aligned}$$

where the last inequality and equality follow from (7.3) and (4.15), respectively. Combining the above two relations, we have

$$\begin{aligned} F(x_0)&- \left[ F(x_1) + \langle F'(x_1), x_0 - x_1 \rangle \right] \le \frac{\Vert A\Vert ^2}{\sigma _v \eta } \Vert x_0 - x_1\Vert ^2\\&+ 2 \left( \frac{2 \Vert A\Vert ^2 {{\mathcal {D}}}_v}{\sigma _v}\right) ^\frac{1}{2} \Vert x_0 - x_1\Vert . \end{aligned}$$

The result now follows by tending $\eta$ to $+\infty$ in the above relation.$\square$