Gearhart–Koshy acceleration for affine subspaces

https://doi.org/10.1016/j.orl.2020.12.007Get rights and content

Abstract

The method of cyclic projections finds nearest points in the intersection of finitely many affine subspaces. To accelerate convergence, Gearhart & Koshy proposed a modification which, in each iteration, performs an exact line search based on minimising the distance to the solution. When the subspaces are linear, the procedure can be made explicit using feasibility of the zero vector. This work studies an alternative approach which does not rely on this fact, thus providing an efficient implementation in the affine setting.

Introduction

Our setting is a real Hilbert space H equipped with inner-product , and induced norm . Consider closed affine subspaces M1,,MnH, and suppose Mi=1nMi.Given x0H, we study the best approximation problem minxHxx02 subject to xM.In this work, our focus is the case in which the nearest point projectors onto the individual spaces, M1,,Mn, are accessible. Recall that the projector onto Mi is the operator PMi:HMi given by PMi(x)arg minzMixz.The method of cyclic projections is an iterative procedure for solving (1) (i.e., for computing PM(x0)) by using only the individual projection operators PM1,,PMn. Although originally studied when M1,,Mn are linear subspaces [11], [12], the following affine variant readily follows from translation properties of the projector.

Theorem 1.1 The Method of Cyclic Projections

Let M1,,Mn be closed affine subspaces of H with M=i=1nMi. Then, for each x0H, limk(PMnPMn1PM1)k(x0)=PM(x0).

The convergence rate of the sequence in Theorem 1.1 can be related to the angle between the subspaces. Recall that the (Friederichs) angle between two closed subspaces A and B is the angle in [0,π2] whose cosine is given by c(A,B)sup|a,b|:aA(AB),a=1bB(AB),b=1.The following result provides a bound on the convergence rate based on this quantity.

Theorem 1.2 [9, Corollary 9.34]

Let M1,,Mn be closed affine subspaces of H with M=i=1nMi. For i{1,,n}, let Mi denote the linear subspace parallel to Mi. Then, for each x0H, (PMnPMn1PM1)k(x0)PM(x0)ckx0PM(x0), where the constant c[0,1] is given by c1i=1n1(1ci2)12   with  cicMi,j=i+1nMj.

When c<1, Theorem 1.2 establishes R-linear convergence of the method of cyclic projections. This is easily seen to be the case, for instance, when ci<1 for all i{1,,n}. In the setting with n=2, this characterisation can be further refined: c<1 if and only if M1+M2 is closed (which always holds in finite dimensions) in which case convergence is linear, else c=1 and the rate of convergence is arbitrarily slow [1], [4].

Let Q:HH denote the cyclic projections operator given by QPMnPM1. In an attempt to accelerate the method of cyclic projections, Gearhart & Koshy [10] proposed the following scheme which iterates by performing an exact line search to choose to nearest point to PM(x0) in the affine span of {xk,Q(xk)}. When M1,,Mn are linear subspaces, it can be shown (see Section 3) that the step size tk can be computed using the expression tk=xkQ(xk),xkxkQ(xk)2if Q(xk)xk.Since it only requires vector arithmetic, evaluating this expression comes with relatively low computational cost. Moreover, Gearhart & Koshy’s scheme gives the following refinement of the upper-bound provided by Theorem 1.2 in (2).

Theorem 1.3 Gearhart–Koshy [10]

Let M1,,Mn be closed affine subspaces of H with M=i=1nMi. For each sequence (xk) generated by Algorithm1, there exists a sequence (fk)[0,1] such that xkPM(x0)cki=1kfix0PM(x0),where the constant c[0,1] is given by (3).

Although Theorem 1.3 still holds for affine subspaces, the efficient expression for tk provided by (4) is only valid for linear subspaces (this will be explained more precisely Section 3). Thus, in the affine case, it is no longer obvious how to efficiently apply the scheme.

In this work, we address the aforementioned problem by deriving an alternative expression for (4) which still holds in the affine case and still only requires vector arithmetic for its evaluation. Our key insight is the observation that (4) implicitly relies on the fact that the zero vector is always feasible for linear subspaces. The remainder of this work is structured as follows: In Section 2, we collect the necessary preliminaries. In Section 3, we discuss Gearhart & Koshy’s derivation of (4) and, in Section 4, we provide an alternative formula which still holds in the affine setting. In Section 5, we discuss some implications for nonlinear fixed iterations and finally, in Section 6, we provide computational examples.

Section snippets

Preliminaries

Let SH be a nonempty subset of H. Recall that the (nearest point) projector onto S is the operator PS:HS defined by PS(x)arg minzSxz.It is well-known (see, for instance, [9, 3.5]) that PS is a well-defined operator whenever S is closed and convex. Further, the definition in (5) also implies the translation formula PS(x)=PSy(xy)+yx,yH,where Sy{syH:sS}. The following proposition collects important properties of projectors for use in the subsequence sections.

Proposition 2.1 Properties of Projectors

Let SH be a nonempty,

Gearhart–Koshy acceleration for linear subspaces

In this section, we recall the derivation of Gearhart & Koshy’s scheme for linear subspaces [10]. This serves to both introduce the scheme, and to highlighting the immediate difficulty with extending the result to affine spaces.

Denote QPMnPM1. Using this notation, the method of cyclic projection (as discussed in Theorem 1.1) generates a sequence (xk) according to the fixed-point iteration xk+1Q(xk)kN.Gearhart & Koshy’s scheme attempts to accelerate convergence by instead defining the

Gearhart–Koshy acceleration for affine subspaces

In this section, we derive an alternate expression for the step size tk in Gearhart & Koshy’s scheme which is still valid for affine subspaces and which can be explicitly computed without knowledge of an intersection point (unlike the expression in (15)). To this end, let Qi:HH denote the operator QiPMiPM1if i{1,,n}Iif i=0.

Lemma 4.1

Let M1,,Mn be closed affine subspaces of H with M=i=1nMi. For i{1,,n}, let Mi denote the linear subspace parallel to Mi. If Q(xk)xk, then the solution of (8) is

Extensions to firmly nonexpansive operators

The orthogonality condition (18) was a key ingredient in the proof of Lemma 4.1. In this section, we investigate what remains true without this property. Our focus will be the following class of operators which generalise affine projectors.

Definition 5.1

An operator T:HH is firmly quasi-nonexpansive if T(x)y2+xT(x)2xy2for all xH and yFixT{yH:T(y)=y}.

It is straightforward to check that the inequality (23) is equivalent to requiring 0T(x)y,xT(x)xH,yFixT.As a consequence of Proposition 2.1

Computational examples

In this section, we provide numerical examples to demonstrate the results from the previous sections. Our presentation will focus on the comparison between the method of cycling projections (Theorem 1.2) and its accelerated counterpart (Theorem 4.1). However analogous conclusions apply for the other methods considered in this paper. All computations were performed in Python 3 on a machine running Ubuntu 18.04 with an Intel Core i7-8665U and 16 GB of memory.

Since the bound in Theorems 1.2 & 4.1

Acknowledgements

This work is supported in part by DE200100063 from the Australian Research Council . The author would like to thank Janosch Rieger for discussions relating to [13] which initiated this work, and the anonymous referee and editor for their helpful comments and suggestions.

References (14)

  • BauschkeH.H. et al.

    Finding best approximation pairs relative to two closed convex sets in Hilbert spaces

    J. Approx. Theory

    (2004)
  • GearhartW.B. et al.

    Acceleration schemes for the method of alternating projections

    J. Comput. Appl. Math.

    (1989)
  • BauschkeH.H. et al.

    The method of cyclic projections for closed convex sets in Hilbert space

    Contemp. Math.

    (1997)
  • BauschkeH.H. et al.

    Convex Analysis and Monotone Operator Theory in Hilbert Spaces

    (2017)
  • BauschkeH.H. et al.

    Characterizing arbitrarily slow convergence in the method of alternating projections

    Int. Trans. Oper. Res.

    (2009)
  • BauschkeH.H. et al.

    Accelerating the convergence of the method of alternating projections

    Trans. Amer. Math. Soc.

    (2003)
  • BorweinJ.M. et al.

    A cyclic Douglas–Rachford iteration scheme

    J. Optim. Theory Appl.

    (2014)
There are more references available in the full text version of this article.
View full text