Bootstrap confidence sets for spectral projectors of sample covariance

Abstract

Let $X_{1},\ldots ,X_{n}$ be i.i.d. sample in $\mathbb {R}^{p}$ with zero mean and the covariance matrix $\varvec{\Sigma }$. The problem of recovering the projector onto an eigenspace of $\varvec{\Sigma }$ from these observations naturally arises in many applications. Recent technique from Koltchinskii and Lounici (Ann Stat 45(1):121–157, 2017) helps to study the asymptotic distribution of the distance in the Frobenius norm $\left\| \mathbf {P}_{r} - \widehat{\mathbf {P}}_{r} \right\| _{2}$ between the true projector $\mathbf {P}_{r}$ on the subspace of the rth eigenvalue and its empirical counterpart $\widehat{\mathbf {P}}_{r}$ in terms of the effective rank of $\varvec{\Sigma }$. This paper offers a bootstrap procedure for building sharp confidence sets for the true projector $\mathbf {P}_{r}$ from the given data. This procedure does not rely on the asymptotic distribution of $\left\| \mathbf {P}_{r} - \widehat{\mathbf {P}}_{r} \right\| _{2}$ and its moments. It could be applied for small or moderate sample size n and large dimension p. The main result states the validity of the proposed procedure for Gaussian samples with an explicit error bound for the error of bootstrap approximation. This bound involves some new sharp results on Gaussian comparison and Gaussian anti-concentration in high-dimensional spaces. Numeric results confirm a good performance of the method in realistic examples.

Auxiliary results

1.1 Concentration inequalities for sample covariances and spectral projectors in $\mathbf {X}$-world

In this section we present concentration inequalities for sample covariance matrices and spectral projectors in $\mathbf {X}$-world.

Theorem 6

Let $X, X_{1}, \ldots , X_{n}$ be i.i.d. centered Gaussian random vectors in $\mathbb {R}^{p}$ with covariance $\varvec{\Sigma }= {{\mathrm{\mathbb {E}}}}(X X^{\mathsf {T}})$. Then

$$\begin{aligned} {{\mathrm{\mathbb {E}}}}\Vert \widehat{\varvec{\Sigma }} - \varvec{\Sigma }\Vert \lesssim \Vert \varvec{\Sigma }\Vert \left( \sqrt{\frac{\mathtt {r}(\varvec{\Sigma })}{n}} + \frac{\mathtt {r}(\varvec{\Sigma })}{n}\right) . \end{aligned}$$

Moreover, for all $t \ge 1$ with probability $1 - e^{-t}$

$$\begin{aligned} \Vert \widehat{\varvec{\Sigma }} - \varvec{\Sigma }\Vert \lesssim \Vert \varvec{\Sigma }\Vert \left[ \sqrt{\frac{\mathtt {r}(\varvec{\Sigma })}{n}} \bigvee \frac{\mathtt {r}(\varvec{\Sigma })}{n} \bigvee \sqrt{\frac{t}{n}} \bigvee \frac{t}{n} \right] . \end{aligned}$$

Proof

See [11, Theorem 6, Corollary 2]. $\square$

To deal with spectral projectors we need the following result which was proved in [12]. Let us introduce additional notations. We denote by $\widetilde{\varvec{\Sigma }}$ an arbitrary perturbation of $\varvec{\Sigma }$ and $\widetilde{\mathbf {E}}{\mathop {=}\limits ^{{\textsf {def}}}}\widetilde{\varvec{\Sigma }}- \varvec{\Sigma }$. Recall that

$$\begin{aligned} \mathbf {C}_{r} {\mathop {=}\limits ^{{\textsf {def}}}}\sum _{s \ne r} \frac{1}{\mu _{r} - \mu _{s}} \mathbf {P}_{s}. \end{aligned}$$

Lemma 4

Let $\widetilde{\varvec{\Sigma }}$ be an arbitrary perturbation of $\varvec{\Sigma }$ and let $\widetilde{\mathbf {P}}_{r}$ be the corresponding projector. The following bound holds:

$$\begin{aligned} \Vert \widetilde{\mathbf {P}}_{r} - \mathbf {P}_{r}\Vert \le 4 \frac{\Vert \widetilde{\mathbf {E}}\Vert }{\overline{g}_{r}}. \end{aligned}$$

Moreover, $\widetilde{\mathbf {P}}_{r} - \mathbf {P}_{r} = L_{r}(\widetilde{\mathbf {E}}) + S_{r}(\widetilde{\mathbf {E}})$, where $L_{r}(\widetilde{\mathbf {E}}) {\mathop {=}\limits ^{{\textsf {def}}}}\mathbf {C}_{r} \widetilde{\mathbf {E}}\mathbf {P}_{r} + \mathbf {P}_{r} \widetilde{\mathbf {E}}\mathbf {C}_{r}$ and

$$\begin{aligned} \Vert S_{r}(\widetilde{\mathbf {E}})\Vert \le 14 \left( \frac{\Vert \widetilde{\mathbf {E}}\Vert }{\overline{g}_{r}} \right) ^{2}. \end{aligned}$$

Proof

See [12, Lemma 1]. $\square$

Theorem 7

(Concentration results in $\mathbf {X}$-world) Assume that the conditions of Theorem 1 hold. Then for all $t: 1 \le t \le n^{1/4}$ and

$$\begin{aligned} \frac{{{\mathrm{tr}}}\varvec{\Sigma }}{\overline{g}_{r}} \left( \sqrt{\frac{t}{n}} + \sqrt{\frac{\log p}{n}} \right) \lesssim 1, \end{aligned}$$

(46)

the following bound holds with probability at least $1 - e^{-t}$

$$\begin{aligned} \Bigl |\Vert \widehat{\mathbf {P}}_{r} - \mathbf {P}_{r}\Vert _{2}^{2} - \Vert L_{r}(\mathbf {E})\Vert _{2}^{2} \Bigr | \lesssim m_{r} \frac{\Vert \varvec{\Sigma }\Vert ^{3} \mathtt {r}^{3}(\varvec{\Sigma })}{\overline{g}_{r}^{3}} \left( \frac{t}{n}\right) ^{3/2}. \end{aligned}$$

Proof

The proof follows from [12, Theorems 3, 5]. $\square$

1.2 Concentration inequalities for sums of random variables and random matrices

In what follows for a vector $a = (a_{1}, \ldots , a_{n})$ we denote $\Vert a\Vert _{s} {\mathop {=}\limits ^{{\textsf {def}}}}\big (\sum _{k=1}^{n} |a_{k}|^{s}\big )^{1/s}$. For a random variable X and $r > 0$ we define the $\psi _{r}$-norm by

$$\begin{aligned} \Vert X\Vert _{\psi _{r}} {\mathop {=}\limits ^{{\textsf {def}}}}\inf \{ C > 0: {{\mathrm{\mathbb {E}}}}\exp (X/C)^{r} \le 2\}. \end{aligned}$$

If a random variable X is such that for any $p \ge 1, {{\mathrm{\mathbb {E}}}}^{1/p} |X|^{p} \le p^{1/r} K$, for some $K > 0$, then $\Vert X\Vert _{\psi _{r}} \le c K$ where $c > 0$ is a numerical constant.

Lemma 5

Let $X, X_{i}, i = 1, \ldots , n$ be i.i.d. random variables with ${{\mathrm{\mathbb {E}}}}X = 0$ and $\Vert X\Vert _{\psi _{r}} \le 1, 1 \le r \le 2$. Then there exists some absolute constant $C>0$ such that for all $p \ge 1$

$$\begin{aligned} {{\mathrm{\mathbb {E}}}}\left| \sum _{k=1}^{n} a_{k } X_{k} \right| ^{p} \le (C p)^{p/2} \Vert a\Vert _{2}^{p} + (C p)^{p} \Vert a\Vert _{r_{*}}^{p}, \end{aligned}$$

where $a = (a_{1}, \ldots , a_{n})$ and $1/r + 1/r_{*} = 1$.

Proof

See [1, Lemma 3.6]. $\square$

Lemma 6

If $0< s < 1$ and $X_{1},\ldots , X_{n}$ are independent random variables satisfying $\Vert X\Vert _{\psi _{s}} \le 1$, then for all $a = (a_{1}, \ldots , a_{n}) \in \mathbb {R}^{n}$ and $p \ge 2$

$$\begin{aligned} {{\mathrm{\mathbb {E}}}}\left| \sum _{k=1}^{n} a_{k} X_{k}\right| ^{p}\le & {} (C p)^{p/2} \Vert a\Vert _{2}^{p} + C_{s} p^{p/s} \Vert a\Vert _{p}^{p}. \end{aligned}$$

Moreover, for $s \ge 1/2$, $C_{s}$ is bounded by some absolute constant.

Proof

See [1, Lemma 3.7]. $\square$

Lemma 7

Let $\eta _{1}, \ldots , \eta _{n}$ be i.i.d. standard normal random variables. For all $t\ge 1$

$$\begin{aligned} {{\mathrm{\mathbb {P}}}}\left( \left| \sum _{i=1}^{n} a_{i}(\eta _{i}^{4} - 3)\right| \gtrsim t^{2} \Vert a\Vert _{2} \right) \le e^{-t}. \end{aligned}$$

(47)

Moreover, if $\overline{\eta }_{1}, \ldots , \overline{\eta }_{n}$ are i.i.d. standard normal random variables and independent of $\eta _{1}, \ldots , \eta _{n}$ then

$$\begin{aligned} {{\mathrm{\mathbb {P}}}}\left( \left| \sum _{i=1}^{n} a_{i}(\eta _{i}^{2} \overline{\eta }_{i}^{2} - 1)\right| \gtrsim t^{2} \Vert a\Vert _{2} \right) \le e^{-t}. \end{aligned}$$

(48)

Proof

We prove (48) only. The proof of (47) is similar. Let $\epsilon _{i}, i = 1, \ldots , n$, be i.i.d. Rademacher r.v. Denote $\xi _{i} {\mathop {=}\limits ^{{\textsf {def}}}}\eta _{i}^{2} \overline{\eta }_{i}^{2} - 1, i = 1, \ldots , n$. Applying Lemma 6 with $s = 1/2$ we write

$$\begin{aligned} {{\mathrm{\mathbb {E}}}}|\sum _{i=1}^{n} a_{i}\xi _{i}|^{p}\le & {} 2^{p} {{\mathrm{\mathbb {E}}}}|\sum _{i=1}^{n} a_{i} \epsilon _{i} \xi _{i}|^{p} \le C^{p} p^{p/2} \Vert a\Vert _{2}^{p} + C^{p} p^{2p} \Vert a\Vert _{p}^{p} \le C^{p} p^{2p} \Vert a\Vert _{2}^{p}. \end{aligned}$$

From Markov’s inequality

$$\begin{aligned} {{\mathrm{\mathbb {P}}}}\left( \left| \sum _{i=1}^{n} a_{i}(\eta _{i}^{2} \overline{\eta }_{i}^{2} - 1)\right| \ge t^{2} \Vert a\Vert _{2} \right)\le & {} \frac{C^{p} p^{2p}}{t^{2 p}}. \end{aligned}$$

Taking $p = t/(Ce)^{1/2}$ we finish the proof of the lemma. $\square$

Lemma 8

(Matrix Gaussian series) Consider a finite sequence $\{\mathbf {A}_{k}\}$ of fixed, self-adjoint matrices with dimension d, and let $\{\xi _{k}\}$ be a finite sequence of independent standard normal random variables. Compute the variance parameter

$$\begin{aligned} \sigma ^{2} {\mathop {=}\limits ^{{\textsf {def}}}}\left\| \sum _{k=1}^{n} \mathbf {A}_{k}^{2} \right\| . \end{aligned}$$

Then, for all $t \ge 0$,

$$\begin{aligned} {{\mathrm{\mathbb {P}}}}\left( \left\| \sum _{k=1}^{n} \xi _{k} \mathbf {A}_{k} \right\| \ge t\right) \le 2d \,\exp (- t^{2}/2\sigma ^{2}). \end{aligned}$$

Proof

See in [20, Theorem 4.1]. $\square$

Lemma 9

(Matrix Bernstein inequality) Consider a finite sequence ${\mathbf {X}_{k}}$ of independent, random, self-adjoint matrices with dimension d. Assume that ${{\mathrm{\mathbb {E}}}}\mathbf {X}_{k} = 0$ and $\lambda _{\max }(\mathbf {X}_{k}) \le R$ almost surely. Compute the norm of the total variance,

$$\begin{aligned} \sigma ^{2} {\mathop {=}\limits ^{{\textsf {def}}}}\left\| \sum _{k=1}^{n} {{\mathrm{\mathbb {E}}}}\mathbf {X}_{k}^{2} \right\| . \end{aligned}$$

Then the following inequalities hold for all $t \ge 0$:

$$\begin{aligned} {{\mathrm{\mathbb {P}}}}\left( \lambda _{\max }\left( \sum _{k=1}^{n} \mathbf {X}_{k} \right) \ge t\right) \le d\, \exp \left( - \frac{t^{2}/2}{\sigma ^{2} + R t/3}\right) . \end{aligned}$$

Moreover, if ${{\mathrm{\mathbb {E}}}}\mathbf {X}_{k} = 0$ and ${{\mathrm{\mathbb {E}}}}\mathbf {X}_{k}^{p} \preceq \frac{p!}{2} R^{p-2} \mathbf {A}_{k}^{2}$ then the following inequalities hold for all $t \ge 0$:

$$\begin{aligned} {{\mathrm{\mathbb {P}}}}\left( \lambda _{\max }\left( \sum _{k=1}^{n} \mathbf {X}_{k} \right) \ge t\right) \le d \,\exp \left( - \frac{t^{2}/2}{{\tilde{\sigma }}^{2} + R t}\right) , \end{aligned}$$

where

$$\begin{aligned} {\tilde{\sigma }}^{2}&{\mathop {=}\limits ^{{\textsf {def}}}}&\Big \Vert \sum _{k=1}^{n} \mathbf {A}_{k}^{2} \Big \Vert . \end{aligned}$$

Proof

See in [20, Theorem 6.1]. $\square$

1.3 Auxiliary lemma

Lemma 10

Assume that $Z_1, Z_2$ be i.i.d. and ${{\mathrm{\mathcal {N}}}}(0,1)$. Let $\lambda _1, \lambda _2$ be any positive numbers and $b \ne 0$. There exists an absolute constant c such that

$$\begin{aligned} \left| \int _{-T}^{T} e^{itb} {{\mathrm{\mathbb {E}}}}\exp \bigg (it \big [\lambda _{1} Z_{1}^{2} + \lambda _{2} Z_{2}^{2}\big ]\bigg ) dt \right| \le \frac{c}{\sqrt{\lambda _{1} \lambda _{2}}}. \end{aligned}$$

(49)

Proof

Denote the l.h.s. of (49) by $I'$. Using Euler’s formula for complex exponential function we get for positive g and any $d \in \mathbb {R}$

$$\begin{aligned} g + i d = \sqrt{g^{2} + d^{2}} e^{i \zeta }, \quad \zeta = \arcsin \frac{d}{\sqrt{g^{2} + d^{2}}}. \end{aligned}$$

Hence, by (34) we get

$$\begin{aligned} I' = \left| \int _{-T}^{T} \exp \bigg (i t b + \sum _{k=1}^{2} \frac{i \phi _{k} }{2}\bigg ) \prod _{k=1}^{2} \left( 1+4t^{2}\lambda ^{2}_{k}\right) ^{-1/4} \right| , \end{aligned}$$

where $\phi _{k} {\mathop {=}\limits ^{{\textsf {def}}}}\phi _{k}(t) {\mathop {=}\limits ^{{\textsf {def}}}}\arcsin \big (2 \lambda _{k} t/(1 + 4 t^{2} \lambda _{k}^{2})^{\frac{1}{2}}\big )$. Since $\prod _{k=1}^{2}\left( 1+4t^{2}\lambda ^{2}_{k}\right) ^{-1/4}$ is even function and $\phi _{k}(t), k = 1,2$, is odd function of t, we may rewrite $I'$ as follows

$$\begin{aligned} I' = \frac{2}{\sqrt{\lambda _{1} \lambda _{2}}} \left| \int _{0}^{T} \frac{1}{t} \sin \bigg (t b + \sum _{k=1}^{2} \frac{1}{2} \bigg (\phi _{k} - \frac{\pi }{2} \bigg ) \bigg ) \prod _{k=1}^{2}\left( \frac{t^{2} \lambda _{k}^{2}}{1+4t^{2}\lambda ^{2}_{k}}\right) ^{1/4} \, dt \right| . \end{aligned}$$

We note that

$$\begin{aligned} \prod _{k=1}^{2}\left( \frac{t^{2} \lambda _{k}^{2}}{1+4t^{2}\lambda ^{2}_{k}}\right) ^{1/4}\le & {} \sqrt{|t| \lambda _{2}} \end{aligned}$$

Hence, to prove (49) it is enough to show that

$$\begin{aligned} I'' {\mathop {=}\limits ^{{\textsf {def}}}}\left| \int _{1/\lambda _{2}}^{T} \frac{1}{t} \sin \bigg (t b + \sum _{k=1}^{2} \frac{1}{2} \bigg (\phi _{k} - \frac{\pi }{2} \bigg ) \bigg ) \prod _{k=1}^{2}\left( \frac{t^{2} \lambda _{k}^{2}}{1+4t^{2}\lambda ^{2}_{k}}\right) ^{1/4} \, dt \right| \le c. \end{aligned}$$

We may rewrite $I''$ as follows

$$\begin{aligned} I'' \le I_{1}'' + \cdots + I_4'', \end{aligned}$$

where

$$\begin{aligned} I_{1}''&{\mathop {=}\limits ^{{\textsf {def}}}} \left| \int _{1/\lambda _{2}}^{T} \frac{1}{t} \sin (t b)\, dt \right| , \\ I_{2}''&{\mathop {=}\limits ^{{\textsf {def}}}} \left| \int _{1/\lambda _{2}}^{T} \frac{1}{t} \left[ \sin \bigg (t b + \sum _{k=1}^{2} \frac{1}{2} \bigg (\phi _{k} - \frac{\pi }{2} \bigg ) \bigg )- \sin (t b) \right] \, dt \right| , \\ I_{3}''&{\mathop {=}\limits ^{{\textsf {def}}}} \left| \int _{1/\lambda _{2}}^{T} \frac{1}{t} \sin \bigg (t b + \sum _{k=1}^{2} \frac{1}{2} \bigg (\phi _{k} - \frac{\pi }{2} \bigg ) \bigg ) \left[ 1 -\left( \frac{t^{2} \lambda _{1}^{2}}{1+4t^{2}\lambda ^{2}_{1}}\right) ^{1/4} \right] \, dt \right| ,\\ I_{4}''&{\mathop {=}\limits ^{{\textsf {def}}}} \left| \int _{1/\lambda _{2}}^{T} \frac{1}{t} \sin \bigg (t b + \sum _{k=1}^{2} \frac{1}{2} \bigg (\phi _{k} - \frac{\pi }{2} \bigg ) \bigg ) \right. \\&\left. \times \left[ 1 -\left( \frac{t^{2} \lambda _{2}^{2}}{1+4t^{2}\lambda ^{2}_{2}}\right) ^{1/4} \right] \left( \frac{t^{2} \lambda _{1}^{2}}{1+4t^{2}\lambda ^{2}_{1}}\right) ^{1/4} \, dt \right| . \end{aligned}$$

The bound $I_{1}'' \le c$ is true since for any positive A and B we have

$$\begin{aligned} \left| \int _{A}^{B} \frac{\sin t}{t} \, dt \right| \le 2\int _0^\pi \frac{\sin t}{t} \, dt. \end{aligned}$$

To estimate $I_{2}''$ we shall use the following inequalities

$$\begin{aligned}&|\sin (x+y) - \sin (x)| \le |y| \quad \text { for all } x,y \in \mathbb {R},\\&0 \le \frac{\pi }{2} - \arcsin (1-z) \le 2^{\frac{3}{2}} z^{\frac{1}{2}} \quad \text { for } 0 \le z \le 1. \end{aligned}$$

Applying these inequalities we get that

$$\begin{aligned} \left| \sin \bigg (t b + \sum _{k=1}^{2} \frac{1}{2} \bigg (\phi _{k} - \frac{\pi }{2} \bigg ) \bigg )- \sin (t b) \right| \le \frac{c'}{\lambda _{2}^{2} t^{2}}, \end{aligned}$$

where $c'$ is some absolute constant. Hence,

$$\begin{aligned} I_3'' \le \frac{c'}{\lambda _{2}^{2}}\int _{1/\lambda _{2}}^{\infty } \frac{1}{t^3} \, dt \le c. \end{aligned}$$

The estimates for $I_3''$ and $I_4''$ are similar. For simplicity we estimate $I_3''$ only. Applying the following inequality

$$\begin{aligned} 0 \le 1 - \left( \frac{t^{2} \lambda _{k}^{2}}{1+4t^{2}\lambda ^{2}_{k}}\right) ^{1/4} \le \frac{1}{4 t^{2} \lambda _{2}^{2}}, \quad k = 1, 2, \end{aligned}$$

we obtain that

$$\begin{aligned} I_3'' \le \frac{c''}{\lambda _{2}^{2}}\int _{1/\lambda _{2}}^{\infty } \frac{1}{t^3} \, dt \le c, \end{aligned}$$

where $c''$ is some absolute constant. $\square$