Low-Rank Approximation and Regression in Input Sparsity Time

We design a new distribution over m × n matrices S so that, for any fixed n × d matrix A of rank r, with probability at least 9/10, ∥SAx∥₂ = (1 ± ε)∥Ax∥₂ simultaneously for all x ∈ R^d. Here, m is bounded by a polynomial in rε^{− 1}, and the parameter ε ∈ (0, 1]. Such a matrix S is called a subspace embedding. Furthermore, SA can be computed in O(nnz(A)) time, where nnz(A) is the number of nonzero entries of A. This improves over all previous subspace embeddings, for which computing SA required at least Ω(ndlog d) time. We call these S sparse embedding matrices.

Using our sparse embedding matrices, we obtain the fastest known algorithms for overconstrained least-squares regression, low-rank approximation, approximating all leverage scores, and ℓ_p regression.

More specifically, let b be an n × 1 vector, ε > 0 a small enough value, and integers k, p ⩾ 1. Our results include the following.

—Regression: The regression problem is to find d × 1 vector x′ for which ∥Ax′ − b∥_p ⩽ (1 + ε)min _x∥Ax − b∥_p. For the Euclidean case p = 2, we obtain an algorithm running in O(nnz(A)) + Õ(d³ε ⁻²) time, and another in O(nnz(A)log(1/ε)) + Õ(d³ log (1/ε)) time. (Here, Õ(f) = f ċ log ^O(1)(f).) For p ∈ [1, ∞), more generally, we obtain an algorithm running in O(nnz(A) log n) + O(r\ε ⁻¹)^C time, for a fixed C.

—Low-rank approximation: We give an algorithm to obtain a rank-k matrix Â_k such that ∥A − Â_k∥_F ≤ (1 + ε )∥ A − A_k∥_F, where A_k is the best rank-k approximation to A. (That is, A_k is the output of principal components analysis, produced by a truncated singular value decomposition, useful for latent semantic indexing and many other statistical problems.) Our algorithm runs in O(nnz(A)) + Õ(nk²ε⁻⁴ + k³ε⁻⁵) time.

—Leverage scores: We give an algorithm to estimate the leverage scores of A, up to a constant factor, in O(nnz(A)log n) + Õ(r³)time.