Universal Matrix Sparsifiers and Fast Deterministic Algorithms for Linear Algebra

Abstract: Let $\mathbf S \in \mathbb R^{n \times n}$ satisfy $|\mathbf 1-\mathbf S|_2\le\epsilon n$, where $\mathbf 1$ is the all ones matrix and $|\cdot|_2$ is the spectral norm. It is well-known that there exists such an $\mathbf S$ with just $O(n/\epsilon<sup>2)$ non-zero entries: we can let $\mathbf S$ be the scaled adjacency matrix of a Ramanujan expander graph. We show that such an $\mathbf S$ yields a $universal$ $sparsifier$ for any positive semidefinite (PSD) matrix. In particular, for any PSD $\mathbf A \in \mathbb{R}^{n\times n}$ with entries bounded in magnitude by $1$, $$|\mathbf A - \mathbf A\circ\mathbf S|_2 \le \epsilon n$$, where $\circ$ denotes the entrywise (Hadamard) product. Our techniques also give universal sparsifiers for non-PSD matrices. In this case, letting $\mathbf S$ be the scaled adjacency matrix of a Ramanujan graph with $\tilde O(n/\epsilon<sup>4)$ edges, we have $$|\mathbf A - \mathbf A \circ \mathbf S |_2 \le \epsilon \cdot \max(n,|\mathbf A|_1)$$, where $|\mathbf A|_1$ is the nuclear norm. We show that the above bounds for both PSD and non-PSD matrices are tight up to log factors. Since $\mathbf A \circ \mathbf S$ can be constructed deterministically, our result for PSD matrices derandomizes and improves upon known results for randomized matrix sparsification, which require randomly sampling ${O}(\frac{n \log n}{\epsilon<sup>2})$ entries. We also leverage our results to give the first deterministic algorithms for several problems related to singular value approximation that run in faster than matrix multiplication time. Finally, if $\mathbf A \in {-1,0,1}^{n \times n}$ is PSD, we show that $\mathbf{\tilde A}$ with $|\mathbf A - \mathbf{\tilde A}|_2 \le \epsilon n$ can be obtained by deterministically reading $\tilde O(n/\epsilon)$ entries of $\mathbf A$. This improves the $1/\epsilon$ dependence on our result for general PSD matrices and is near-optimal.