On the computational and statistical complexity of over-parameterized matrix sensing

Abstract: We consider solving the low rank matrix sensing problem with Factorized Gradient Descend (FGD) method when the true rank is unknown and over-specified, which we refer to as over-parameterized matrix sensing. If the ground truth signal $\mathbf{X}^* \in \mathbb{R}<sup>{d*d}$ is of rank $r$, but we try to recover it using $\mathbf{F} \mathbf{F}<sup>\top$ where $\mathbf{F} \in \mathbb{R}<sup>{d*k}$ and $k&gt;r$, the existing statistical analysis falls short, due to a flat local curvature of the loss function around the global maxima. By decomposing the factorized matrix $\mathbf{F}$ into separate column spaces to capture the effect of extra ranks, we show that $$|\mathbf{F}<em>t \mathbf{F}_t - \mathbf{X}^*|</em>{F}<sup>2$$ converges to a statistical error of $\tilde{\mathcal{O}} ({k d \sigma<sup>2/n})$ after $$\tilde{\mathcal{O}}(\frac{\sigma_{r}}{\sigma}\sqrt{\frac{n}{d}})$$ number of iterations where $\mathbf{F}<em>t$ is the output of FGD after $t$ iterations, $\sigma<sup>2$ is the variance of the observation noise, $\sigma</em>{r}$ is the $r$-th largest eigenvalue of $\mathbf{X}<sup>*$, and $n$ is the number of sample. Our results, therefore, offer a comprehensive picture of the statistical and computational complexity of FGD for the over-parameterized matrix sensing problem.