Papers
Topics
Authors
Recent
Search
2000 character limit reached

Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

Published 26 Dec 2017 in cs.LG, cs.DS, math.OC, and stat.ML | (1712.09203v5)

Abstract: We show that the gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given O~(dr2)\tilde{O}(dr^{2}) random linear measurements of a rank rr positive semidefinite matrix X⋆X^{\star}, we can recover X⋆X^{\star} by parameterizing it by UU⊤UU^\top with U∈Rd×dU\in \mathbb R^{d\times d} and minimizing the squared loss, even if r≪dr \ll d. We prove that starting from a small initialization, gradient descent recovers X⋆X^{\star} in O~(r)\tilde{O}(\sqrt{r}) iterations approximately. The results solve the conjecture of Gunasekar et al.'17 under the restricted isometry property. The technique can be applied to analyzing neural networks with one-hidden-layer quadratic activations with some technical modifications.

Authors (3)
Citations (140)

Summary

No one has generated a summary of this paper yet.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Continue Learning

We haven't generated follow-up questions for this paper yet.