On the Power and Limitations of Random Features for Understanding Neural Networks (1904.00687v4)

Abstract: Recently, a spate of papers have provided positive theoretical results for training over-parameterized neural networks (where the network size is larger than what is needed to achieve low error). The key insight is that with sufficient over-parameterization, gradient-based methods will implicitly leave some components of the network relatively unchanged, so the optimization dynamics will behave as if those components are essentially fixed at their initial random values. In fact, fixing these explicitly leads to the well-known approach of learning with random features. In other words, these techniques imply that we can successfully learn with neural networks, whenever we can successfully learn with random features. In this paper, we first review these techniques, providing a simple and self-contained analysis for one-hidden-layer networks. We then argue that despite the impressive positive results, random feature approaches are also inherently limited in what they can explain. In particular, we rigorously show that random features cannot be used to learn even a single ReLU neuron with standard Gaussian inputs, unless the network size (or magnitude of the weights) is exponentially large. Since a single neuron is learnable with gradient-based methods, we conclude that we are still far from a satisfying general explanation for the empirical success of neural networks.

• The paper demonstrates that random feature methods cannot learn a single ReLU neuron with Gaussian inputs without exponentially large networks.
• The paper employs theoretical bounds and self-contained proofs to reveal that these methods fail to ensure polynomial-time learnability for complex functions.
• The paper highlights practical implications by suggesting that adaptive initialization strategies may overcome the inherent constraints of static random features.

An Examination of Random Features for Understanding Neural Networks

The paper authored by Yehudai and Shamir focuses on the interplay between over-parameterized neural networks and random feature methods. It begins with an exploration of how sufficient over-parameterization in neural networks allows for learning complex functions successfully. The authors draw a parallel to random feature methods, where certain components or weights remain relatively unchanged from their initial values, thus simplifying the dynamics. They argue that while these methods present a significant advancement in understanding neural networks, they inherently exhibit limitations in explaining the full extent of learnability.

The authors focus on the ReLU activation function, using it to demonstrate the limitations of random features. Their core proposition is that random features methods struggle to learn even a single ReLU neuron with Gaussian inputs under standard conditions unless the size of the network or the magnitude of its weights becomes exponentially large in the dimensionality $d$. This finding stands in contrast to known results where single neurons can be learned efficiently with gradient-based methods. Consequently, this discrepancy highlights that random features cannot satisfactorily explain the empirical successes of neural networks.

The authors further strengthen their argument by providing bounds through theoretical analysis. They prove that for achieving a low degree of approximation error for functions such as low-degree polynomials using over-parameterized networks, the current frameworks based on random features cannot adequately predict polynomial-time learnability of neural networks.

The paper also dedicates a section to demonstrating positive aspects of over-parameterized networks, providing self-contained proofs illustrating how such networks can learn polynomials with bounded degrees and coefficients using stochastic gradient descent and standard initialization schemes. They delve into coordinate-wise linear combinations and coupling techniques to show how these are pivotal in enabling such learnability.

Strongly numerically founded, their analyses hinge on deriving a connection between target functions and the approximation capabilities of random features. They contend that while random features can indeed concentrate around expected values — a necessary condition for effective learnability — the magnitude of features and associated parameters still exhibit notable constraints.

The implications of these findings extend to both theoretical and practical dimensions. On the theoretical front, the paper suggests that the often hailed explanatory frameworks surrounding random features in AI lack comprehensive capability — especially in scenarios of higher-dimensionality and complex function mappings. Practically, the paper delivers insights into neural network designs that could focus more on adaptive rather than static initializations, thus paving the way for more robust learning models.

Looking forward, this paper sets the stage for future research into overcoming the limitations presented. It calls for advancements in understanding the intrinsic power of neural networks beyond the confines of random features. This could include investigating models with adaptable architectures or weights that evolve more dynamically as learning progresses. Such exploration may redefine approaches to harnessing the full potential of over-parameterized networks in machine learning applications.