Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks (1905.13210v3)

Abstract: We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i.e., number of hidden nodes per layer) is much larger than the number of training data points. We show that, the expected $0$-$1$ loss of a wide enough ReLU network trained with stochastic gradient descent (SGD) and random initialization can be bounded by the training loss of a random feature model induced by the network gradient at initialization, which we call a neural tangent random feature (NTRF) model. For data distributions that can be classified by NTRF model with sufficiently small error, our result yields a generalization error bound in the order of $\tilde{\mathcal{O}}(n^{-1/2})$ that is independent of the network width. Our result is more general and sharper than many existing generalization error bounds for over-parameterized neural networks. In addition, we establish a strong connection between our generalization error bound and the neural tangent kernel (NTK) proposed in recent work.

• The paper introduces the NTRF model to derive tighter generalization error bounds for over-parameterized ReLU networks trained with SGD.
• It demonstrates that training near initialization keeps network behavior nearly linear, yielding error bounds independent of network width.
• The analysis refines sample complexity estimates and bridges the NTK framework with NTRF, offering deeper insights into deep learning generalization.

Overview of Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks

The paper "Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks," authored by Yuan Cao and Quanquan Gu, addresses a critical question in the paper of deep neural networks (DNNs): how can over-parameterized neural networks generalize well without overfitting? This work provides significant insights into the generalization properties of deep ReLU networks trained with stochastic gradient descent (SGD) under the over-parameterization regime.

Core Contributions

The paper contributes to the theoretical understanding of DNNs' generalization through the introduction of a new analytical framework for ReLU networks. The key contributions are:

1. Neural Tangent Random Feature Model: It introduces the concept of a neural tangent random feature (NTRF) model, which serves as a reference function class for deriving generalization error bounds for DNNs. The NTRF model is built upon the random feature model induced by the network gradient at initialization and is an extension of the neural tangent kernel (NTK) framework.
2. Generalization Error Bound: The authors demonstrate that, given the NTRF model, the expected $0$-$1$ loss can be bounded by the training loss of the model. They establish that for data distributions that are classifiable with a small error by the NTRF model, the generalization error achieves a bound of $\tilde\cO(n^{-1/2})$, which interestingly remains independent of the network's width. This result surpasses several existing bounds for over-parameterized networks.
3. Sample Complexity Improvements: The analysis provides a sharper sample complexity bound compared to previous works, particularly improving on the results by a factor related to the target generalization error. The bound suggests how sample complexity scales in terms of network width, network depth, and the "classifiability" of the data by the NTRF model.
4. NTK Connection: The paper highlights the connection between the NTRF model and the NTK framework, providing an interpretation of the generalization bound within the context of kernel methods. This link extends the NTK results to more profound settings and offers more general and tighter bounds.

Methodological Insights

The methodological backbone of the paper is the examination of SGD's behavior near random initialization. The authors leverage the fact that the neural network behaves almost linearly in terms of its parameters near initialization, which in turn allows for bounds traditionally associated with convex optimization. Their approach includes:

• A rigorous proof demonstrating that the neural network function's deviations from linearity are minimal in the initial training phases, facilitating cumulative loss analysis.
• An advanced application of online-to-batch conversion techniques for generating generalization bounds.
• Establishing a connection between NTRF properties and NTK, which is pivotal in understanding the data-dependent nature of neural network generalization and in guiding further theoretical studies on DNN learning dynamics.

Implications and Future Directions

This work offers theoretical grounding for empirical observations that over-parameterized DNNs generalize well, despite their capacity to memorize any dataset given enough parameters. Practically, these findings suggest that even very large neural networks can be trained effectively using SGD without fear of substantial overfitting, provided that the data distribution has certain properties highlighted by the NTRF analysis.

The implications extend to developing training algorithms and architectures that explicitly consider NTRF characteristics, potentially leading to networks that leverage over-parameterization safely.

Future research directions, as posited by the authors, include refinement of the over-parameterization conditions necessary for these generalization bounds and exploration into non-asymptotic results that further clarify the relationship between NTK and NTRF in broader neural network settings. Other outstanding questions include the development of SGD variants with specific generalization properties and further empirical validation of these bounds across diverse learning tasks.

In conclusion, this paper makes a substantial contribution to understanding the intricacies of deep learning generalization and provides a solid foundation for both theoretical and practical advancements in the field of neural networks.