Three Factors Influencing Minima in SGD (1711.04623v3)

Abstract: We investigate the dynamical and convergent properties of stochastic gradient descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the relation between learning rate, batch size and the properties of the final minima, such as width or generalization, remains an open question. In order to tackle this problem we investigate the previously proposed approximation of SGD by a stochastic differential equation (SDE). We theoretically argue that three factors - learning rate, batch size and gradient covariance - influence the minima found by SGD. In particular we find that the ratio of learning rate to batch size is a key determinant of SGD dynamics and of the width of the final minima, and that higher values of the ratio lead to wider minima and often better generalization. We confirm these findings experimentally. Further, we include experiments which show that learning rate schedules can be replaced with batch size schedules and that the ratio of learning rate to batch size is an important factor influencing the memorization process.

• The paper shows that SGD behaves as a discretization of a continuous SDE when the LR/BS ratio is maintained, capturing intrinsic training dynamics.
• The study establishes a theoretical link between the LR/BS ratio and minima width, with higher ratios yielding wider minima and enhanced generalization.
• Experimental analyses validate that proportional adjustments of learning rate and batch size preserve loss landscape geometry, confirmed by Hessian-based observations.

Analysis of "Three Factors Influencing Minima in SGD"

This paper investigates the impact of stochastic gradient descent (SGD) dynamics on deep neural networks (DNNs), emphasizing the confluence of learning rate, batch size, and gradient covariance in determining the nature of final minima. Insightfully, SGD is approximated by a stochastic differential equation (SDE), elucidating how the learning rate to batch size ratio (LR/BS) significantly influences both the convergence properties and the resulting minima's generalization.

Key Contributions

1. SGD as an SDE Discretization: This work posits that any SGD processes sharing the same LR/BS are merely different discretizations of an identical SDE. This assertion underscores that modifying either learning rate or batch size without altering their ratio preserves the intrinsic dynamics of SGD.
2. Relation between Minima Width and LR/BS: A clear theoretical relationship is presented, associating a higher LR/BS ratio with wider minima. This is substantiated by the derivation that suggests wider minima at higher stochastic noise levels linked to better generalization capabilities.
3. Experimental Corroboration: Through rigorous experimental analysis, it is demonstrated that changes in learning dynamics and the geometry of loss minima are consistent when learning rate and batch size are adjusted proportionally. The LR/BS ratio's impact on the Hessian corroborates theory, indicating wider minima with smaller Hessian values are achieved at larger ratios.
4. Practical Applicability: The paper explores the interchangeability of learning rate schedules and batch size schedules, reinforcing the importance of maintaining the LR/BS ratio throughout training phases to influence memorization mechanisms and generalization performance effectively.

Implications and Future Directions

The findings have significant implications for designing training protocols and hyperparameter optimization strategies in DNNs. By focusing on the LR/BS ratio, practitioners can exert more explicit control over the training process, potentially enhancing network generalization without compromising convergence rates. This approach offers a path forward for more effective and efficient neural network training regimes.

Theoretically, the outcomes suggest exploring further into the SGD's dynamics through the lens of SDEs, offering a robust framework to understand optimization in non-convex landscapes. This work could inspire further investigation into adaptive algorithms that dynamically adjust the LR/BS ratio to traverse complex loss surfaces, potentially reducing the reliance on manual hyperparameter tuning.

In conclusion, the paper elucidates the significant role of the LR/BS ratio in determining the nature of minima achieved in SGD, establishing a foundational understanding that bridges theoretical predictions with practical observations. This provides a refined perspective for researchers and practitioners aiming to optimize deep learning models efficiently.