Complex fractal trainability boundary can arise from trivial non-convexity (2406.13971v1)

Abstract: Training neural networks involves optimizing parameters to minimize a loss function, where the nature of the loss function and the optimization strategy are crucial for effective training. Hyperparameter choices, such as the learning rate in gradient descent (GD), significantly affect the success and speed of convergence. Recent studies indicate that the boundary between bounded and divergent hyperparameters can be fractal, complicating reliable hyperparameter selection. However, the nature of this fractal boundary and methods to avoid it remain unclear. In this study, we focus on GD to investigate the loss landscape properties that might lead to fractal trainability boundaries. We discovered that fractal boundaries can emerge from simple non-convex perturbations, i.e., adding or multiplying cosine type perturbations to quadratic functions. The observed fractal dimensions are influenced by factors like parameter dimension, type of non-convexity, perturbation wavelength, and perturbation amplitude. Our analysis identifies "roughness of perturbation", which measures the gradient's sensitivity to parameter changes, as the factor controlling fractal dimensions of trainability boundaries. We observed a clear transition from non-fractal to fractal trainability boundaries as roughness increases, with the critical roughness causing the perturbed loss function non-convex. Thus, we conclude that fractal trainability boundaries can arise from very simple non-convexity. We anticipate that our findings will enhance the understanding of complex behaviors during neural network training, leading to more consistent and predictable training strategies.

Complex Fractal Trainability Boundary Arising from Trivial Non-Convexity

The research presented in the paper investigates the origins of fractal trainability boundaries in the optimization of neural networks and elucidates how complexities can arise from seemingly simple non-convex perturbations in loss landscapes. The paper primarily explores the behavior of gradient descent (GD) in the context of fractal trainability boundaries, revealing insights into how and why these fractal patterns emerge and the implications for neural network training.

Core Contributions

The principal contribution of this work is the identification of fractal trainability boundaries in loss landscapes that result from very elementary non-convex perturbations. Specifically, the authors examine how these fractals can arise from modifications as simple as cosine perturbations applied to quadratic loss functions. They investigate both additive and multiplicative perturbations, demonstrating how these modifications influence trainability.

1. Additive and Multiplicative Perturbations: The paper introduces two ways of perturbing a basic quadratic function—additive and multiplicative cosine functions. In the additive case, the perturbation is directly added to the function, while in the multiplicative case, it modifies the quadratic function through multiplication. The paper finds that both types can lead to fractal trainability boundaries, albeit with different characteristics and dependencies.
2. Roughness as a Determinant: The findings introduce the concept of "roughness" to measure the gradient's sensitivity to changes in parameters as a critical determinant of the fractal dimension of trainability boundaries. Roughness is identified as a pivotal factor transitioning the boundaries from non-fractal to fractal as it increases, particularly when the roughness leads the perturbed loss to become non-convex.
3. Dependency on Hyperparameters: The research also explores how the fractal dimension is influenced by hyperparameters such as perturbation wavelength and amplitude. The paper discovers that for additive perturbations, fractal dimensions increase with larger amplitudes and smaller wavelengths, demonstrating dependence on specific perturbation characteristics.
4. Numerical Investigations and Renormalization Approach: Through extensive numerical experiments leveraging renormalization techniques, the authors confirm that fractal trainability boundaries are not only theoretical constructs but realistic artifacts of certain optimization settings. The renormalization approach bridges different loss functions and their corresponding trainability boundaries.

Implications for Neural Network Training

The paper holds both theoretical and practical implications for neural network training and the broader domain of machine learning optimization:

• Enhanced Understanding of Loss Landscapes: By demystifying how fractal structures can organically arise even from straightforward non-convex cases, this paper advances the understanding of loss landscapes, a fundamental concept in machine learning.
• Optimization Strategy Development: The insights into how perturbation characteristics affect fractal dimensions can inform more robust hyperparameter tuning strategies, potentially leading to more efficient and consistent training of neural networks.
• Tool for Diagnosing and Designing Loss Functions: This work can serve as a guideline for diagnosing problematic training regimes that may be subject to chaotic behaviors and for designing loss functions less prone to such issues.

Future Directions

This research opens several avenues for future work. Key areas for exploration include extending the renormalization technique to a wider class of functions, examining roughness in complex neural networks with multiple layers or components, and formally proving the observed dependency of fractal dimensions on roughness. Additionally, addressing the impact of network architecture and data set characteristics on trainability boundary behaviors could further bridge these findings with practical deep learning applications.

Overall, the paper provides a rigorous investigation into the emergence of complex, fractal trainability boundaries from simple perturbations, offering both a theoretical framework and empirical evidence that deepen our understanding of neural network optimization dynamics.