Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition (1503.02101v1)

Abstract: We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allows for efficient optimization. Using this property we show that stochastic gradient descent converges to a local minimum in a polynomial number of iterations. To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points. Our analysis can be applied to orthogonal tensor decomposition, which is widely used in learning a rich class of latent variable models. We propose a new optimization formulation for the tensor decomposition problem that has strict saddle property. As a result we get the first online algorithm for orthogonal tensor decomposition with global convergence guarantee.

• The paper introduces strict saddle points in non-convex functions and shows that noisy SGD reliably escapes these points.
• It presents the first online algorithm for orthogonal tensor decomposition with global convergence under standard smoothness assumptions.
• Empirical evaluations confirm the method's efficiency and robustness in applications such as Independent Component Analysis.

Overview of "Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition"

The paper "Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition" by Rong Ge et al., presents a thorough examination of stochastic gradient descent (SGD) in the context of optimizing non-convex functions, particularly focusing on escaping saddle points to achieve local minima. The authors introduce the concept of strict saddle points in non-convex optimization problems, where each saddle point has a negative curvature, and demonstrate that SGD can efficiently escape these points, providing a global convergence guarantee. This work breaks new ground by applying these theoretical advances to the orthogonal tensor decomposition problem, furnishing the first online algorithm for this complex task with global convergence guarantees.

Strict Saddle Property for Non-Convex Optimization

The paper begins by defining the strict saddle property for non-convex functions. Specifically:

1. Stationary Points: Points where the gradient vanishes can be categorized into local minima, local maxima, and saddle points based on the eigenvalues of the Hessian.
2. Strict Saddle Definition: A non-convex function is deemed strict saddle if all its saddle points have at least one negative eigenvalue in the Hessian, thereby excluding degenerate cases where eigenvalues can be zero.

The significant contribution here is the recognition that the strict saddle property ensures that gradient-based methods can find a descent direction near saddle points with high probability due to the negative curvature, leading to efficient escape from these points.

SGD for Strict Saddle Functions

With this property, the authors analyze the behavior of stochastic gradient descent (SGD) and its ability to escape saddle points:

• Algorithm Design: They propose an enhanced algorithm, Noisy Gradient Descent, which includes an additional noise term to ensure robustness in escaping saddle points.
• Theoretical Analysis: The key theoretical result is that for strict saddle functions, SGD converges to a local minimum in polynomial time. This efficiency is demonstrated under common smoothness assumptions for the function and its Hessian.

Application to Orthogonal Tensor Decomposition

The application of these theoretical results to the orthogonal tensor decomposition problem is noteworthy. Tensor decomposition is central to several machine learning models, particularly latent variable models such as Hidden Markov Models, topic models, and more:

• Problem Formulation: The authors present a new objective function for tensor decomposition, ensuring it satisfies the strict saddle property.
• Global Convergence: Leveraging their SGD framework, they establish the first online algorithm for orthogonal tensor decomposition with a global convergence guarantee, thus bridging a crucial gap between theory and practical algorithm design.

Experimental Validation

The paper is complemented by empirical evaluations:

• Performance Analysis: Comparing the proposed algorithm with traditional reconstruction error-based methods, the new approach shows enhanced stability and convergence efficiency.
• Application Example: In the context of Independent Component Analysis (ICA), they demonstrate that their method effectively decomposes tensors, corroborating their theoretical findings.

Implications and Future Directions

This work's implications extend both practically and theoretically:

• Practical Implications: For machine learning practitioners, the new algorithm offers a scalable and efficient tool for tensor decomposition tasks, vital for complex applications in data analysis and pattern recognition.
• Theoretical Insights: The concept of strict saddle points and the corresponding analysis of SGD establish a new framework for understanding and managing the complexities of non-convex optimization.

Looking forward, the paper opens several avenues for future research:

1. Broader Classes of Non-Convex Functions: Extending the strict saddle property to a wider range of non-convex functions and enhancing the robustness of SGD in various contexts.
2. Applications Beyond Tensor Decomposition: Applying the theoretical insights to other challenging optimization problems in machine learning and data science domains.
3. Refinements in Online Algorithms: Further refining online algorithms to handle greater levels of noise and uncertainty, enhancing their applicability in real-time data processing scenarios.

Conclusion

In conclusion, this paper presents seminal work in the paper of non-convex optimization, particularly in how stochastic gradient descent can be employed to effectively escape saddle points and converge to local minima in polynomial time. The application to tensor decomposition not only demonstrates the practical utility of these theoretical insights but also sets a new benchmark for algorithmic efficiency in this domain. The careful blend of theory and application exemplified by this work is poised to influence future research and practice in machine learning optimization profoundly.