Optimal Machine Intelligence at the Edge of Chaos (1909.05176v2)

Abstract: It has long been suggested that the biological brain operates at some critical point between two different phases, possibly order and chaos. Despite many indirect empirical evidence from the brain and analytical indication on simple neural networks, the foundation of this hypothesis on generic non-linear systems remains unclear. Here we develop a general theory that reveals the exact edge of chaos is the boundary between the chaotic phase and the (pseudo)periodic phase arising from Neimark-Sacker bifurcation. This edge is analytically determined by the asymptotic Jacobian norm values of the non-linear operator and influenced by the dimensionality of the system. The optimality at the edge of chaos is associated with the highest information transfer between input and output at this point similar to that of the logistic map. As empirical validations, our experiments on the various deep learning models in computer vision demonstrate the optimality of the models near the edge of chaos, and we observe that the state-of-art training algorithms push the models towards such edge as they become more accurate. We further establishes the theoretical understanding of deep learning model generalization through asymptotic stability.

• The paper presents a theoretical framework linking Neimark-Sacker bifurcation and asymptotic Jacobian norms to identify the edge of chaos.
• The paper shows that systems at the edge of chaos enable optimal information transfer, mirroring dynamics seen in classic chaotic maps.
• The paper empirically validates that deep learning models operating near the edge of chaos achieve superior generalization and performance in computer vision tasks.

The paper "Optimal Machine Intelligence at the Edge of Chaos" explores the longstanding hypothesis that biological brains may function optimally at a critical transition between order and chaos. This concept, known as the "edge of chaos," suggests that systems operating at this boundary can achieve optimal information processing.

Key Contributions:

1. Theoretical Framework:
   • The authors develop a general theory identifying the edge of chaos as the boundary between chaotic behavior and (pseudo)periodic behavior in nonlinear systems. This boundary is linked to Neimark-Sacker bifurcation, which is a type of bifurcation leading to the emergence of a torus in the system's phase space.
   • The edge of chaos is theoretically characterized using the asymptotic Jacobian norm values of nonlinear operators, highlighting the influence of system dimensionality.
2. Information Transfer:
   • The paper argues that at the edge of chaos, systems exhibit optimal information transfer between inputs and outputs. This phenomenon is similar to that observed in the logistic map, a classic chaotic system.
3. Empirical Validation:
   • Experiments conducted using various deep learning models in computer vision demonstrate that these models achieve optimal performance when operating near the edge of chaos. This is evidenced by superior information processing capabilities.
   • The paper observes that state-of-the-art training algorithms naturally drive models towards the edge of chaos, enhancing accuracy and efficiency.
4. Theoretical Insights into Generalization:
   • The authors propose a theoretical perspective on deep learning model generalization, linking it to asymptotic stability, which is partially attained when models are situated at this critical boundary.

Significance:

The research provides a novel theoretical basis for understanding optimal neural computation in both biological and artificial systems, contributing to the broader debate about the role of chaos in neural processing. It also offers practical insights into how deep learning models can be optimized, suggesting that this edge-of-chaos paradigm could be a fundamental principle in machine learning. This work bridges empirical observations with theoretical insights, enhancing our understanding of how complex systems can leverage the edge of chaos to achieve superior performance.