One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention (2307.03576v1)

Abstract: Recent works have empirically analyzed in-context learning and shown that transformers trained on synthetic linear regression tasks can learn to implement ridge regression, which is the Bayes-optimal predictor, given sufficient capacity [Aky\"urek et al., 2023], while one-layer transformers with linear self-attention and no MLP layer will learn to implement one step of gradient descent (GD) on a least-squares linear regression objective [von Oswald et al., 2022]. However, the theory behind these observations remains poorly understood. We theoretically study transformers with a single layer of linear self-attention, trained on synthetic noisy linear regression data. First, we mathematically show that when the covariates are drawn from a standard Gaussian distribution, the one-layer transformer which minimizes the pre-training loss will implement a single step of GD on the least-squares linear regression objective. Then, we find that changing the distribution of the covariates and weight vector to a non-isotropic Gaussian distribution has a strong impact on the learned algorithm: the global minimizer of the pre-training loss now implements a single step of $\textit{pre-conditioned}$ GD. However, if only the distribution of the responses is changed, then this does not have a large effect on the learned algorithm: even when the response comes from a more general family of $\textit{nonlinear}$ functions, the global minimizer of the pre-training loss still implements a single step of GD on a least-squares linear regression objective.

• The paper shows that one-layer linear transformers implement a one-step gradient descent by minimizing pre-training loss on linear regression tasks.
• The study demonstrates that variations in data distributions lead to distinct pre-conditioning in the gradient descent step, underlining the model's adaptability.
• The analysis reveals that even for nonlinear target functions, the optimal approach reduces to a linear gradient descent step, highlighting the architecture's inherent limitations.

Analyzing the Optimality of Single-Step Gradient Descent in Linear Transformers

This paper provides a comprehensive theoretical analysis of one-layer transformers with linear self-attention, elucidating their efficacy in performing in-context learning on linear regression tasks. It builds on empirical observations suggesting that transformers with specific architectural constraints can approximate certain algorithmic behaviors, notably single-step gradient descent (GD), when exposed to linear regression tasks with either isotropic or non-isotropic data distributions.

Theoretical Exploration of In-Context Learning

The authors begin by situating their work within the existing landscape of in-context learning, where recent studies have uncovered that transformers, under various conditions, can learn to perform linear regression tasks by conditioning on example task data. A significant aspect of this analysis is identifying the theoretical underpinnings of why a one-layer transformer with linear self-attention implements one step of GD as a global minimizer of the pre-training loss given noise and Gaussian-distributed covariates.

Key Contributions and Methodological Approaches

• Gradient Descent Optimality: The paper confirms that using synthetic linear regression data with Gaussian-distributed covariates, a transformer minimizing pre-training loss performs one step of GD on a least-squares linear regression objective. This theoretical result substantiates findings from prior empirical work, providing a mathematical framework to understand why such equivalence occurs.
• Impact of Data Distributions: The paper extends this theoretical model by demonstrating how variance in data distribution affects learned algorithms. It asserts that a non-isotropic Gaussian distribution of covariates prompts the one-layer transformer to learn a pre-conditioned GD step, adapting to the altered statistical properties of the data.
• Nonlinear Target Functions: For regression tasks where outputs derive from a nonlinear function of inputs, the analysis interestingly reveals that the optimal pre-training loss minimization method still maps to a linear GD step, emphasizing the architecture's inherent constraints and its inability to leverage complex nonlinearity.

The derivation of these results is primarily rooted in linear algebra and expectations regarding the transformation of data distributions and the properties of Gaussian noise. Particularly, the paper delineates how adjusting parameters such as covariance matrices influences loss minimization outcomes, underscoring the adaptability yet limited scope of linear transformers in modeling complex problem domains.

Implications and Speculation on Future Directions

The findings advance both practical and theoretical understandings of transformer architectures. Practically, the insights into linear self-attention layers and their GD-like behavior can inform the design of simpler models for specific tasks where computational efficiency is paramount. Theoretically, understanding these dynamics opens pathways for further explorations into the limits of transformer architectures and the exploration of more sophisticated mechanisms to model nonlinearity within limited capacity systems.

Anticipating the future landscape of artificial intelligence, this work suggests that examining model behavior under varied data conditions can significantly enhance our grasp of transformer-based architectures. Further research might involve extending these concepts to multi-layer or multi-head self-attention setups or exploring different classes of optimization algorithms within the same architectural constraints.

In summary, this paper makes a substantive contribution by providing a rigorous theoretical framework that explains the optimality of using one step of gradient descent within one-layer linear transformer models for specific regression tasks, thereby addressing gaps previously filled by empirical observations alone.