How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model (2305.00586v5)

Abstract: Pre-trained LLMs can be surprisingly adept at tasks they were not explicitly trained on, but how they implement these capabilities is poorly understood. In this paper, we investigate the basic mathematical abilities often acquired by pre-trained LLMs. Concretely, we use mechanistic interpretability techniques to explain the (limited) mathematical abilities of GPT-2 small. As a case study, we examine its ability to take in sentences such as "The war lasted from the year 1732 to the year 17", and predict valid two-digit end years (years > 32). We first identify a circuit, a small subset of GPT-2 small's computational graph that computes this task's output. Then, we explain the role of each circuit component, showing that GPT-2 small's final multi-layer perceptrons boost the probability of end years greater than the start year. Finally, we find related tasks that activate our circuit. Our results suggest that GPT-2 small computes greater-than using a complex but general mechanism that activates across diverse contexts.

• The paper identifies a minimal circuit within GPT-2 responsible for executing greater-than tasks using detailed path patching interventions.
• It utilizes a year-span prediction task to demonstrate an 81.7% probability difference in correctly inferring greater-than relationships.
• The study reveals that specific attention heads and MLP units systematically contribute to GPT-2’s latent arithmetic, enhancing AI interpretability methods.

Interpreting Mathematical Abilities in GPT-2: A Case Study of the Greater-Than Operation

The paper addresses the often underexplored mechanistic capabilities of pre-trained LLMs, specifically focusing on GPT-2 small, a widely recognized model in the broader discussion of NLP. Despite its relatively small architecture, this model demonstrates latent mathematical abilities that invite further scrutiny. The research in question zeroes in on GPT-2’s capability to execute a greater-than function within textual contexts and elaborates on the intricacies of these operations by employing a methodologically detailed analysis.

Investigative Approach and Key Findings

The investigation is anchored on the conceptual framework of circuits—an analytical lens focusing on specific subgraphs of a model's computational structure. The research meticulously progresses through identifying a "circuit," or a minimal subset of its computational graph that handles the greater-than task, and elucidating how different components of GPT-2's layers contribute to this arithmetic functionality.

Experimentally, the paper introduces "year-span prediction" as a task whereby the model receives input in a structured form—e.g., "The war lasted from the year 1732 to the year 17"—with the expectation that GPT-2 will assign a higher probability to numbers greater than 32 as the sentence's continuation. This task is instrumental in probing latent math abilities by evaluating whether the model's internal mechanisms can naturally conclude a greater-than relationship.

Quantitatively, GPT-2 demonstrates substantial performance, with a probability difference of 81.7% and a cutoff sharpness of 6%, affirming its tendency to compute correctly in proposed greater-than scenarios. These findings underscore the nuances in GPT-2's behavior, harnessing a blend of memorization and contextual reasoning.

Mechanistic Interpretability and Circuit Construction

Central to the interpretive methodology is path patching, a detailed causal intervention strategy that specifies how specific model components—particularly attention heads and MLPs—interact within the identified circuit. This technique allows for granular isolation of a component's effect by altering inputs and dissecting the resultant computational flow. The paper's nuanced path patching results reveal that the selected MLPs (primarily 8-11) and specific attention heads are key players in the task of transmuting input data into appropriate year predictions.

Attention heads are noted for their propensity to focus on the decisive year components, while MLPs are methodically deduced to manage the scalar comparisons, effectively composing the greater-than operation across multiple neurons. The logit lens analysis further verifies that individual neurons are implicated in various stages of computation, with neurons showing distinct activation patterns pertinent to greater-than logic.

Generalization and Application Contexts

An intriguing facet of the paper is the exploration of circuit generalization. The findings indicate that while the established circuit correctly handles other, non-trivial mathematical prompt variations—predicting higher subsequent values where applicable—GPT-2's circuit sometimes misconstrues scenarios requiring less-than logic, evidencing overgeneralization. This inclination suggests that while GPT-2 harbors some degree of mathematical abstraction, it neither fully embodies nor mimics a genuine mathematical reasoning process.

Theoretical and Practical Implications

The research contributes to an enhanced understanding of how latent arithmetic operations manifest in LLMs, framed through a mechanistic lens. The paper augments the discourse on AI interpretability, highlighting the extent of GPT-2's innate competence and the structural encoding of computational logic. Looking forward, these insights harbor potential for advancing interpretability techniques and fine-tuning methodologies within broader AI tasks, compelling the AI community to revisit assumptions about the boundaries between memorization and true generalization.

In conclusion, the paper not only enriches our comprehension of mathematical processing within neural architectures but also fosters crucial dialogues on the implications of these internal dynamics. Such work prompts further methodological and theoretical advancements that can bridge the gaps in our understanding of sophisticated LLM operations.