Physics of Language Models: Part 1, Learning Hierarchical Language Structures (2305.13673v4)

Abstract: Transformer-based LLMs are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge. Previous research has primarily explored how these models handle simple tasks like name copying or selection, and we extend this by investigating how these models perform recursive language structure reasoning defined by context-free grammars (CFGs). We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences (e.g., hundreds of tokens) that are locally ambiguous and require dynamic programming to parse. Despite this complexity, we demonstrate that generative models like GPT can accurately learn and reason over CFG-defined hierarchies and generate sentences based on it. We explore the model's internals, revealing that its hidden states precisely capture the structure of CFGs, and its attention patterns resemble the information passing in a dynamic programming algorithm. This paper also presents several corollaries, including showing why absolute positional embeddings is inferior to relative and rotary embeddings; uniform attention alone is surprisingly effective (motivating our follow-up work on Canon layers); encoder-only models (e.g., BERT, DeBERTa) struggle with deep structure reasoning on CFGs compared to autoregressive models (e.g., GPT); and injecting structural or syntactic noise into pretraining data markedly improves robustness to corrupted language prompts.

• The paper demonstrates transformers’ ability to learn and generate valid CFG strings, achieving near-perfect accuracy and high output diversity.
• The study reveals that hidden states encode hierarchical non-terminal structures through linear probing, mirroring dynamic programming strategies.
• The research shows that boundary-based attention and error-correcting mode-switches enhance robustness, yielding improved performance on noisy data.

Physics of LLMs: Part 1, Learning Hierarchical Language Structures

This paper explores how modern generative LLMs, particularly transformers, learn and encode context-free grammars (CFGs). CFGs have a tree-like structure and play a crucial role in modeling various structured expressions in language, such as grammar, coding, and logic.

Context-Free Grammars in LLMs

CFGs consist of terminal symbols, non-terminal symbols, a root symbol, and production rules. They can generate complex expressions, including grammars of natural languages and mathematical expressions. LLMs like GPT are trained to predict the next token in a sequence, learning the probabilistic dependencies and structures inherent in language.

Transformer's Learned Representations

Generative Capability

The paper describes synthetic CFG datasets to paper the transformer's ability to generate valid CFG strings. Performance metrics such as completion accuracy, diversity measured by entropy, and KL divergence are used to evaluate the model's proficiency. The results demonstrate that transformers can achieve near-perfect accuracy and high output diversity, indicating an understanding of CFG rules beyond memorization.

Encoding Hierarchical Structures

A key finding is that the transformer's hidden states encode NT ancestor and boundary information, similar to dynamic programming (DP) solutions used in CFG parsing. Linear probing experiments reveal these structures are learned hierarchically across the layers of the transformer model, starting with shallow structures and progressing to deeper levels.

Attention Mechanisms in Learning

Position and Boundary-Based Attention

Transformers exhibit position-based attention preferences, with layers and heads attending to tokens based on their relative positions. Importantly, boundary-based attention is observed, where tokens at non-terminal boundaries tend to attend to adjacent boundaries, mimicking memory links used in DP algorithms for CFG parsing.

Dynamic Programming Analogies

The attention patterns suggest transformers implement a form of dynamic programming, where information is stored and recurrently accessed through attention weights corresponding to NT boundaries. This enables efficient parsing and generation of CFG-derived sequences.

Extension to Implicit CFGs

The paper extends to implicit CFGs, where terminal symbols account for a distribution of overlapping tokens. Transformers learn these implicit CFGs by encoding terminal symbol information in token embeddings, indicating adaptability to even more complex linguistic structures.

Robustness and Error Correction

Testing robustness, models trained on perturbed data with grammatical errors show improved recovery and generation accuracy on corrupted input, demonstrating adaptability. The phenomenon of a learned 'mode switch' for handling errors suggests practical benefits in incorporating noisy data during pre-training.

Conclusion and Implications

This research elucidates the mechanisms by which transformers learn CFGs, revealing parallels with dynamic programming methods. The findings provide insight into improving LLM architectures and training strategies, applying these models to complex hierarchical structures. Future research directions include exploring context-sensitive grammars and domain-specific adaptations.