Reverse Training to Nurse the Reversal Curse (2403.13799v3)

Abstract: LLMs have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue.

• The paper presents reverse training as a novel approach to mitigate the reversal curse by training LLMs in both forward and reverse directions while preserving key entities.
• The methodology employs various reversal types, including token, word, entity-preserving, and random segment reversals to double the effective training dataset.
• Experimental results across symbolic, biography, and real-world tasks show that reverse training significantly enhances LLM performance by improving bidirectional relational understanding.

Reverse Training: An Effective Mitigation Strategy for the Reversal Curse in LLMs

Introduction

Recent advancements in LLMs, such as GPT-4 and Llama-2, have significantly improved performance on various language tasks. Despite these improvements, a major flaw termed the "Reversal Curse" has been identified, which limits LLMs' ability to generalize bidirectional relational knowledge. This issue, inherent even when models are trained on extensive datasets, poses a critical challenge due to Zipf's law implications.

Addressing the reversal curse, this paper introduces "reverse training," a novel approach doubling the effective token dataset. By training LLMs in both forward and reverse directions while preserving certain substrings (e.g., entities), this method aims to enhance model performance across traditional and reversal tasks. Significant improvements are noted in resolving the reversal curse and in general task performance under certain conditions.

Reverse Training Methodology

The reverse training methodology revolves around the autoregressive training modification where inputs are manipulated to appear in reverse order. This alteration involves several reversal types, namely token, word, entity-preserving, and random segment reversal. Each type serves to adjust the granularity of reversal and maintain the integrity of specific substrings or segments in the process. Implemented within the LLM training framework, reverse training enlarges the training dataset scope and introduces a new dimension of linguistic variation for models to encapsulate.

Experimental Insights

Symbolic Reverse Task

The symbolic reverse task experiment highlights the fundamental challenge of the reversal curse within a simplified context. Here, reverse training, especially entity-preserving reversal, demonstrated a complete mitigation of the reversal curse across varying entity lengths. This finding underscores the importance of preserving the internal structure of entities or chunks within reverse training for effective learning.

Reversing Biography Task

Utilizing synthetic and real-world biography datasets, the reversing biography task further validates the efficacy of reverse training. Entity-preserving and random segment reversal were instrumental in achieving high accuracy for full-name recall in reverse tasks. These results illuminate reverse training's adaptability and potential in refining LLMs' relational understanding.

Real-World Knowledge Evaluation

Implementing reverse training in LLM pre-training showcased significant advancements in real-world knowledge capture. By effectively reducing the impact of the reversal curse, reverse training strengthens LLMs’ grasp over bidirectional facts, a critical aspect of comprehensive knowledge representation.

Implications and Future Directions

The successful implementation of reverse training presents both practical and theoretical implications. Practically, it offers a robust solution to the reversal curse, enhancing LLM performance on a spectrum of tasks without detriment to forward-direction task proficiency. Theoretically, it opens new avenues for understanding the mechanisms underlying LLMs' knowledge acquisition and generalization capabilities.

Future research may explore the integration of reverse training with other model architectures, the optimization of entity and segment preservation strategies, and the expansion of reverse training applications across diverse language domains.

Conclusion

This research marks a significant step forward in addressing the reversal curse in LLMs, proposing reverse training as an efficient strategy to enrich model understanding and performance bidirectionally. The presented experimental evidence across various tasks and datasets substantiates reverse training's effectiveness, spotlighting its potential as a foundational technique in future LLM development endeavors.