The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
(2401.06751)Abstract
How can we train models to perform well on hard test data when hard training data is by definition difficult to label correctly? This question has been termed the scalable oversight problem and has drawn increasing attention as language models have continually improved. In this paper, we present the surprising conclusion that current language models often generalize relatively well from easy to hard data, even performing as well as "oracle" models trained on hard data. We demonstrate this kind of easy-to-hard generalization using simple training methods like in-context learning, linear classifier heads, and QLoRA for seven different measures of datapoint hardness, including six empirically diverse human hardness measures (like grade level) and one model-based measure (loss-based). Furthermore, we show that even if one cares most about model performance on hard data, it can be better to collect and train on easy data rather than hard data, since hard data is generally noisier and costlier to collect. Our experiments use open models up to 70b in size and four publicly available question-answering datasets with questions ranging in difficulty from 3rd grade science questions to college level STEM questions and general-knowledge trivia. We conclude that easy-to-hard generalization in LMs is surprisingly strong for the tasks studied, suggesting the scalable oversight problem may be easier than previously thought. Our code is available at https://github.com/allenai/easy-to-hard-generalization
Overview
-
The study explores the concept of easy-to-hard generalization in language models, where models trained on simple data are tested on complex tasks.
-
Criteria to determine difficulty include education level needed, expert ratings, cognitive skills, and metrics such as word count and minimum description length (MDL).
-
Results reveal that language models, including those with up to 70 billion parameters, can generalize from easy to hard data with performance nearing an oracle model.
-
Training language models on clean, easy data is suggested as a cost-effective and efficient alternative to using complex and harder-to-acquire datasets.
-
The study offers solutions to the scalability oversight problem, showing consistent performance across models of different sizes and potential for efficient model training.
Introduction
The challenge of teaching language models (LMs) to excel on difficult tasks, especially when annotating complex training data can be costly and error-prone, is a significant obstacle in the advancement of artificial intelligence. Investigating the potential of these models to learn from simpler, more accessible training data to perform complex tasks, a recent study scrutinized the concept of easy-to-hard generalization—where models are trained on "easy" data, then tested on "hard" data—to find out if and how efficiently language models can bridge the gap between these varying complexities of tasks.
Measuring Difficulty Levels
Understanding what constitutes "easy" or "hard" data is essential for studying this discrepancy. The researchers established several criteria for determining difficulty, ranging from the required grade level of education to solve a problem, to expert-determined ratings, and specific cognitive skill requirements based on Bloom's taxonomy. Additionally, metrics like the number of words in a question, the character count in an answer, or the number of reasoning steps involved, provide quantitative measures of complexity. Importantly, they included a model-based difficulty metric, minimum description length (MDL), which reflects how challenging a problem is for a model based on how much learning data it has been exposed to. These various metrics are crucial for a comprehensive assessment because no single measure can fully encapsulate what makes certain data more complex than others.
Generalization from Easy to Hard Data
In examining the performance of language models trained on easy data versus those trained on hard data, the findings were quite impressive. Across several different language models with up to 70 billion parameters and methodologies like in-context learning and linear classifier heads, the study consistently found that models generalized from easy to hard data surprisingly well. In some cases, the performance was almost as good as an "oracle" model with access to hard training data. This suggests that training on straightforward data may be sufficient for models to perform at a high level on complex tasks.
Practical Implications
When considering the practical side of data acquisition for training LMs, the study suggests a paradigm shift. Given that hard data can be more expensive to collect and potentially comes with higher rates of annotation errors, training on easy data could be a more efficient approach. The research provides evidence that cleanly labeled, easy-to-acquire data could be better for training models than investing resources in gathering more complex datasets, as long as the simpler data is of sufficient quality.
Scalability and Oversight
The scalability oversight problem revolves around how to give meaningful feedback to models when it's difficult to judge the correctness of their outputs. The study's findings imply that this challenge might not be as daunting as previously thought, since models show robust generalization capacities from easy to hard data. Furthermore, the proportion of supervision gap recovered by training on easy data remains largely consistent across models of different sizes, indicating that scalability issues do not necessarily intensify as models become more capable.
Conclusion
The study paints an encouraging picture of the flexible learning capabilities of language models. By showing that models can generalize effectively from simple to complex tasks, this work offers promising pathways for model training that are both cost-effective and resilient to challenges in data annotation. The revelation that quality supervision is possible with less complex datasets may pave the way for more efficient AI training methodologies in the future. Overall, this research marks a significant step towards solving the scalable oversight problem, providing valuable insights for both model developers and stakeholders in AI.
Create an account to read this summary for free: