Understanding Emergent Abilities of Language Models from the Loss Perspective
(2403.15796)Abstract
Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks. We also discover that a model exhibits emergent abilities on certain tasks -- regardless of the continuity of metrics -- when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.
Overview
-
This paper challenges the conventional belief that emergent abilities in language models (LMs) are solely tied to model size or training compute, proposing a new focus on pre-training loss.
-
It is discovered that language models with identical pre-training losses exhibit similar performances on various tasks, suggesting pre-training loss as a reliable predictor of LM capability.
-
A distinction is made between datasets showing smooth performance improvement and those displaying emergent abilities only beyond a specific pre-training loss threshold.
-
A novel definition of emergent abilities is introduced, highlighting the importance of pre-training loss levels in the manifestation of such abilities.
Understanding Emergent Abilities of Language Models from the Loss Perspective
Introduction
The scalability of language models (LMs), concerning both model and data sizes, has been pivotal in enhancing performance across a wide range of tasks, leading to significant advancements in LM applications. This success is principally driven by scaling laws, which predict pre-training loss based on model size and training data size. However, the conventional belief that emergent abilities—a set of abilities observed in large LMs but absent in smaller counterparts—are intrinsically tied to model size or training compute has been recently called into question. This skepticism arises from observations that smaller models can outperform larger ones on emergent tasks and doubts about the metrics used to evaluate these abilities.
In response to these challenges, this paper shifts the focus to examining emergent abilities through the lens of pre-training loss rather than conventional metrics like model size or training compute. Our findings reveal that models with identical pre-training losses, irrespective of their model or data sizes, display equivalent performances across various downstream tasks. Furthermore, we ascertain that emergent abilities on specific tasks are observable regardless of metric continuity once the pre-training loss dips below a certain threshold. Prior to reaching this threshold, performance remains akin to random guessing. This insight compels us to propose a new definition of emergent abilities, emphasizing the crucial role of pre-training loss in their manifestation.
The Predictive Power of Pre-training Loss
Through extensive experimentation involving over 30 LMs of varying sizes pre-trained from scratch, we examine the relationship between pre-training loss and downstream task performance. Our study spans 12 diverse datasets covering different tasks, languages, prompting types, and answer forms. The results consistently indicate that the pre-training loss of an LM is a reliable predictor of its performance on downstream tasks, independent of its specific configuration. This finding is further validated by analyzing the performance and loss relationship of the LLaMA model series, which was trained under different conditions but exhibited similar trends.
Task Performance Analysis
Our analysis distinguishes between two groups of datasets based on their performance trends: those that show smooth improvement with decreasing pre-training loss and those that exhibit emergent performance improvements only after crossing a specific loss threshold. Remarkably, this threshold appears to be consistent across tasks that display emergent behavior, suggesting a uniform tipping point for the activation of emergent abilities. Our investigation into the influence of different metrics, including continuous metrics like the Brier Score, confirms that the phenomenon of emergent abilities persists irrespective of metric continuity.
A New Definition of Emergent Abilities
Encouraged by our observations, we introduce a novel definition of emergent abilities centered on pre-training loss. This definition frames emergent abilities as capabilities absent in models with higher pre-training losses but present in those with lower pre-training losses. We argue that this perspective provides a more exact characterization of emergent abilities, emphasizing the critical junctures at which they become apparent within training trajectories. This approach holds the promise of guiding future research endeavors aimed at understanding and leveraging these pivotal moments in the development of LMs.
Conclusion
In conclusion, our research offers a fresh perspective on the study of emergent abilities in LMs by centering on pre-training loss as a fundamental indicator. The evidence presented supports the existence of emergent abilities as distinct phenomena observable when LMs achieve certain pre-training loss thresholds. This insight not only challenges previous understandings of emergent abilities but also paves the way for new directions in AI research focused on exploring the latent capabilities of LMs beyond traditional scaling methods.
Create an account to read this summary for free: