Understanding Emergent Abilities of Language Models from the Loss Perspective (2403.15796v3)

Abstract: Recent studies have put into question the belief that emergent abilities in LLMs are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities on certain tasks -- regardless of the continuity of metrics -- when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.

• The paper redefines emergent abilities by showing that pre-training loss, not model size, predicts downstream task performance.
• It demonstrates a threshold effect where performance improves significantly once a critical low loss is achieved.
• Experiments across various model scales confirm that optimized pre-training loss is key to unlocking novel LLM capabilities.

Understanding Emergent Abilities of LLMs from the Loss Perspective

The paper "Understanding Emergent Abilities of LLMs from the Loss Perspective" investigates the enigmatic phenomena of emergent abilities exhibited in LLMs. It challenges the conventional wisdom that these emergent capabilities are solely a function of model or data size and proposes a novel exploration through the lens of pre-training loss.

Emergent Abilities and Pre-training Loss

Emergent abilities in LLMs refer to capabilities that become apparent in larger models but are absent in their smaller counterparts. This work argues against the prevailing belief that these abilities are size-dependent. Instead, it suggests that the pre-training loss—a metric indicative of a model's learning status—serves as a more relevant predictor of emergent abilities.

Key Observations

1. Loss as a Predictor: The paper demonstrates that models with the same pre-training loss achieve comparable performance on downstream tasks, irrespective of their size. This finding implies that pre-training loss, rather than sheer model size or compute, is predictive of downstream task performance.
2. Threshold Effect: Emergent abilities appear when the pre-training loss falls below a certain threshold. Before reaching this threshold, model performance aligns with random guessing, suggesting a phase transition akin to grokking once this critical pre-training loss value is surpassed.

   Figure 1: The performance-vs-loss curves of 1.5B, 6B, and 32B models indicating the threshold beyond which performance improves significantly.

3. Verification on Different Models: The relationship between pre-training loss and task performance was validated through experiments with models of various sizes, including the LLaMA model series, confirming that the observed trends are robust and not confined to specific architectures or datasets.

   Figure 2: The performance-vs-loss curves of smaller models pre-trained with different numbers of training tokens.

Defining Emergent Abilities

The paper redefines emergent abilities from the loss perspective as abilities that manifest in models achieving lower pre-training loss. The emergent capabilities cannot be anticipated by extrapolating performance trends from models with higher loss values. This reconceptualization implies that previously observed size-related emergent properties may be better explained by examining pre-training loss milestones.

Figure 3: The performance-vs-loss curves of LLaMA models across various downstream tasks, reinforcing the consistency of the loss-performance relationship.

Implications and Future Directions

This paper provides a critical insight into the potential of using pre-training loss as a proxy for predicting the emergence of new capabilities. By doing so, it invites a reconsideration of scaling strategies solely based on enlarging model architectures. Further exploration into optimizing training dynamics and learning frameworks to strategically reduce pre-training loss might unlock new abilities in LLMs.

Figure 4: The performance-vs-loss curves of different metrics on MMLU and C-Eval, indicating continuous metrics do not negate the tipping point effect.

Future Work

Future research could explore understanding how tokenizer selection and corpus distribution impact pre-training loss and task performance, potentially paving the way for more data-efficient strategies to reach desired performance thresholds. Additionally, beyond pre-training, the role of fine-tuning regimes and their interplay with pre-training loss in shaping emergent abilities could be a promising area for exploration.

Conclusion

The definitive insight from this work is that pre-training loss serves as a crucial metric in understanding and predicting the capabilities of LLMs, enabling a departure from traditional model-size-centric views of emergent abilities. This provides a roadmap for future optimizations in the training of LLMs, focusing on achieving lower loss values to elicit novel capabilities. By redefining emergent abilities in terms of loss, this paper offers a structural framework for future advancements in the field of LLM development and application.