DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2401.02954v1)

Abstract: The rapid development of open-source LLMs has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source LLMs with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

• The paper presents DeepSeek LLM which scales open-source language models using scaling laws derived from non-embedding FLOPs/token metrics.
• It employs a novel multi-step learning rate scheduler and advanced techniques like Rotary Embedding and Grouped-Query Attention to optimize performance.
• The study demonstrates that balancing model and data scaling through computed IsoFLOP curves yields superior benchmarks compared to models such as LLaMA-2 70B.

DeepSeek LLM: Scaling Open-Source LLMs with Longtermism

The paper presents "DeepSeek LLM," a project focused on the scaling of open-source LLMs, with a specific emphasis on models of two different scales: 7 billion (7B) and 67 billion (67B) parameters. The paper is centered around the derivation and application of scaling laws intended to guide the effective scaling of LLMs within computational and data constraints.

Introduction to Scaling Laws and DeepSeek LLM

The paper begins by addressing the variability in conclusions from prior research on scaling laws, which are critical for understanding the relationship between model performance, model size ($N$), dataset size ($D$), and compute budget ($C$). These scaling laws guide developers in optimizing resource allocation among models and data as compute budgets increase, aiming for AGI.

DeepSeek LLM contributes to this field by exploring these scaling behaviors to build models that might surpass existing standards like LLaMA in various benchmark settings. By pre-training on a large dataset with 2 trillion tokens and using strategic fine-tuning approaches, DeepSeek aims to construct models that excel in diverse tasks, outperforming competitors such as LLaMA-2 70B in metrics for code, math, and reasoning.

Pre-Training and Model Architecture

Data Processing

Deduplication, filtering, and remixing are crucial stages in ensuring data quality and diversity in training datasets. The deduplication strategy highlighted has demonstrated the importance of cross-dump analysis in removing duplicates more effectively than within single data dumps. This helps in maximizing the uniqueness and relevance of training instances.

Model Architecture

DeepSeek LLM adopts architectural designs akin to LLaMA's Pre-Norm structure, employing RMSNorm and SwiGLU for normalization and activation functions, respectively, while incorporating advanced techniques like Rotary Embedding and Grouped-Query Attention (GQA) for optimized inference. The distinctions lie in the macro design where DeepSeek models modify layer counts to stratify resource distribution effectively.

Detailed Model Specifications:

Parameters are adjusted, with 7B models utilizing 30 layers and 67B models using 95 layers, balancing depth and inference cost-effective scalability.

Hyperparameter Optimization

The paper emphasizes the choice of hyperparameters, highlighting the use of a novel multi-step learning rate scheduler over the conventional cosine scheduler.

Figure 1: Training loss curves with different learning rate schedulers or different parameters for schedulers. The model size is 1.6 billion parameters, trained on a dataset of 100 billion tokens.

The multi-step LR scheduler was shown to be beneficial in allowing consistent performance across varied training scales without necessitating complete retraining, which enhances its utility in incremental training scenarios.

Scaling Laws Evaluation

Deriving Optimal Model/Data Scaling Strategies

DeepSeek LLM investigates the representation of model scale using non-embedding FLOPs/token, aiming for precision in scaling predictions and thus improving on traditional model parameter metrics that include potential computational overhead biases.

Figure 2: IsoFLOP curve and optimal model/data allocation. The metric in IsoFLOP curve is bits-per-byte on the validation set. The dotted lines in optimal model/data scaling curves represent the power law fitting the smaller model (grey circles).

The IsoFLOP method allows efficient fitting of scaling curves, providing insight into optimal allocation strategies between model size and data scale.

Figure 3: Performance scaling curve. The metric is the bits-per-byte on the validation set. The dotted line represents the power law fitting the smaller model (grey circles). The blue stars represent DeepSeek LLM 7B and 67B. Their performance is well-predicted by the scaling curve.

The performance of DeepSeek LLM 7B and 67B models aligns with the predictions from these scaling laws, showcasing its applicability for scaling guidance.

Impact of Data Quality

Analysis of data sets indicated that higher-quality data necessitates allocating more resources towards model scaling rather than data expansion.

Alignment and Fine-Tuning

The DeepSeek project explores strategies like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to enhance model capabilities in dialogue and open-ended task settings. This phase emphasizes the transition from model pre-training to tailored applications in chat models, leading to superior open-field performance compared to contenders like GPT-3.5.

Evaluation Results

DeepSeek LLM outperforms other models such as LLaMA-2 70B in multiple benchmarks, emphasized by a more significant performance leap with the 67B model, suggesting notable sophistication gains with scale. This is further confirmed through evaluations on standardized tests, math reasoning benchmarks, and in-house test datasets.

Conclusion

DeepSeek LLM marks a significant step towards refining open-source LLMs with a focus on scalable, guided improvements. Its insights into data and model scaling laws not only enhance performance in benchmark settings but also optimize resource use for AGI progression. Future iterations aim to expand dataset quality and explore advanced alignment strategies, indicating ongoing contributions to open-source AI research.