gzip Predicts Data-dependent Scaling Laws (2405.16684v1)

Abstract: Past work has established scaling laws that predict the performance of a neural LLM (LM) as a function of its parameter count and the number of tokens it's trained on, enabling optimal allocation of a fixed compute budget. Are these scaling laws agnostic to training data as some prior work suggests? We generate training datasets of varying complexities by modulating the syntactic properties of a PCFG, finding that 1) scaling laws are sensitive to differences in data complexity and that 2) gzip, a compression algorithm, is an effective predictor of how data complexity impacts scaling properties. We propose a new data-dependent scaling law for LM's that accounts for the training data's gzip-compressibility; its compute-optimal frontier increases in dataset size preference (over parameter count preference) as training data becomes harder to compress.

• The paper demonstrates that data complexity, measured via gzip compressibility, significantly alters traditional scaling laws for language models.
• The study employs PCFG-generated datasets to systematically vary syntactic complexity, revealing a shift towards favoring larger datasets over increased parameter counts.
• The findings highlight practical benefits, including potential GPU-hour savings, and suggest tailored scaling strategies for specialized domains like code generation.

The Influence of Data Complexity on Scaling Laws for LLMs

Introduction

The research explores the sensitivity of scaling laws for neural LLMs (LMs) to the complexity of training data. Traditionally, scaling laws have provided a framework for predicting the performance of LMs based on the model's parameter count and the number of training tokens. These laws guide optimal resource allocation within a fixed compute budget. However, the prevalent assumption has been that these scaling laws are largely agnostic to the specifics of the training data distribution. This paper challenges that assumption and introduces gzip-compressibility as a quantifiable metric of data complexity that influences scaling behavior.

Methodology

To interrogate the relationship between data complexity and scaling laws, the paper employs Probabilistic Context-Free Grammars (PCFGs) to generate datasets with varying syntactic complexities. Six distinct datasets are created, each modulated by different syntactic properties such as the number of production rules and non-terminals. These datasets facilitate a controlled environment where the complexity of the data is systematically adjustable. The complexity is measured using gzip, a compression algorithm, thus providing a proxy for data entropy.

LLMs of six different sizes, ranging from 4.4 million to 1.4 billion parameters, are trained on these datasets. Training steps vary across multiple magnitudes, enabling the researchers to fit scaling laws to the performance results for each dataset, exploring how these laws shift with increased data complexity.

Numerical Results

One of the pivotal findings is that as training data becomes less compressible (i.e., more complex), the optimal compute frontier shifts to favor dataset size over model parameter count. This is contrary to the traditional equal scaling of parameters and data tokens, as recommended by Chinchilla scaling laws. Specifically, the paper notes that for high-complexity data, achieving similar performance requires a relatively larger dataset and fewer parameters. This nuanced allocation can lead to significant computational savings, illustrated by the estimation that an adjusted compute allocation could save $278,000 in GPU hours when training a model like StarCoder.

Theoretical and Practical Implications

The theoretical implications point toward a broader principle where data complexity, quantified through gzip-compressibility, should inform scaling laws. This raises critical questions about the underlying mechanisms that cause different scaling behaviors across datasets of varying complexity. Practically, these findings are highly consequential for domains such as code generation, where datasets exhibit significantly different compressibility profiles compared to natural language.

For instance, code datasets generally have lower gzip-compressibility, necessitating different scaling strategies—favoring parameters over data. This insight has practical ramifications for optimizing resource allocation in training LMs, rendering the training of large models more efficient and cost-effective.

Future Directions

Future work could delve into theoretical explanations for why scaling laws are sensitive to data complexity. Investigations could leverage linguistics and information theory to demystify the relationship between entropy, syntactic complexity, and LM performance. Empirical validation on real-world datasets beyond PCFGs would further bolster the generalizability of these findings. Additionally, extending this approach to other modalities, such as images or audio, could reveal if and how these principles apply more broadly across AI domains.

Exploring alternative complexity metrics or compression algorithms might yield even finer adjustments to scaling laws for more effective training regimen design. Real-world applications could involve dynamic compute resource allocation based on real-time data complexity assessments, enhancing both training efficiency and model performance.

Conclusion

The paper makes a significant contribution by demonstrating that the prevailing assumption of data-agnostic scaling laws is insufficient. It positions gzip-compressibility as a viable measure to adjust these laws based on data complexity, potentially leading to more optimal and cost-effective training regimes, particularly in specialized contexts like code generation. This paradigm shift urges the AI research community to reconsider how we design and allocate compute resources for training large-scale LLMs.