Compression Represents Intelligence Linearly (2404.09937v2)

Abstract: There is a belief that learning to compress well will lead to intelligence. Recently, language modeling has been shown to be equivalent to compression, which offers a compelling rationale for the success of LLMs: the development of more advanced LLMs is essentially enhancing compression which facilitates intelligence. Despite such appealing discussions, little empirical evidence is present for the interplay between compression and intelligence. In this work, we examine their relationship in the context of LLMs, treating LLMs as data compressors. Given the abstract concept of "intelligence", we adopt the average downstream benchmark scores as a surrogate, specifically targeting intelligence related to knowledge and commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our study brings together 31 public LLMs that originate from diverse organizations. Remarkably, we find that LLMs' intelligence -- reflected by average benchmark scores -- almost linearly correlates with their ability to compress external text corpora. These results provide concrete evidence supporting the belief that superior compression indicates greater intelligence. Furthermore, our findings suggest that compression efficiency, as an unsupervised metric derived from raw text corpora, serves as a reliable evaluation measure that is linearly associated with the model capabilities. We open-source our compression datasets as well as our data collection pipelines to facilitate future researchers to assess compression properly.

• The paper demonstrates that improved compression efficiency correlates linearly with higher benchmark scores across domains.
• It leverages language models as data compressors, using bits-per-character metrics to quantify performance.
• Empirical results show Pearson correlation coefficients near -0.95, validating compression as an unsupervised intelligence metric.

Compression Represents Intelligence Linearly

The paper "Compression Represents Intelligence Linearly" (2404.09937) presents an empirical paper examining the relationship between LLMs as data compressors and their intelligence, quantified by average benchmark scores. This investigation hypothesizes that efficient compression reflects greater intelligence, a concept grounded in the equivalence between language modeling and lossless compression.

Background: Language Modeling and Compression

The relationship between language modeling and compression is founded on the premise that any predictive model can act as a data compressor. According to previous foundational work, optimal data encoding relies on understanding the underlying data distribution ($p_{data}$) to achieve minimal average coding length through methods like arithmetic coding. However, given $p_{data}$'s lack of availability in practice, empirical substitutes such as LLMs, trained to minimize cross-entropy loss—a metric identical to compression efficiency—are employed. This equivalence suggests that training LLMs is akin to refining them as compressors, thus tying their performance to compression efficacy.

Experimental Protocols and Evaluation

Intelligence Assessment

The paper evaluates LLM intelligence based on average performance across tasks requiring knowledge and commonsense, coding, and mathematical reasoning. For each domain, suitable benchmarks such as MMLU and GSM8K were selected. Models' scores across these benchmarks serve as proxies for their domain-specific intelligence.

Compression Evaluation

Compression efficiency is measured using bits per character (BPC), providing a uniform metric across diverse models with varying tokenization schemes. Importantly, the paper adjusts for context window size—set to 1900 tokens—ensuring that evaluations reflect comparable access to context across models.

Figure 1: Correlation between the average benchmark scores and the models' compression efficiency evaluated with bits per character (BPC).

Results: Linear Correlation Between Compression and Intelligence

The paper finds a striking linear relationship between benchmark scores and compression efficiency across various models and benchmarks, reinforcing the hypothesis that greater compression efficiency indicates enhanced intelligence. In particular, Pearson correlation coefficients nearing -0.95 in domains like coding and mathematical reasoning demonstrate a consistent pattern where compression efficiency reliably predicts benchmark success.

Figure 2: Correlation between benchmark scores and BPC in the knowledge and commonsense area, showcasing the linear relationship across multiple tasks.

Discussion: Practical and Theoretical Implications

The results underscore compression efficiency as a valuable, unsupervised metric for LLM evaluation. This metric's flexibility—allowing for updating corpora—addresses concerns of data leakage and overfitting prevalent in other evaluation standards. Furthermore, the empirical validation of long-held theoretical beliefs about intelligence and compression encourages considering compression efficiency in broader AI assessment frameworks.

Potential limitations include the focus on well-trained, base models exclusively, leaving the exploration of fine-tuned models and emergent characteristics for future research. These findings, however, establish a sound foundation for subsequent studies on compression's role in AI capabilities.

Conclusion

This paper provides robust empirical evidence confirming that efficient compression captures intelligence linearly, suggesting profound implications for LLM evaluations. By validating the theoretical equivalence of language modeling and compression, the paper advocates for mainstreaming compression efficiency as an intelligence metric, facilitating robust AI evaluation and development.