RULER: What's the Real Context Size of Your Long-Context Language Models? (2404.06654v3)

Abstract: The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context LMs. However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.

• The paper introduces a novel synthetic benchmark called Ruler to evaluate long-context language models across diverse tasks beyond simple retrieval.
• The paper evaluates ten long-context models up to 128K tokens, revealing discrepancies between claimed and effective context sizes with significant performance degradation.
• The paper demonstrates that non-retrieval tasks, especially multi-hop tracing and aggregation, suffer as context length increases, with GPT-4 being the notable exception.

"RULER: What's the Real Context Size of Your Long-Context LLMs?" (2404.06654)

Introduction

The paper introduces "Ruler," a synthetic benchmark specifically designed to evaluate the long-context capabilities of LMs. Traditional evaluation methods like the needle-in-a-haystack (NIAH) test focus primarily on retrieval capabilities, leaving other aspects of long-context understanding unexplored. Ruler encompasses a suite of tasks aimed at challenging these models beyond mere retrieval, including multi-hop tracing, aggregation, and question answering (QA). The benchmark's flexibility in task configuration enables a comprehensive assessment across different sequence lengths and complexities.

Benchmark Structure

Ruler comprises four major task categories:

1. Retrieval (NIAH Extension): This category extends the vanilla NIAH test by introducing diverse needles and distractor setups, necessitating retrieval proficiency across various contexts.
2. Multi-hop Tracing (Variable Tracking): This task emulates coreference chain resolution, requiring models to track entity references over long contexts.
3. Aggregation (Common and Frequent Word Extraction): Tasks in this category simulate summarization, evaluating a model's ability to aggregate dispersed, relevant information.
4. Question Answering (QA): By augmenting existing short-context QA datasets with distracting information, these tasks test models' QA capabilities at scale.

Figure 1: In aggregation tasks, we sample words from a vocabulary following the two distributions above. The common words extraction (CWE) samples from uniform distributions. In the frequent words extraction (FWE), the frequency of each word is determined by its rank in the vocabulary and the parameter alpha of Zeta distribution.

Experimental Setup

The researchers benchmarked ten long-context LMs, including prominent models such as GPT-4, Command-R, and Yi-34B. The models were assessed on 13 representative tasks across varying complexity levels and context lengths up to 128K tokens. Using a combination of static thresholds and weighted averaging, the paper establishes "effective length" as a measure of a model's contextual performance capability.

Figure 2: Performance of Yi-34B in the needle-in-a-haystack (NIAH) tasks. By default, we use word-number as the key-value pair and Paul Graham essays as the haystack. Yi is not robust to the change of needle types and degrades with the increasing amount of distractors. (W: words; N: numbers; U: UUIDs; Full: entire haystack).

Key Findings

The benchmark revealed significant discrepancies between claimed and effective context sizes, with all models exhibiting performance degradation as input length increased. Notably, only GPT-4 maintained satisfactory performance at 128K tokens. The paper also demonstrated that greater model parameter size and training context length generally correlates with improved performance in long-context scenarios.

Figure 3: Performance of Yi-34B in variable tracking (VT), frequent words extraction (FWE), and QA tasks across different task complexities. Yi shows large degradation and distinct trends with scaled context size in these non-retrieval tasks, demonstrating the need to evaluate behavior beyond retrieval from context.

Task-Specific Insights

• Retrieval (NIAH): All models showed perfect scores on straightforward passkey retrieval tasks, but performance declined considerably as tasks incorporated hard distractors or required multi-value retrieval.
• Multi-hop Tracing: Models struggled with reliably tracing variable bindings through complex chains, especially as distractor tasks increased.
• Aggregation: Performance varied significantly based on the input word distribution, with models frequently misjudging word frequencies at large context sizes.
• QA: Models frequently hallucinated answers, indicating reduced reliance on the provided context, a behavior attributed to the increased length.

Model Analysis

An analysis of non-Transformer architectures (e.g., Mamba, RWKV) highlighted their inferiority compared to Transformer-based models in length extrapolation tasks. Furthermore, increasing the base frequency in RoPE positively correlated with enhanced length extrapolation capabilities.

Figure 4: (Left {content} middle left): Comparison of LargeWorldModel (LWM) series trained up to various context sizes with fixed parameter size of 7B. (Middle right): Comparison of Yi suite models with different parameter sizes with controlled training context length of 200K. (Right): Performance of non-Transformer architectures lags behind the Transformer baseline Llama2-7B by large margin. Length extrapolation is presented with dashed lines.

Conclusion

The introduction of Ruler sets a new standard for comprehensive evaluation of long-context LMs by incorporating tasks that extend beyond basic retrieval. The benchmark exposes the limitations of current models, especially regarding their effective context utilization and the gap between claimed and actual performance at longer sequences. Ruler will likely spur further research and development in models capable of handling truly extended contexts, ultimately enhancing their applicability in real-world scenarios requiring long-range dependencies.