In long context scenarios, LLMs face three main challenges: higher computational/financial cost, longer latency, and inferior performance. Some studies reveal that the performance of LLMs depends on both the density and the position of the key information (question relevant) in the input prompt. Inspired by these findings, we propose LongLLMLingua for prompt compression towards improving LLMs' perception of the key information to simultaneously address the three challenges. We conduct evaluation on a wide range of long context scenarios including single-/multi-document QA, few-shot learning, summarization, synthetic tasks, and code completion. The experimental results show that LongLLMLingua compressed prompt can derive higher performance with much less cost. The latency of the end-to-end system is also reduced. For example, on NaturalQuestions benchmark, LongLLMLingua gains a performance boost of up to 17.1% over the original prompt with ~4x fewer tokens as input to GPT-3.5-Turbo. It can derive cost savings of \$28.5 and \$27.4 per 1,000 samples from the LongBench and ZeroScrolls benchmark, respectively. Additionally, when compressing prompts of ~10k tokens at a compression rate of 2x-10x, LongLLMLingua can speed up the end-to-end latency by 1.4x-3.8x. Our code is available at https://aka.ms/LLMLingua.
LongLLMLingua introduces a prompt compression technique to address the challenges LLMs face when processing extensive contexts, thereby reducing computational and financial costs and improving performance.
The methodology includes question-aware coarse-to-fine compression, document reordering mechanism, dynamic compression ratios, and post-compression subsequence recovery to maintain information integrity.
Experimental results demonstrated performance improvements, substantial cost savings, and latency reductions, making LongLLMLingua suitable for cost-sensitive and latency-critical applications involving long context scenarios.
The paper "LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression" addresses significant challenges faced by LLMs when processing extensive context. These challenges include higher computational and financial costs, longer latency, and degraded performance. Previous studies have noted that the effectiveness of LLMs is influenced by the density and position of key information within the input. Building on these insights, LongLLMLingua proposes a technique for prompt compression to enhance the perception of key information by LLMs, thus alleviating the identified challenges.
The paper makes several notable contributions:
Question-Aware Coarse-to-Fine Compression:
Document Reordering Mechanism:
Dynamic Compression Ratios:
Post-Compression Subsequence Recovery:
The problem is framed as an optimization problem where the goal is to compress a given prompt while preserving the distribution of the target LLM's output as close as possible to the distribution obtained from the original prompt. The process incorporates both token-level subsequence selection and document reordering.
Coarse-Grained Compression:
The authors calculate a relevance score r_k
for each document in the prompt using question-conditioned perplexities. Irrelevant documents are discarded to reduce noise in the compressed prompt.
Fine-Grained Compression:
The token-level importance within the documents retained after coarse compression is calculated using contrastive perplexity. This ensures that only the most relevant tokens are retained, further compressing the prompt.
Document Reordering:
Documents are reordered based on their relevance scores to position the most pertinent information at the beginning or end of the prompt, where LLMs are more effective.
Subsequence Recovery:
During response generation, a token-level subsequence recovery method is employed to correct potential distortions in key information caused by token removal, thus improving the accuracy and reliability of the LLM’s output.
The efficacy of LongLLMLingua was evaluated on several benchmarks encompassing various long context scenarios, including multi-document QA, few-shot learning, summarization, synthetic tasks, and code completion. Key results include:
Performance Improvement:
On the NaturalQuestions benchmark, LongLLMLingua achieved performance gains of up to 17.1% over the original prompt with approximately 4x fewer input tokens to GPT-3.5-Turbo.
Cost Savings:
LongLLMLingua demonstrated substantial financial savings, reducing inference costs by \$28.5 and \$27.4 per 1,000 samples on the LongBench and ZeroScrolls benchmarks, respectively.
Latency Reduction:
When prompts of approximately 10,000 tokens were compressed at rates between 2x to 10x, the end-to-end latency was reduced by 1.4x to 3.8x.
LongLLMLingua has practical implications for efficiently deploying LLMs in cost-sensitive and latency-critical applications, particularly those involving long context scenarios such as extensive document retrieval, legal text analysis, and scientific literature summarization.
The work provides insights into the importance of information structuring within prompts and suggests further exploration into optimizing information retrieval and alignment techniques. The proposed methods could be extended to other domains, such as employing different kinds of conditioning for relevance score calculations.
Future research may focus on integrating LongLLMLingua with other LLM frameworks to further improve its applicability and efficiency. Additionally, the development of more sophisticated relevance metrics and advanced sequence recovery techniques could enhance performance further.
LongLLMLingua provides a sophisticated approach to managing long contexts in LLMs, addressing both efficiency and performance issues through innovative prompt compression techniques. The experimental results affirm the method's efficacy, and its broader applicability suggests significant potential for optimizing LLM performance in various real-world applications.