Efficiently serving LLMs requires batching of many requests to reduce the cost per request. Yet, with larger batch sizes and longer context lengths, the key-value (KV) cache, which stores attention keys and values to avoid re-computations, significantly increases memory demands and becomes the new bottleneck in speed and memory usage. Additionally, the loading of the KV cache causes the computational core to be idle, which limits the inference speed. A straightforward and effective solution to reduce KV cache size is quantization, which decreases the total bytes taken by KV cache. However, there is a lack of in-depth studies that explore the element distribution of KV cache to understand the hardness and limitation of KV cache quantization. To fill the gap, we conducted a comprehensive study on the element distribution in KV cache of popular LLMs. Our findings indicate that the key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In contrast, the value cache should be quantized per-token. From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $\mathbf{2.6\times}$ less peak memory (including model weight). This reduction in memory usage enables up to $\mathbf{4\times}$ larger batch size, bringing $\mathbf{2.35\times \sim 3.47\times}$ throughput on real LLM inference workload. The source code is available at https://github.com/jy-yuan/KIVI.
The paper introduces KIVI, a tuning-free 2bit quantization algorithm for key-value (KV) cache of LLMs, aiming to reduce memory and computational overhead without affecting model accuracy.
KIVI uses an asymmetric quantization approach, quantizing key cache per-channel and value cache per-token, leading to significant reductions in peak memory usage and improvements in throughput.
Extensive experiments validate KIVI's efficiency and accuracy, demonstrating up to 2.6x reduction in peak memory usage and 3.47x increase in throughput, supporting its robustness without requiring complex hyperparameter tuning.
The research paper titled "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache" addresses the core issue of efficient deployment of LLMs. The paper focuses on optimizing the key-value (KV) cache used during batch processing to significantly reduce memory and computational overhead while maintaining model accuracy. The innovative approach proposed in the paper revolves around the quantization of KV cache, allowing for reduced memory footprint and enhanced throughput without necessitating complex hyperparameter tuning.
Comprehensive Analysis of KV Cache Distribution:
Proposed Methodology - KIVI:
Hardware-Friendly Implementation:
Experimental Validation:
Accuracy and Efficiency:
Memory and Throughput:
Detailed Ablation Studies:
The research has significant implications in both theoretical and practical domains:
Theoretical Insights:
Practical Deployments:
The paper lays a strong foundation for future research in several areas:
Integration with Other Techniques:
Broadening Applicability:
Optimization of Implementation:
The paper makes significant strides in the efficient deployment of LLMs by addressing the KV cache bottleneck through innovative quantization strategies. KIVI offers a practical, tuning-free solution that enhances memory efficiency and throughput without compromising on accuracy. This work paves the way for the broader adoption of large-scale neural networks in resource-constrained environments, bringing us closer to deploying advanced AI models in everyday applications.