QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2404.00456v2)

Abstract: We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLaMa2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLaMa2 models without any calibration data using round-to-nearest quantization. Code is available at: https://github.com/spcl/QuaRot.

• The paper introduces QuaRot, a novel quantization scheme that leverages rotational transformations to eliminate outliers for effective 4-bit inference.
• It employs randomized Hadamard transformations on weights and activations, achieving up to 2.16x prefill speedups and 3.39x memory savings.
• Empirical results maintain 99% zero-shot performance and under 0.29 perplexity loss on WikiText-2, demonstrating its real-world applicability.

Advanced 4-Bit Quantization for LLMs: Introducing QuaRot

Overview

In the pursuit of enhancing efficiency in LLMs, the paper presents QuaRot, a novel quantization scheme that substantially simplifies the process of reducing the bit-width of model parameters, activations, and KV cache to 4-bits end-to-end. By devising a unique approach that utilizes rotational transformations to mitigate outliers in the data, QuaRot facilitates high fidelity in low-precision inference, overcoming the prevalent challenge of significant performance degradation associated with conventional quantization techniques. The quantitative results reveal minimal losses in model performance, with WikiText-2 perplexity losses not exceeding 0.29 and retention of 99% zero-shot performance accuracy.

Key Contributions

QuaRot's methodology leverages computational invariance, employing randomized Hadamard transformations to eliminate outlier features thus simplifying the quantization. This process does not alter the model's output but makes both weights and activations more amenable to quantization, enabling 4-bit matrix multiplications throughout the model. This approach is distinctly beneficial for LLMs, offering a practical solution for deploying advanced neural network models in resource-constrained environments. The paper articulates the following major contributions:

• Introduction of the QuaRot quantization strategy that utilizes rotational operations to eliminate outliers in LLMs, facilitating effective 4-bit quantization.
• Demonstration of its application to a quantized Llama model, showing significant efficiency gains including up to 2.16x prefill speedups and 3.39x memory savings during the decoding stage with minimal impact on performance.
• Provision of empirical evidence showing that QuaRot maintains high accuracy levels across diverse tasks, underscoring its practicality for real-world applications.
• Release of the accompanying codebase, offering the community a robust toolset for efficient LLM deployment.

The Insights Behind QuaRot

The pivotal insight QuaRot capitalizes on is the computational invariance property inherent in LLMs. This allows for the hidden states and activations within these models to undergo rotation transformations without affecting the output. By applying randomized Hadamard transformations, QuaRot effectively neutralizes outlier features that traditionally complicate low-bit quantization.

The technique significantly extends beyond the capabilities of existing approaches by ensuring that all components - including the KV cache pivotal for the attention mechanism - are quantized. This holistic approach contrasts with previous studies which primarily focused on quantizing either weights or activations separately, often leaving the KV cache in higher precision to avoid performance losses.

Implications and Future Directions

The introduction of QuaRot offers several theoretical and practical implications. It not only advances our understanding of the quantization landscape for LLMs but also opens up new avenues for deploying sophisticated LLMs on hardware with limited computational resources. This is particularly relevant in scenarios where deploying full precision models is unfeasible due to power, speed, or memory constraints.

Looking forward, the exploration of applying QuaRot to a broader range of models and tasks stands as a promising research direction. Further, investigating the integration of QuaRot within hardware accelerators could elucidate pathways to achieving even higher efficiency gains.

In conclusion, QuaRot represents a significant step forward in the field of model quantization, offering a viable solution to the pressing challenge of deploying LLMs efficiently. The methodology's ability to preserve model integrity while dramatically reducing computational load holds considerable promise for the future of AI applications, particularly in resource-constrained environments.