LLMs have fundamentally transformed the capabilities of numerous applications, from natural language processing to more intricate domain-specific tasks in robotics and autonomous driving. Moreover, the importance of on-device LLMs has grown significantly in the recent years. Running LLMs on edge devices not only promises reduced latency and improved user experience but also aligns with the increasing need for user privacy, as data processing can occur locally. However, the astronomical model sizes of modern LLMs and constraints of the edge devices, primarily in terms of memory size and bandwidth, pose significant deployment challenges. In this paper, we propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. Our method is based on the observation that weights are not equally important: protecting only 1% of salient weights can greatly reduce quantization error. We then propose to search for the optimal per-channel scaling that protects the salient weights by observing the activation, not weights. AWQ does not rely on any backpropagation or reconstruction, so it can well preserve LLMs' generalization ability on different domains and modalities, without overfitting to the calibration set. AWQ outperforms existing work on various language modeling and domain-specific benchmarks (coding and math). Thanks to better generalization, it achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. Alongside AWQ, we implement TinyChat, an efficient and flexible inference framework tailored for on-device LLM/VLMs, offering more than 3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs. It also democratizes the deployment of the 70B Llama-2 model on mobile GPUs.
AWQ introduces a novel low-bit, weight-only quantization method for LLMs, focusing on scaling weights based on activation to reduce quantization errors.
It differentiates weights by impact on performance, finding minimal yet critical weight protection can substantially reduce errors without mixed-precision formats.
Empirical evaluations show AWQ's superiority in speed and performance across various LLM benchmarks, including multi-modal language models.
AWQ aims to make LLM deployment more efficient, suggesting a potential standard approach for LLM quantization and encouraging further research in optimization techniques.
Activation-aware Weight Quantization (AWQ) represents a significant advancement in the quantization of LLMs, which are pivotal in natural language understanding and generation tasks. The core innovation of AWQ lies in its method for low-bit, weight-only quantization that strategically scales certain weights based on activation rather than weight values themselves. This approach is grounded in the understanding that a small subset of weights, when appropriately scaled, can significantly mitigate the negative impacts of quantization on model performance.
The introduction and implementation of AWQ have multiple implications for the field of AI and LLM research:
An efficient and flexible serving framework accompanies the AWQ approach, translating theoretical benefits into practical performance improvements. This framework leverages kernel fusion and other optimization techniques to realize speed gains in LLM inference tasks. The successful deployment of LLMs, such as the 70B parameter Llama-2 model on constrained devices like the NVIDIA Jetson Orin, exemplifies the real-world applicability and benefits of AWQ.
Activation-aware Weight Quantization (AWQ) represents a significant leap forward in the efficient and effective quantization of LLMs. Through a meticulous focus on the saliency of weights and innovative per-channel scaling, AWQ not only preserves but enhances model performance post-quantization. Its application across a variety of LLMs and tasks, coupled with significant speed and efficiency gains, positions AWQ as a pivotal advancement in the broader endeavor to make powerful AI models more accessible and practical for a wide range of applications.