LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization (2403.01136v1)

Abstract: Recent breakthroughs in Large-scale LLMs have demonstrated impressive performance on various tasks. The immense sizes of LLMs have led to very high resource demand and cost for running the models. Though the models are largely served using uniform high-caliber GPUs nowadays, utilizing a heterogeneous cluster with a mix of available high- and low-capacity GPUs can potentially substantially reduce the serving cost. There is a lack of designs to support efficient LLM serving using a heterogeneous cluster, while the current solutions focus on model partition and uniform compression among homogeneous devices. This paper proposes LLM-PQ, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. We carefully decide on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm, to greatly enhance inference throughput while fulfilling user-specified model quality targets. Extensive experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference, showing great advantages over state-of-the-art works.

• The paper introduces a novel LLM-PQ system that partitions models based on inference phases and adapts quantization to optimize GPU utilization.
• It demonstrates up to 2.88x throughput improvement over existing methods through extensive testing on 11 heterogeneous GPU clusters.
• The work offers a cost model and variance indicator to guide precision selection for deploying large-scale language models cost-effectively.

Optimizing LLM Deployment on Heterogeneous GPU Clusters with LLM-PQ

In the cutting-edge sphere of generative AI and Large-scale LLMs, the computational and memory demands for effectively serving these models are formidable. This paper introduces LLM-PQ, a system designed to enhance the efficiency of LLM serving on heterogeneous GPU clusters by incorporating adaptive model quantization and phase-aware model partitioning.

Phase-Aware Model Partitioning and Adaptive Quantization

The paper addresses a critical bottleneck in the deployment of LLMs – their immense size and corresponding resource demands. The authors propose a novel approach, LLM-PQ, which stands for Phase-aware Partition and Adaptive Quantization, tailored to optimize LLM serving on heterogeneous GPU clusters.

The key insight is two-fold:

1. Phase-Aware Partitioning: Recognizes that mainstream LLMs like GPT3 and BERT experience two distinct phases during inference – prompt processing and token generation. By adopting a phase-aware approach to model partition among GPUs, LLM-PQ ensures a more balanced workload distribution, leading to improved resource utilization.
2. Adaptive Quantization: Diverging from uniform, one-size-fits-all quantization strategies, LLM-PQ adapts the quantization precision based on the memory capacity and computational prowess of individual GPUs within a heterogeneous cluster. This strategy mitigates memory wastage on high-capacity GPUs and reduces the risk of out-of-memory errors on constrained devices.

Experimental Validation

LLM-PQ's efficacy is demonstrated through extensive experimentation across 11 different heterogeneous clusters using production inference workloads. The results are compelling, showcasing up to 2.88x throughput improvement in inference relative to state-of-the-art methods. Such significant gains underscore the utility of adaptive quantization and phase-aware partitioning in enhancing the efficiency of LLM serving.

Theoretical Contributions and Practical Implications

The authors offer a meticulously designed cost model that enables accurate prediction of memory requirements and inference latency under mixed-precision quantization schemes. The introduction of a variance indicator to gauge layer sensitivity towards various levels of quantization emerges as a noteworthy theoretical contribution, facilitating optimal bitwidth selection.

Practically, LLM-PQ has profound implications for cloud-based AI services and machine learning clusters, where heterogeneity of computing resources is common. By optimizing the deployment of LLMs across diverse GPU setups, LLM-PQ paves the way for cost-efficient, high-performance AI applications.

Future Directions

Looking ahead, the integration of LLM-PQ with tensor-parallelism techniques and exploration of its applicability to online serving tasks represent exciting avenues for research. Additionally, the adaptation of LLM-PQ to accommodate emerging quantization schemes could further refine its effectiveness and broaden its applicability.

Conclusion

LLM-PQ stands as a significant advancement in the domain of LLM serving, addressing the challenge of efficiently deploying these colossal models in heterogeneous computing environments. Through intelligent layer partitioning and adaptively adjusting quantization precision, LLM-PQ unlocks new possibilities for leveraging the full potential of mixed-capability GPU clusters, marking a pivotal step forward in the scalable and cost-effective execution of large-scale LLMs.