Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models (2402.07033v3)

Abstract: LLMs with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks. However, due to the huge model sizes, running them in resource-constrained environments where the GPU memory is not abundant is challenging. Some existing systems propose to use CPU resources to solve that, but they either suffer from the significant overhead of frequently moving data between CPU and GPU, or fail to consider distinct characteristics of CPUs and GPUs. This paper proposes Fiddler, a resource-efficient inference system for MoE models with limited GPU resources. Fiddler strategically utilizes CPU and GPU resources by determining the optimal execution strategy. Our evaluation shows that, unlike state-of-the-art systems that optimize for specific scenarios such as single batch inference or long prefill, Fiddler performs better in all scenarios. Compared against different baselines, Fiddler achieves 1.26 times speed up in single batch inference, 1.30 times in long prefill processing, and 11.57 times in beam search inference. The code of Fiddler is publicly available at https://github.com/efeslab/fiddler.

• The paper introduces Fiddler, a novel inference engine that offloads expert computations to the CPU to reduce data transfer overhead.
• It employs CPU-GPU orchestration optimized for single-batch processing, achieving up to 10.1x speed improvements over traditional methods.
• The method enhances MoE model deployment in limited-memory settings by efficiently balancing computational resources between CPU and GPU.

Overview of Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

The paper introduces Fiddler, an advanced inference engine designed to efficiently deploy Mixture-of-Experts (MoE) models in resource-constrained environments using CPU-GPU orchestration. MoE architectures, which can dynamically activate subsets of experts, present a significant challenge when operating under limited GPU memory due to their extensive parameter sizes. Traditional methods often incur high overhead by moving weights between CPU and GPU, negatively impacting performance. Fiddler addresses this issue by leveraging both CPU memory and computational resources to minimize data transfers.

Core Contributions

Fiddler's primary innovation lies in its approach to handling expert computations. It selectively uses CPU computational capabilities to execute expert layers, reducing the need to transfer large weights to the GPU. This strategy is particularly effective in single-batch, latency-critical scenarios where small batch sizes exacerbate inefficiencies in data transfer.

1. CPU Utilization: Fiddler shifts the computation of certain operations to the CPU. This significantly reduces latency associated with transferring large weights over PCIe connections, typically a bottleneck in such setups.
2. Single-Batch Efficiency: The inference system is optimized for local, single-request processing. By managing expert layers directly on CPUs, Fiddler provides a solution tailored for environments where only one GPU with constrained memory is available.
3. Performance Improvement: The results show that Fiddler achieves substantial speedups—up to 10.1 times faster—compared to existing techniques by effectively orchestrating CPU and GPU activities.

Performance Evaluation

Fiddler's performance was evaluated using the Mixtral-8x7B model with over 90GB of parameters. Tests conducted on a Quadro RTX 6000 GPU and an L4 GPU evidenced a significant throughput improvement, generating over 3 tokens per second, a marked advance over other offloading methods. The evaluation considered various token lengths for input and output, emphasizing Fiddler's robustness and versatility across different scenarios.

Implications

Fiddler's design presents noteworthy advancements for MoE model deployment in resource-limited settings. By effectively utilizing heterogeneous hardware resources, it sets a precedent for balancing memory and compute management across CPUs and GPUs. This methodology not only enhances the practical deployment of large-scale LLMs in such environments but also provides a potential blueprint for future optimizations in AI model orchestration.

Future Directions

The development of Fiddler opens new avenues for research, particularly in exploring further efficiency gains in MoE architectures. Future work might investigate the integration of compression techniques with Fiddler's framework, potentially offering enhanced performance without significant loss in model quality. Additionally, adapting Fiddler to support evolving hardware configurations and ensuring compatibility with newer AI models could further its applicability and impact.

In conclusion, Fiddler represents a significant step forward in the efficient and practical deployment of MoE models, providing a solution to circumvent the limitations of current resource-constrained inference approaches by fully exploiting the capabilities of available hardware.