Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads (2401.11181v1)

Abstract: Transformer-based LLM inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

• The paper presents TetriInfer, revealing that disaggregating LLM inference into prefill and decode tasks can reduce TTFT by 97% and job completion times by 47%.
• It employs fixed-size prefill chunking and predictive scheduling using a smaller LLM classifier to efficiently manage mixed workload requests.
• Empirical results demonstrate a 38% reduction in resource usage, enabling scalable and cost-effective LLM inference in cloud environments.

"Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads"

Introduction

The paper "Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads" (2401.11181) presents an innovative approach to managing inference serving in cloud environments for LLMs using a system named TetriInfer. Inference serving for LLMs, ubiquitous in modern cloud services, consists of two distinct phases: the prefill and decode phases. Existing deployment paradigms often conflate these distinct phases leading to considerable resource interference and inefficiencies. The authors address the challenge of minimizing such interference through the strategic disaggregation and scheduling of inference tasks.

Motivation

Given the variation in LLM inference requests, particularly in their prefill and decode token lengths, traditional approaches result in severe contention and inefficiency. An analysis detailed in the paper shows that such interference can lead to significant performance degradation, up to a 10x slowdown for prefill requests and a 16% throughput hit for decode requests. The authors propose disaggregating prefill from decode tasks, enabling each to be processed independently and more efficiently, thereby avoiding the inefficiencies inherent in the simultaneous execution of heterogeneous task characteristics.

Methodology

TetriInfer's foundational strategy lies in dividing inference into three critical components: fixed-size prefill chunking, disaggregated instances for prefill and decode, and predictive scheduling. Prefill inputs are segmented into fixed-size chunks to maintain accelerators near their computational limits without incurring penalties (Figure 1). Separate instances are maintained for prefill and decode tasks, with a dynamic scheduler leveraging predicted resource usage to optimally assign decode instances.

Figure 1: Length Distribution. Prompt Tokens for Prefill and Generated Tokens during Decode.

The paper describes the implementation whereby prefill phases are executed in fixed-size batches, and decode tasks are managed by a scheduling algorithm that predicts and accommodates the length of generated tokens. This predictive model, which employs a smaller LLM as a classifier, anticipates the token length for more efficient scheduling. The decoupling extends to using different instances for prefill and decode tasks, further mitigating interference and enabling independent scaling and management.

Results

Empirical results demonstrate that TetriInfer dramatically improves key performance metrics such as time-to-first-token (TTFT) and job completion time (JCT). Specifically, the system uses 38% fewer resources while reducing average TTFT and JCT by 97% and 47%, respectively, indicating significant enhancements in computational efficiency. The system's ability to dynamically handle inference requests of varying lengths enhances its adaptability to real-world, high-variance workloads. This results in a marked improvement in performance per dollar compared to existing methods.

Figure 2: TetriInfer's Workflow and Architecture.

Implications and Future Work

TetriInfer illustrates significant advancements in handling LLM workload variability and interference. The decoupling model proposed is scalable and adaptable, providing a framework that can be expanded in future research to encompass more complex inference environments and possibly integrate with other emergent LLM optimizations like model partitioning and distributed inference.

Moreover, the implementation paves the way for further exploration into fine-grained predictive modeling for decoding phases, potentially improving scheduling decisions and reducing latency even further. The exploration of alternative network stack optimizations, as indicated, could further improve efficiency in distributed settings.

Conclusion

The paper outlined in "Inference without Interference" successfully delineates a path forward for more effective and efficient handling of LLM inference tasks in cloud-based environments by strategically separating and scheduling prefill and decode tasks. TetriInfer represents a substantial step towards maximizing the utility of computational resources in the face of the growing complexity and demand for LLM services. This work offers a promising avenue for scaling AI applications while maintaining sustainable resource expenditure, setting a benchmark for future efforts in the area.