CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference (2401.11240v1)

Abstract: Pre-trained LLMs often need specialization for domain-specific tasks. Low-Rank Adaptation (LoRA) is a popular approach that adapts a base model to multiple tasks by adding lightweight trainable adapters. In this paper, we present CaraServe, a system that efficiently serves many LoRA adapters derived from a common base model. CaraServe maintains the base model on GPUs and dynamically loads activated LoRA adapters from main memory. As GPU loading results in a cold-start that substantially delays token generation, CaraServe employs a CPU-assisted approach. It early starts the activated adapters on CPUs for prefilling as they are being loaded onto GPUs; after loading completes, it then switches to the GPUs for generative LoRA inference. CaraServe develops a highly optimized synchronization mechanism to efficiently coordinate LoRA computation on the CPU and GPU. Moreover, CaraServe employs a rank-aware scheduling algorithm to optimally schedule heterogeneous LoRA requests for maximum service-level objective (SLO) attainment. We have implemented CaraServe and evaluated it against state-of-the-art LoRA serving systems. Our results demonstrate that CaraServe can speed up the average request serving latency by up to 1.4$\times$ and achieve an SLO attainment of up to 99%.

• The paper presents CaraServe, a system that uses CPU prefill and rank-aware scheduling to reduce cold-start latency for LoRA adapters in LLM inference.
• It employs asynchronous memory copy and shared memory coordination to optimize GPU-CPU workload during inference.
• Experimental results demonstrate up to 1.4x faster latency and 99% SLO compliance, enhancing performance in multi-tenant environments.

CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference

CaraServe is a system designed to efficiently serve Low-Rank Adaptation (LoRA) adapters derived from a common base model in generative LLM inference. It confronts issues inherent in traditional LLM deployment, specifically aiming to reduce latency and meet service-level objectives (SLO). Utilizing both CPU and GPU resources, CaraServe dynamically manages LoRA adapter activation and employs rank-aware scheduling.

CPU-Assisted Serving Mechanism

CaraServe mitigates GPU cold-start delays by employing CPU-assisted serving. This approach involves prefilling using CPU before the GPU completes loading the activated LoRA adapters.

Figure 1: Illustration of CPU-assisted LoRA serving.

In practical terms, when a request arrives, CaraServe starts computation of the activated LoRA adapter on the CPU while it is being loaded to the GPU. The system employs asynchronous memory copy and signaling to coordinate the computations efficiently across devices. This synchronization between GPU and CPU reduces the overhead typically associated with loading adapters onto the GPU.

Efficient GPU-CPU Coordination

CaraServe optimizes the synchronization needed between the GPU and CPU to execute LoRA computations. It leverages shared memory for fast data exchange, significantly reducing inter-process communication overheads.

Figure 2: Illustration of coordinated LoRA computation on GPU and CPU per transformer block's attention layer.

Additionally, CaraServe implements a profiling-guided parallelization scheme allowing LoRA computations to scale across multiple CPUs, addressing potential bottlenecks when processing long input prompts.

Rank-Aware Scheduling

In multi-tenant environments, requests trigger activation of LoRA adapters with varying ranks. CaraServe introduces a rank-aware scheduling algorithm informed by performance models developed through profiling.

Figure 3: Performance models for BGMV (Left) and MBGMV (Right) kernels. Both linear regression models achieve a high coefficient of determination ($R^2$) of 0.96.

CaraServe's scheduler evaluates server options based on the batch heterogeneity in rank and selects the server with the minimal cost score, ensuring optimal batch compositions and meeting SLOs efficiently.

Architecture Overview

CaraServe comprises LLM inference servers, a scheduler, and a global LoRA registry. The servers manage base models on GPUs and LoRA adapters in memory for efficient multiplexing. The scheduler adopts rank-aware invocation strategies for request routing.

Figure 4: An architecture overview of CaraServe.

Unlike existing systems that either suffer from cold-start latency or inefficient scheduling, CaraServe simultaneously addresses both challenges, providing a scalable solution for serving numerous adapters efficiently.

Experimental Evaluation

CaraServe was evaluated against state-of-the-art systems and demonstrated a significant reduction in serving latency—up to 1.4 times faster—while achieving a high SLO attainment of up to 99%. Experimental setups included various workload conditions, accommodating the differences in request traffic and rank configurations.

Figure 5: Prefill performance of different kernels on Llama2-7B model. Native: PyTorch default kernels. CaraServe: Implementation with our optimized kernels.

The evaluations highlighted CaraServe's ability to leverage CPU and GPU effectively, minimizing latency and enhancing user experience in real-time LLM deployments.

Conclusion

CaraServe is a robust solution for serving LoRA adapters in multi-tenant cloud environments, addressing challenges of cold-start latency and SLO compliance through intelligent rank-awareness and CPU-assisted pre-filling. Its architecture enables the efficient virtualization of computational resources while maintaining high performance and user satisfaction in generative AI applications.