Speculative Decoding with Big Little Decoder (2302.07863v4)

Abstract: The recent emergence of LLMs based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these models have long inference latency, which limits their deployment and makes them prohibitively expensive for various real-time applications. The inference latency is further exacerbated by autoregressive generative tasks, as models need to run iteratively to generate tokens sequentially without leveraging token-level parallelization. To address this, we propose Big Little Decoder (BiLD), a framework that can improve inference efficiency and latency for a wide range of text generation applications. The BiLD framework contains two models with different sizes that collaboratively generate text. The small model runs autoregressively to generate text with a low inference cost, and the large model is only invoked occasionally to refine the small model's inaccurate predictions in a non-autoregressive manner. To coordinate the small and large models, BiLD introduces two simple yet effective policies: (1) the fallback policy that determines when to hand control over to the large model; and (2) the rollback policy that determines when the large model needs to correct the small model's inaccurate predictions. To evaluate our framework across different tasks and models, we apply BiLD to various text generation scenarios encompassing machine translation on IWSLT 2017 De-En and WMT 2014 De-En, and summarization on XSUM and CNN/DailyMail. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Furthermore, our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture. Our code is open-sourced

• The paper presents a dual-model approach that uses a small autoregressive model complemented by a larger model for error correction to enhance inference efficiency.
• It details fallback and rollback policies that dynamically switch between models based on prediction confidence to maintain quality.
• Experimental results demonstrate up to a 2.12× speedup on benchmark datasets with minimal degradation, making it viable for real-time applications.

Overview of Speculative Decoding with Big Little Decoder

The research paper titled "Speculative Decoding with Big Little Decoder" proposes a novel framework named Big Little Decoder (BiLD) which aims to enhance the efficiency and reduce the inference latency of LLMs in text generation tasks. The persistent challenge with LLMs has been their prohibitive inference latency, especially prevalent in autoregressive text generation tasks where the sequential generation of tokens limits opportunities for parallelization. This paper addresses these challenges by introducing a collaborative model execution framework that incorporates two models of differing sizes, a small model running autoregressively for low-cost inference and a large model invoked as needed for error correction in a non-autoregressive manner.

Methodology

The BiLD framework leverages two primary strategies—the fallback and rollback policies. The fallback policy is designed to hand over control to the larger model if the smaller model’s confidence in its predictions drops below a predefined threshold. Conversely, the rollback policy allows for the large model to correct any inaccuracies made by the small model. This dual-model orchestration is designed to maximize inference efficiency without significantly compromising the quality of text generated.

Experimental Results

BiLD was tested across multiple text generation scenarios, including machine translation on IWSLT 2017 De-En and WMT 2014 De-En benchmarks and summarization tasks on XSUM and CNN/DailyMail datasets. The experiments demonstrated significant latency improvements, with a speedup of up to 2.12× on an NVIDIA T4 GPU while allowing only minimal degradation in generation quality. This makes BiLD a competitive alternative to conventional autoregressive methods.

Implications and Future Directions

The findings have both practical and theoretical implications. Practically, BiLD provides a feasible solution for deploying LLMs in real-time applications where latency is a critical factor. The ability to boost inference speed without necessitating model retraining or architectural changes is particularly beneficial in resource-constrained environments. Theoretically, this work contributes to the growing body of research focused on enhancing model efficiency through architectural innovations.

Future developments could explore the integration of BiLD with various non-transformer-based architectures to examine its applicability and effectiveness. Further research could also aim to refine the fallback and rollback policies, potentially through machine learning-based techniques that dynamically adjust these policies based on model performance during inference.

In summary, the Big Little Decoder presents a strategically balanced approach to overcoming the latency challenges encountered in autoregressive text generation tasks, positioning itself as a viable solution for efficient LLM deployment in real-world applications.