Multistep Consistency Models (2403.06807v3)

Abstract: Diffusion models are relatively easy to train but require many steps to generate samples. Consistency models are far more difficult to train, but generate samples in a single step. In this paper we propose Multistep Consistency Models: A unification between Consistency Models (Song et al., 2023) and TRACT (Berthelot et al., 2023) that can interpolate between a consistency model and a diffusion model: a trade-off between sampling speed and sampling quality. Specifically, a 1-step consistency model is a conventional consistency model whereas a $\infty$-step consistency model is a diffusion model. Multistep Consistency Models work really well in practice. By increasing the sample budget from a single step to 2-8 steps, we can train models more easily that generate higher quality samples, while retaining much of the sampling speed benefits. Notable results are 1.4 FID on Imagenet 64 in 8 step and 2.1 FID on Imagenet128 in 8 steps with consistency distillation, using simple losses without adversarial training. We also show that our method scales to a text-to-image diffusion model, generating samples that are close to the quality of the original model.

• The paper presents a framework that interpolates between traditional multi-step diffusion models and single-step consistency models, achieving a balance between speed and output quality.
• It introduces the Adjusted DDIM (aDDIM) sampler to reduce integration errors and improve fidelity, demonstrating competitive FID scores in as few as 8 steps.
• The approach leverages step schedule annealing and synchronized dropout to simplify training and enhance overall sample generation efficiency.

Unifying Consistency Models and TRACT for Efficient Diffusion Model Sampling

Introduction to Multistep Consistency Models

In recent developments within the field of generative modeling, particularly concerning diffusion models, the trade-off between sampling efficiency and output quality has been a focal point of research. Traditional diffusion models, despite their efficacy in generating high-quality samples, suffer from the drawback of necessitating numerous iterative steps to produce outputs, thereby increasing computational costs and sampling times. On the other hand, Consistency Models, as introduced by Song et al., aim to mitigate this inefficiency by proposing a model capable of generating samples in a single iteration. However, this approach often comes at the expense of sample quality.

In this context, we introduce Multistep Consistency Models, a novel methodology that effectively bridges the gap between the traditional multi-step diffusion models and the single-step Consistency Models. Our proposed model allows for a flexible middle ground by enabling sample generation in multiple steps, providing a customizable balance between quality and efficiency.

Key Contributions

Our research presents several key contributions to the domain of generative modeling and LLMs:

• We propose a novel framework termed Multistep Consistency Models, which unifies the concepts underlying Consistency Models and TRACT. This framework enables interpolation between the traditional diffusion model and single-step consistency models, allowing users to choose an optimal point in terms of sampling speed versus quality.
• Through extensive experimentation, particularly on challenging datasets such as ImageNet, we demonstrate that by increasing the steps from one to a modest range (2-8 steps), we can significantly enhance sample quality while retaining the benefits of reduced sampling time. Remarkably, we attain competitive FID scores on par with baseline diffusion models in as few as 8 steps.
• A critical aspect of our methodology is the introduction of the Adjusted DDIM (aDDIM) sampler, a deterministic sampling technique that mitigates the integration errors inherent in traditional deterministic samplers like DDIM, effectively reducing sample blurriness and improving fidelity.
• Theoretical discussions within our paper illustrate that as the number of steps in Multistep Consistency Training increases, the model increasingly resembles a standard diffusion model, thereby reinforcing the intuition behind our approach.
• Our research underscores the importance of step schedule annealing and synchronized dropout, which were pivotal in training models that not only achieve higher quality samples but also facilitate an easier training process.

Implications and Speculations on Future Developments

The introduction of Multistep Consistency Models heralds a significant advancement in the field of generative AI and diffusion models. By offering a flexible framework that interpolates between speed and quality, our methodology presents a compelling solution to one of the primary bottlenecks in diffusion model sampling. This balance is particularly relevant in scenarios requiring rapid sample generation without substantially compromising on output quality.

Looking forward, the versatility of Multistep Consistency Models opens avenues for deeper exploration into efficient training strategies, the further evolution of deterministic samplers, and the potential integration of these concepts into broader applications beyond image generation, including video and audio synthesis.

Moreover, our findings invite further investigation into the theoretical underpinnings of consistency models and diffusion processes, potentially paving the way for novel generative models that transcend the limitations of current methodologies.

In summary, Multistep Consistency Models represent a pivotal step toward the refinement of diffusion-based generative models, promising not only enhanced efficiency and sample quality but also inspiring future innovations in generative AI research.