Diffusion Model with Perceptual Loss (2401.00110v7)

Abstract: Diffusion models without guidance generate very unrealistic samples. Guidance is used ubiquitously, and previous research has attributed its effect to low-temperature sampling that improves quality by trading off diversity. However, this perspective is incomplete. Our research shows that the choice of the loss objective is the underlying reason raw diffusion models fail to generate desirable samples. In this paper, (1) our analysis shows that the loss objective plays an important role in shaping the learned distribution and the MSE loss derived from theories holds assumptions that misalign with data in practice; (2) we explain the effectiveness of guidance methods from a new perspective of perceptual supervision; (3) we validate our hypothesis by training a diffusion model with a novel self-perceptual loss objective and obtaining much more realistic samples without the need for guidance. We hope our work paves the way for future explorations of the diffusion loss objective.

• The paper proposes integrating a self-perceptual loss into diffusion models, replacing conventional MSE to enhance image realism.
• The methodology leverages the model’s latent space and noise scheduling to compute perceptual loss, leading to improved FID and Inception scores.
• Extensive ablation studies validate that the self-perceptual approach boosts performance in unconditional generation without relying on external guidance.

Diffusion Model with Perceptual Loss

Introduction

The paper presents an enhancement to diffusion models for generative tasks by incorporating a self-perceptual loss as opposed to the conventional mean squared error (MSE) loss. Diffusion models have emerged as a popular class of generative models that transform noise into data samples via an iterative denoising process. This transformation process traditionally uses MSE loss, which has been shown to often result in unrealistic image samples. The paper creatively suggests that perceptual loss derived inherently from the diffusion model can yield more realistic outputs, bypassing the need for external perceptual networks.

Diffusion Models and Perceptual Loss

Diffusion models are generally parameterized as neural networks, trained using MSE to minimize the difference between model predictions and ground truth. However, due to ambiguity in the underlying data distribution (Figure 1), these models struggle to generate high-quality outputs intrinsically. The paper highlights the limitations of classifier-free guidance, a popular technique employed to enhance sample quality, and proposes using the model itself to generate perceptual loss, which can be applied even to unconditional generation models—a feat previously unattainable.

Figure 1: The underlying data distribution is ambiguous given finite training data.

Self-Perceptual Objective

The novel self-perceptual objective leverages the diffusion model as its perceptual network, exploiting the inherent architectural properties of the model to improve the quality of generated samples. The approach focuses on aligning with human perceptual metrics better than MSE. It capitalizes on the latent space and noise levels at different training phases to backpropagate perceptual loss, thereby enhancing output quality without sacrificing diversity.

Methodology

The diffusion process is set up to select image latent samples, noise, and time steps, advancing through a defined diffusion schedule. This schedule incorporates perceptual objectives by freezing a copy of the diffusion model trained with traditional MSE loss. The training uses pseudo-code implementations to streamline integration into existing diffusion frameworks, enhancing reproducibility and application across various model architectures.

Evaluation and Results

Quantitative results demonstrate improvements in Fréchet Inception Distance (FID) and Inception Score (IS), showcasing better alignment with perceptual quality standards over conventional methods. Qualitative assessments further exhibit superior result authenticity and visual appeal when utilizing the proposed methodology (Figure 2).

Figure 2: Unconditional generation. Both use DDIM 1000 steps with the same seed. Our self-perceptual objective can improve unconditional generation quality. This was previously not possible with classifier-free guidance because it only works for conditional models.

Ablation Studies

Extensive ablation studies highlight optimal configurations, including layer influence on perceptual loss and timestep selections for generating features. The paper demonstrates the superior balance and performance attained through perceptual loss across various tested parameters.

Conclusions

The research demonstrates the effectiveness of integrating self-perceptual objectives into diffusion training, thereby enhancing sample quality, particularly for unconditional models. Although classifier-free guidance still surpasses in text-alignment scenarios, the proposed method sets a precedent for exploring internal perceptual capacities of generative models, potentially informing future architecture designs, sampling strategies, and loss function optimizations.

The exploration of perceptual loss directly harnessed from diffusion models offers a promising avenue for advancing generative model quality, with implications for broader applications across image, video, and audio modalities. The ability to enhance outputs without additional guidance mechanisms marks a significant step in generative model advancements. The findings encourage further exploration of model-inherent properties to guide training dynamics and output quality, potentially reshaping approaches to diffusion-based generative modeling.