Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization (2404.09956v4)

Abstract: Generative multimodal content is increasingly prevalent in much of the content creation arena, as it has the potential to allow artists and media personnel to create pre-production mockups by quickly bringing their ideas to life. The generation of audio from text prompts is an important aspect of such processes in the music and film industry. Many of the recent diffusion-based text-to-audio models focus on training increasingly sophisticated diffusion models on a large set of datasets of prompt-audio pairs. These models do not explicitly focus on the presence of concepts or events and their temporal ordering in the output audio with respect to the input prompt. Our hypothesis is focusing on how these aspects of audio generation could improve audio generation performance in the presence of limited data. As such, in this work, using an existing text-to-audio model Tango, we synthetically create a preference dataset where each prompt has a winner audio output and some loser audio outputs for the diffusion model to learn from. The loser outputs, in theory, have some concepts from the prompt missing or in an incorrect order. We fine-tune the publicly available Tango text-to-audio model using diffusion-DPO (direct preference optimization) loss on our preference dataset and show that it leads to improved audio output over Tango and AudioLDM2, in terms of both automatic- and manual-evaluation metrics.

• The paper demonstrates that integrating Direct Preference Optimization significantly improves semantic alignment in diffusion-based text-to-audio models.
• The authors develop the Audio-alpaca dataset, pairing text prompts with audio samples to guide preference-based model training.
• Tango 2 outperforms previous models like AudioLDM2 on CLAP scores, underscoring its effective human-aligned audio synthesis.

Enhancing Diffusion-based Text-to-Audio Generative Models through Direct Preference Optimization-based Alignment

Introduction

In the evolving field of generative AI, particularly within the multimedia content creation domain, the capability to convert text prompts into high-fidelity audio has immense applications, from aiding in pre-production mockups to supporting diverse creative endeavors. One core challenge in this area is the development of models that can accurately translate textual descriptions into coherent and contextually appropriate audio outputs. The paper by Majumder et al. addresses this challenge by presenting an innovative approach that leverages Direct Preference Optimization (DPO) for refining the performance of text-to-audio generative models, specifically through the enhancement of the Tango text-to-audio model.

Background and Related Work

Recent developments in text-to-audio generation have demonstrated promising results with models like AudioLDM and Tango, leveraging diffusion architectures to generate audio from text. Notably, the introduction of AudioLM has pushed the boundaries further by integrating semantic tokens derived from audio prompts into the generation process. However, these models often struggle with ensuring the presence and correct temporal ordering of described concepts or events in the generated audio, especially in data-constrained training environments. To address this, the paper builds on the recent success in aligning model outputs with human preferences using DPO, a method previously applied within LLM training, to the domain of audio generation.

Preference Dataset Generation

A pivotal contribution of this work is the creation of a novel preference dataset, termed Audio-alpaca, designed specifically for text-to-audio generation. This dataset comprises pairs of textual prompts and corresponding audios categorized into preferred (winner) and less desired (loser) outputs. These pairings are systematically generated through strategies that involve perturbing text prompts and employing adversarial filtering to produce audio variations. Through manual and automatic thresholding on model scores (e.g., CLAP scores) for these audios, the paper ensures the dataset effectively reflects preferences indicative of better alignment with human expectations.

Model Development and Evaluation

The augmented Tango model, dubbed Tango 2, undergoes fine-tuning with the DPO-diffusion loss on the Audio-alpaca dataset. Empirical evaluations showcase Tango 2’s superiority over its predecessor and another leading model, AudioLDM2, across a spectrum of automatic and manual assessment metrics. Notably, Tango 2 outperforms in terms of CLAP score, which directly measures the semantic correspondence between the audio output and the input prompt. This outcome underlines the effectiveness of the preference-based optimization approach in enhancing the model's ability to generate contextually and semantically aligned audio content.

Implications and Future Directions

The paper by Majumder et al. not only presents a significant leap in the text-to-audio generation space by introducing a preference-optimized model but also contributes a rich dataset that could serve as a foundation for future research. The success of the DPO-driven fine-tuning methodology in this domain opens up new avenues for exploring preference-aligned generative models across various content formats. Looking ahead, this approach may inspire further innovations in multimodal content generation, potentially leading to more intuitive and human-aligned AI tools for creative expression.

In conclusion, the incorporation of Direct Preference Optimization into the text-to-audio generation process represents a notable advance in the field. By more closely aligning generated audio with the semantic and contextual nuances of textual prompts, models like Tango 2 hold the promise of significantly enhancing the quality and utility of AI-generated audio content across a range of applications.