Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning (2405.18386v2)

Abstract: Recent advances in text-to-music editing, which employ text queries to modify music (e.g.\ by changing its style or adjusting instrumental components), present unique challenges and opportunities for AI-assisted music creation. Previous approaches in this domain have been constrained by the necessity to train specific editing models from scratch, which is both resource-intensive and inefficient; other research uses LLMs to predict edited music, resulting in imprecise audio reconstruction. To Combine the strengths and address these limitations, we introduce Instruct-MusicGen, a novel approach that finetunes a pretrained MusicGen model to efficiently follow editing instructions such as adding, removing, or separating stems. Our approach involves a modification of the original MusicGen architecture by incorporating a text fusion module and an audio fusion module, which allow the model to process instruction texts and audio inputs concurrently and yield the desired edited music. Remarkably, Instruct-MusicGen only introduces 8% new parameters to the original MusicGen model and only trains for 5K steps, yet it achieves superior performance across all tasks compared to existing baselines, and demonstrates performance comparable to the models trained for specific tasks. This advancement not only enhances the efficiency of text-to-music editing but also broadens the applicability of music LLMs in dynamic music production environments.

• The paper introduces Instruct-MusicGen, which leverages specialized audio and text fusion modules to enable resource-efficient text-to-music editing.
• It achieves superior performance on metrics such as FAD, CLAP, and SSIM, ensuring high-quality audio reconstructions.
• The model adds only 8% new parameters to the base MusicGen, enabling rapid training and precise editing for advanced music production.

Efficient Text-to-Music Editing with Instruct-MusicGen: A Comprehensive Overview

Introduction

The paper on Instruct-MusicGen introduces a novel approach to text-to-music editing that significantly enhances the efficiency and applicability of AI in music production. Leveraging the pretrained MusicGen model, the authors present a mechanism to follow editing instructions, thus addressing the existing limitations of previous methods in this domain.

Background and Motivation

Text-to-music editing involves modifying music using textual queries, a process that encompasses intra-stem and inter-stem editing. The current state-of-the-art models face significant limitations, such as the resource-intensive requirement to train specific editing models from scratch or the imprecision associated with using LLMs to predict edited music. This paper targets the challenges of ensuring high-quality audio reconstruction and precise adherence to editing instructions.

Methodology

MusicGen and Extensions

MusicGen, the base model, exploits EnCodec for compressing and reconstructing music audio and a multi-layer transformer for modeling latent code sequences. Instruct-MusicGen builds upon this foundation by introducing two critical modules:

1. Audio Fusion Module: This module processes the input music audio to be edited. It incorporates a duplicated encoder and subsequent transformers to embed the conditional audio, allowing for concurrent text and audio processing.
2. Text Fusion Module: This module handles text instructions. By finetuning the cross-attention mechanism rather than the entire text encoder, it introduces minimal additional parameters, maintaining high computational efficiency.

Together, these adjustments allow Instruct-MusicGen to interpret and execute a wide range of editing tasks such as adding, removing, or separating stems with significantly reduced computational cost and training time.

Training and Data

Training was conducted on synthetic instructional datasets derived from the Slakh2100 dataset, with the model finetuned for only 5,000 steps on a single NVIDIA A100 GPU. This approach introduced approximately 8% new parameters to the original MusicGen model, showcasing the method's resource efficiency.

Evaluation and Results

The performance of Instruct-MusicGen was comprehensively evaluated against multiple baselines using various metrics:

• Fréchet Audio Distance (FAD): A measure of overall audio quality.
• CLAP Score: Aligns audio content with textual descriptions.
• Kullback-Leibler Divergence (KL): Assesses information loss.
• Structural Similarity (SSIM): Evaluates structural similarity.
• Scale-Invariant Signal-to-Distortion Ratio (SI-SDR/i): Measures audio clarity and improvement.

Instruct-MusicGen demonstrated superior performance in nearly all tasks across both the Slakh2100 and MoisesDB datasets. Notably, it achieved the lowest FAD and highest CLAP and SSIM scores in the addition task, indicative of its high audio quality and semantic coherence. Although it showed some limitations in accurately isolating stems (e.g., experiencing challenges with SI-SDRi in complex scenarios), its overall performance was robust and competitive.

Implications and Future Work

The implications of this research are multifaceted:

• Practical: It enhances the efficiency of music production processes, allowing for high-quality and accurate modifications with minimal computational resources.
• Theoretical: The paper contributes to the broader understanding of multimodal AI, illustrating how pretrained models can be adapted for specific editing tasks with minimal new parameters.

Speculations on Future Developments in AI

Future developments may involve extending Instruct-MusicGen's capabilities to handle a wider range of musical genres and complexities, potentially integrating with more diverse real-world datasets. Enhancements in the clarity and precision of stem isolation could be pursued to address the current limitations in certain metrics.

Conclusion

Instruct-MusicGen presents a significant advancement in the field of text-to-music editing. By efficiently leveraging pretrained LLMs and introducing specialized modules for audio and text fusion, it significantly improves the practical applicability and computational efficiency of AI-assisted music editing. This approach paves the way for further innovations in dynamic music production environments and multimodal AI research.

By providing detailed empirical evaluations, the authors convincingly demonstrate the model's robustness and versatility, validating the approach's potential to transform the landscape of AI-driven music creation.