Enhancing Audio Generation Diversity with Visual Information (2403.01278v1)

Abstract: Audio and sound generation has garnered significant attention in recent years, with a primary focus on improving the quality of generated audios. However, there has been limited research on enhancing the diversity of generated audio, particularly when it comes to audio generation within specific categories. Current models tend to produce homogeneous audio samples within a category. This work aims to address this limitation by improving the diversity of generated audio with visual information. We propose a clustering-based method, leveraging visual information to guide the model in generating distinct audio content within each category. Results on seven categories indicate that extra visual input can largely enhance audio generation diversity. Audio samples are available at https://zeyuxie29.github.io/DiverseAudioGeneration.

• The paper demonstrates that integrating visual cues into audio generation notably improves diversity by capturing fine-grained sub-categories.
• It employs a combination of VAEs, VQ-VAEs, and latent diffusion techniques alongside CLIP-extracted image features to enhance audio representations.
• Experimental results on the DCASE2023 dataset confirm that visual information not only increases diversity metrics but also maintains high audio quality.

Enhancing Audio Generation Diversity with Visual Information

Introduction to Vision-guided Audio Generation

The integration of visual information into the process of category-based audio generation offers a promising avenue to mitigate the homogeneity typically observed in generated audio samples within specific categories. This paper introduces a novel framework that leverages clustering-based methodology and visual cues to produce a more diverse array of audio content. This approach is predicated on the observation that visual context can significantly augment the generation process by providing additional, fine-grained distinctions within audio categories that are not readily captured through audio data or textual labels alone.

Methodology

The proposed model architecture comprises several key components, each designed to contribute to the generation of diverse and high-quality audio content:

• Modal Fusion Module: This component integrates visual information with category labels, using the rich detail available in images to produce embeddings that better represent sub-categories within broader audio classes.
• Audio Representation Models: Employing Variational Autoencoders (VAE) and Vector Quantized VAE (VQ-VAE), the framework compresses audio into a latent representation. This step is crucial for capturing the essence of audio content in a more manageable form for subsequent generation processes.
• Token Prediction Models: These models predict the latent representations of future audio samples based on the fused visual-textual input. The paper explores both auto-regressive models and Latent Diffusion Models (LDM) for this purpose, each offering distinct advantages for the generation task.

The integration of visual data involves manually querying relevant images for each audio sub-category created through spectral clustering. CLIP is utilized to extract features from these images, producing a rich, multimodal input for the generation model.

Experimental Setup

The experimental validation of the proposed framework is conducted using the DCASE2023 task 7 dataset, encompassing a diverse set of audio categories. Two primary generative frameworks are employed: VAE & LDM, and VQ-VAE & Transformer. Evaluation metrics focus on both the quality and diversity of generated audio, leveraging objective measures such as Fre´chet Audio Distance (FAD) and Mean Squared Distance (MSD), alongside subjective assessments through Mean Opinion Score (MOS) evaluations.

Results and Discussion

The introduction of visual information unequivocally enhances the diversity of generated audio across various categories. This is particularly evident when comparing models that utilize prototype images versus those that average visual features, with the former consistently outperforming the latter in diversity metrics. The paper highlights several key findings:

• Diversity Improvement: Substantial improvements in the diversity of generated audio, as evidenced by higher MSD values across most categories when visual cues are incorporated.
• Quality Maintenance: The quality of audio generated with visual guidance remains on par with, if not superior to, audio generated purely from category labels. This is significant, as it demonstrates the feasibility of enriching audio diversity without compromising the overall quality of generated content.
• Visual Information as a Control Mechanism: The use of more representative images not only enhances diversity but also provides a means to control the specifics of generated audio, underscoring the potential for customized audio generation.

Conclusion and Future Outlook

This paper presents a compelling case for the integration of visual information into the audio generation process to surmount limitations in diversity observed in current generative models. The proposed clustering-based framework adeptly leverages the complementary nature of audio and visual data to produce audio samples that are not only diverse but also of high quality. Future research directions might explore automated methods for image retrieval to scale this approach and further refine the use of visual data to control generation parameters, potentially leading to even more nuanced and tailored audio generation capabilities.