Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders (1704.01279v1)

Abstract: Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.

• The paper introduces a novel WaveNet-style autoencoder that learns temporal audio embeddings for synthesizing musical notes without external conditioning.
• It leverages the NSynth dataset’s 306,000 four-second musical notes to perform rigorous qualitative and quantitative evaluations.
• The model demonstrates superior reconstruction and interpolation performance, indicating robust disentanglement of pitch and timbre.

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

The paper presents a detailed exploration of neural audio synthesis through the application of WaveNet autoencoders, specifically targeting the generation and manipulation of musical notes. This work is pivotal in bridging the gap between algorithmic advancements in generative modeling and the burgeoning field of audio synthesis.

Key Contributions

1. WaveNet-Style Autoencoder: The authors propose a novel autoencoder model that effectively conditions an autoregressive decoder on temporal codes derived directly from raw audio waveforms. This model negates the necessity for external conditioning, thus providing a more autonomous synthesis method compared to conventional approaches.
2. NSynth Dataset: The introduction of NSynth, a large-scale dataset of musical notes, is crucial for comprehensive evaluation and further development of generative audio models. With approximately 306,000 four-second musical notes, it surpasses previous datasets in scale and variety, thus offering a robust platform for audio modeling experiments.

Technical Approach

The WaveNet autoencoder is designed to learn temporal embeddings that represent the longer-term structure of audio signals. This is achieved through a WaveNet-like encoder that generates temporally distributed embeddings, followed by a decoder that reconstructs the original audio using these embeddings. The process allows for a nuanced capture of audio characteristics such as tone, timbre, and dynamics over extended timeframes.

Dataset and Experiments

The NSynth dataset encompasses a diverse collection of pitches and instruments, providing a fertile ground for testing audio generation models. The dataset's extensive size and quality facilitate both qualitative and quantitative evaluations. The paper contrasts the performance of the proposed WaveNet autoencoder against baseline spectral autoencoder models, using tasks like reconstruction and interpolation to assess efficacy.

Results and Observations

The qualitative evaluation revealed that the WaveNet autoencoder adeptly reconstructs realistic and expressive sounds, outperforming spectral baseline models in capturing intricate audio features. The model also successfully demonstrates meaningful interpolation between different instruments, suggesting the learned manifold is robust for morphing timbres.

Quantitatively, the classification results for pitch and quality derived from reconstructions further reinforce the superior performance of the WaveNet approach. The paper also discusses the extent of disentanglement between pitch and timbre, showcasing the potential for conditioned generation across different pitches while retaining timbral identity.

Implications and Future Directions

The advancements illustrated in this paper have significant implications for the field of neural audio synthesis. The WaveNet autoencoder model extends the capabilities of previous generative models, particularly in terms of modeling long-term dependencies without external conditioning. The NSynth dataset offers a benchmarking tool that could accelerate progress and innovation in audio synthesis research.

Future research might focus on overcoming the limitations of temporal context within these models and exploring new datasets that could foster understanding and innovation in more complex audio synthesis tasks. There is also potential for expanding these techniques into multi-note or polyphonic contexts, increasing their applicability in more generalized music generation and transcription tasks.