AV-RIR: Audio-Visual Room Impulse Response Estimation (2312.00834v2)

Abstract: Accurate estimation of Room Impulse Response (RIR), which captures an environment's acoustic properties, is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and the visual cues of its corresponding environment. AV-RIR builds on a novel neural codec-based architecture that effectively captures environment geometry and materials properties and solves speech dereverberation as an auxiliary task by using multi-task learning. We also propose Geo-Mat features that augment material information into visual cues and CRIP that improves late reverberation components in the estimated RIR via image-to-RIR retrieval by 86%. Empirical results show that AV-RIR quantitatively outperforms previous audio-only and visual-only approaches by achieving 36% - 63% improvement across various acoustic metrics in RIR estimation. Additionally, it also achieves higher preference scores in human evaluation. As an auxiliary benefit, dereverbed speech from AV-RIR shows competitive performance with the state-of-the-art in various spoken language processing tasks and outperforms reverberation time error score in the real-world AVSpeech dataset. Qualitative examples of both synthesized reverberant speech and enhanced speech can be found at https://www.youtube.com/watch?v=tTsKhviukAE.

• The paper introduces AV-RIR, a novel multi-task learning framework that leverages audio and visual data to improve room impulse response estimation by 36%-63%.
• It employs a specialized neural codec architecture with Geo-Mat feature extraction and Contrastive RIR-Image Pre-training to integrate environmental geometry and material properties.
• Empirical evaluations show robust improvements in acoustic metrics, speech recognition, and speaker verification on real-world datasets like AVSpeech.

AV-RIR: Audio-Visual Room Impulse Response Estimation

Introduction

The paper "AV-RIR: Audio-Visual Room Impulse Response Estimation" (2312.00834) addresses the challenges in Room Impulse Response (RIR) estimation, which is crucial for applications in speech processing and augmented/virtual reality (AR/VR). The authors propose AV-RIR, a novel multi-modal multi-task learning framework that leverages both reverberant speech signals and visual inputs from the environment to estimate RIR. This novel approach not only enhances speech dereverberation but also significantly improves the estimation of the late reverberation components through a process called Contrastive RIR-Image Pre-training (CRIP).

Methodology

AV-RIR is built on a sophisticated neural codec-based architecture designed to capture environmental geometry and material properties. The method integrates audio and visual data to solve the primary task of RIR estimation, with speech dereverberation considered as an auxiliary task. The architecture features specialized encoders and decoders, a Residual Vector Quantizer (RVQ), and utilizes Geo-Mat features that embed material and geometric information of the environment for more accurate RIR estimation.

Figure 1: Overview of AV-RIR: Given a source reverberant speech in any environment, AV-RIR estimates the RIR from the reverberant speech using additional visual cues. The estimated RIR can be used to transform any target clean speech as if it is spoken in that environment.

The multi-modal approach combines the strengths of visual and auditory cues, substantially outperforming conventional audio-only or visual-only methods. It features a dual-branch system in its architecture—one branch dedicated to RIR estimation and the other targeting speech dereverberation—thereby enabling a comprehensive learning objective that encompasses both tasks.

Geo-Mat Feature and CRIP

AV-RIR introduces Geo-Mat features to augment the learning process by providing essential material information and geometric context from panoramic images. This approach is backed by SOTA object tagging and depth mapping for accurate representation of materials' absorption coefficients.

Figure 2: The computation pipeline of Geo-Mat feature map. The first two channels of the Geo-Mat feature ($\mathcal{I}_{G}$) comprise the absorption coefficients ($\mathcal{AC}$) of each acoustic material.

Additionally, the CRIP module is designed to refine late reverberation components. By retrieving relevant RIR data from an extensive database using joint audio-visual embeddings, CRIP enhances the late-stage RIR estimation by supplementing noise-like components, which are traditionally difficult to estimate accurately.

Figure 3: Illustration of CRIP training. Like CLIP, we propose two networks, one to encode a panoramic image and the other to encode the RIR to learn a joint embedding space between both.

Results and Performance

Empirical evaluations demonstrate significant improvements over previous models, with AV-RIR surpassing others by 36% - 63% in various acoustic metrics. The integration of visual and audio data allows AV-RIR to excel in both quantitative metrics and human evaluations, making it a robust solution for RIR and dereverberated speech synthesis.

The experiments conducted illustrate that AV-RIR achieves superior performance across several tasks, including speech recognition and speaker verification, by facilitating more accurate reverberation time error scores and improving dereverberation outcomes in real-world datasets such as AVSpeech.

Figure 4: Qualitative Results. (Left) We show the Geo-Mat feature generated using our approach. The cushion chairs with a similar material absorption property are represented in green. The table and window with similar material are represented in red.

Conclusion

AV-RIR is a sophisticated framework that sets a new standard for RIR estimation by utilizing both audio and visual cues. The integration of CRIP ensures superior performance in estimating late reverberation components, while the multi-task learning approach improves both RIR estimation and speech dereverberation. The framework's capabilities pave the way for future applications in AR/VR and provide robust solutions for a variety of speech processing tasks. Future research could explore extending AV-RIR's methodology to accommodate multi-channel inputs and real-time application scenarios.