Phonetic-aware speaker embedding for far-field speaker verification

Abstract: When a speaker verification (SV) system operates far from the sound sourced, significant challenges arise due to the interference of noise and reverberation. Studies have shown that incorporating phonetic information into speaker embedding can improve the performance of text-independent SV. Inspired by this observation, we propose a joint-training speech recognition and speaker recognition (JTSS) framework to exploit phonetic content for far-field SV. The framework encourages speaker embeddings to preserve phonetic information by matching the frame-based feature maps of a speaker embedding network with wav2vec's vectors. The intuition is that phonetic information can preserve low-level acoustic dynamics with speaker information and thus partly compensate for the degradation due to noise and reverberation. Results show that the proposed framework outperforms the standard speaker embedding on the VOiCES Challenge 2019 evaluation set and the VoxCeleb1 test set. This indicates that leveraging phonetic information under far-field conditions is effective for learning robust speaker representations.

• The paper presents the JTSS framework that jointly trains speaker verification and speech recognition to incorporate phonetic information.
• It leverages unsupervised phonetic extraction from a pre-trained wav2vec 2.0 to preserve low-level acoustic dynamics in noisy and reverberant environments.
• The approach achieves a 12.9% reduction in EER and a 14.4% reduction in minDCF, demonstrating significant improvements over baseline models.

Phonetic-aware Speaker Embedding for Far-field Speaker Verification

Introduction

The presented paper, "Phonetic-aware Speaker Embedding for Far-field Speaker Verification" (2311.15627), addresses the challenges associated with speaker verification (SV) in far-field conditions, where noise and reverberation significantly degrade performance. Traditional SV techniques, such as using Gaussian Mixture Models (GMMs), i-vectors, and more recent deep learning approaches like Time Delay Neural Networks (TDNNs) and ECAPA-TDNN, have shown competency primarily under near-field conditions. The degradation of SV performance in far-field settings necessitates novel approaches.

Leveraging prior observations that phonetic information can enhance SV performance, this study introduces a joint-training framework—termed Joint-Training of Speech and Speaker recognition (JTSS)—which integrates phonetic content into speaker embedding learning. This framework aims to mitigate the challenges posed by far-field conditions by aligning phonetic information extracted via wav2vec 2.0 vectors with frame-based feature maps, thus preserving low-level acoustic dynamics and improving speaker recognition robustness.

Methods

The proposed JTSS framework incorporates both speech recognition and speaker verification tasks without requiring manual phonetic transcriptions, employing a pre-trained wav2vec 2.0 model for phonetic extraction. This unsupervised strategy allows the preservation of acoustic dynamics critical to speaker identity, addressing performance degradation due to noise and reverberation in far-field environments.

Figure 1: Framework of joint training of speech recognition and speaker classification (JTSS). The utterance-based speaker network in the speaker classification part comprises a pooling layer and a fully connected layer.

The speech recognition component and speaker classification component share frame-level layers to ensure the preservation of phonetic information. The JTSS framework jointly optimizes both tasks using a composite loss function that integrates the AAMSoftmax loss and a cosine similarity metric between phonetic content representations.

Results

The JTSS framework was evaluated using the VOiCES Challenge 2019 and VoxCeleb datasets. It demonstrated superior performance relative to baseline models employing ECAPA-TDNN and x-vector architectures. Notably, the ECAPA-TDNN variant of JTSS achieved a 12.9% reduction in Equal Error Rate (EER) and a 14.4% reduction in minimum Detection Cost Function (minDCF) compared to its baseline, clearly evidencing the efficacy of incorporating phonetic information in improving far-field SV.

On both clean and noisy Vox-O datasets, JTSS outperformed traditional methods, reinforcing its robustness under varied acoustic conditions. The reduced impact of noise and reverberation on JTSS performance highlights its potential to significantly enhance speaker discrimination capabilities in real-world settings.

Discussion

The study supports the hypothesis that integrating phonetic information, particularly from lower-level frame representations, enhances speaker verification under adverse acoustic conditions. The proposed framework offers a promising direction for further development of SV systems that are resilient to environmental noise and reverberation.

Future work could explore the refinement of phonetic extraction techniques and the integration of these frameworks into broader biometric security systems. Additionally, optimizing hyperparameters such as $\lambda$, which determines the balance between phonetic and speaker loss contributions, could further refine the proposed methodology.

Conclusion

The paper successfully demonstrates that the JTSS framework, utilizing phonetic information extracted via unsupervised learning from a pre-trained wav2vec 2.0 model, significantly improves far-field speaker verification performance. Through robust empirical validation, it establishes a meaningful contribution to the ongoing evolution of speaker recognition methodologies, especially in challenging acoustic environments. Further research and development could amplify the impact and applicability of these findings, opening avenues for advanced biometric security systems.