The Whisper Speaker Identification (WSI) framework revolutionizes speaker recognition by harnessing the power of the Whisper speech recognition model’s multilingual pre-training. By repurposing Whisper’s encoder, WSI generates robust speaker embeddings optimized through innovative loss techniques, including online hard triplet loss and self-supervised NT-Xent loss. Extensive evaluations across diverse multilingual datasets reveal WSI’s superior performance over existing methods, significantly boosting equal error rate reduction and AUC scores. WSI’s ability to efficiently manage various linguistic inputs makes it an exceptional tool for both multilingual and single-language contexts.
Whisper Speaker Identification: Leveraging Pre-Trained Multilingual Transformers for Robust Speaker Embeddings
The paper presents Whisper Speaker Identification (WSI), a novel framework leveraging the Whisper speech recognition model’s multilingual pre-training for robust speaker identification. The approach repurposes Whisper’s encoder to generate speaker embeddings optimized using an online hard triplet loss and self-supervised NT-Xent loss. Extensive testing on various multilingual datasets shows that WSI outperforms current state-of-the-art methods in reducing equal error rates (EER) and increasing AUC scores. The framework proves effective across both multilingual and single-language contexts due to its capacity to handle diverse linguistic inputs efficiently.
Key Points
- WSI utilizes a pre-trained multilingual ASR model, Whisper, for extracting robust, language-agnostic speaker embeddings.
- By leveraging joint loss optimization, WSI effectively enhances speaker discrimination in multilingual environments.
- WSI demonstrates superior performance over established speaker recognition models across various datasets and languages.
Action Items
- Consider adopting multilingual pre-trained models in your projects to improve model robustness and performance across diverse scenarios.
- Use joint loss optimization techniques, such as combining triplet and self-supervised losses, to enhance the discriminative power of your models.
- Explore leveraging existing large-scale ASR models for tasks beyond speech recognition, such as speaker identification, to benefit from their comprehensive linguistic representations.