Leveraging Out-of-Domain Noise for Unsupervised Domain Adaptation in Speech Enhancement

Abstract

When there’s a mismatch between the training and test domains, supervised speech enhancement (SE) models trained on synthetic paired noisy-clean data often struggle in real-world scenarios, highlighting the industry’s strong demand for unsupervised training and domain adaptation methods. In this study, we introduce PHA-ReMixIT, a novel approach for leveraging out-of-domain (OOD) noise signals to enhance unsupervised domain adaptation in SE. Our method builds upon the state-of-the-art ReMixIT by introducing a paired unsupervised remixing technique, which augments the diversity of target domain training data with OOD noise signals. We further propose a heterogeneous noise invariant training to align the OOD augmented noisy mixtures with their paired heterogeneous counterparts, encouraging the model to output cleaner speech. Additionally, an adaptive focal weighting mechanism is also introduced to dynamically emphasize the data importance of both in-domain and OOD noisy mixtures during model adaptation. Experiments on CHiME-7 unsupervised domain adaptation for conversational speech enhancement (UDASE) task demonstrate that PHA-ReMixIT significantly outperforms the ReMixIT baseline, boosting SE performance on both real and synthesized test sets.