Egocentric Speaker Diarization with Vision-Guided Clustering and Adaptive Speech Re-detection

Abstract

Speaker diarization aims to identify "who spoke when" in multi-person conversational scenarios. State-of-the-art audio-only diarization methods divide the task into multi-stages of speech segmentation, neural speaker embedding and unsupervised clustering. Egocentric speaker diarization (i.e., diarization in egocentric videos) is characterized by natural conversational scenarios involving a variety of noisy backgrounds, changing sound levels and overlapping speech. These cause difficulty in predicting the number of speakers from audio input, and heavily influence the performance of audio-based clustering. Although audio-visual modeling has been studied recently to enhance speaker diarization, unreliable visual information in egocentric videos may even degrade the performance. In this work, we propose a unified audio-visual diarization framework by incorporating visual guidance into the audio-only diarization pipeline. In addition, we also propose an adaptive speech re-detection strategy to detect and assign speaker identity to the mistakenly undetected audio segments. Experiments on the Ego4D dataset show that our method achieves state-of-the-art diarization performance in challenging egocentric scenarios. The code and model weights are available at https://github.com/YellowRiver2001/EgoDiarization.