Semi-Supervised Speaker Diarization Using Graph Transformers and LLMs on Naturalistic Apollo 11 Data

Abstract

Speaker diarization is the process of segmenting and tagging audio streams based on speaker identity. Traditional methods face significant challenges in real-world scenarios when applied to spontaneous multi-speaker conversational speech. The Fearless Steps Apollo 11 corpus (FS-A11) presents real-world challenges such as inconsistent number of speakers per channel, varying speaker utterance duration, and diverse acoustic environments. In this study, we introduce a novel speaker diarization framework that leverages Large Language Models (LLMs) to generate initial speaker change labels by analyzing both speech content and conversational dynamics. These labels are further refined using a proposed multi-segmentation system to obtain refined speaker turns. Finally, we propose a Graph Transformer architecture to build robust speaker embeddings by modeling relationships between audio segments. The embeddings are then clustered using agglomerative hierarchical clustering (AHC) to produce the final diarization output. Evaluation of the proposed system demonstrates a relative improvement in Diarization Error Rate (DER) of +12.8% over baseline systems on the FS-A11 dataset. Furthermore, we analyze and track 5 key speaker roles over the entire Apollo-11 mission to analyze primary speaker engagement and conversational dynamics across mission communication channels.