Aligned Contrastive Learning for Text-to-Music Retrieval

Abstract

This paper proposes aligned contrastive learning for text-to-music retrieval. The proposed method introduces a new similarity measure, 'aligned similarity', which captures the frame-level and token-level correspondence within text and audio sequences. Unlike traditional approaches that aggregate sequence into clip-level and sentence-level embeddings, our method aligns the text token exhibiting the highest cosine similarity with each temporal frame of the audio sequence and averages these maximum similarity values across the entire sequence. This approach enables the capture of fine-grained relationships between audio and text that are often overlooked when sequences are aggregated into a single embedding. Retrieval experiments show significant performance improvements, with a notable gain being a 17.8% increase in Recall@5. Moreover, the alignment elucidates how specific audio frames correlate with textual tokens, enhancing the model's transparency and interpretability.