FUTGA-MIR: Enhancing Fine-grained and Temporally-aware Music Understanding with Music Information Retrieval

Abstract

Recent music large language models (music LLMs) have shown great potential in music understanding through large-scale multimodal pre-training. While some existing music LLMs have been augmented with temporally-aware music captions, music information retrieval (MIR) features conventionally do not exist in music caption datasets, thus neglected by music-LLMs. To bridge the gap between recent music LLMs and conventional music information retrieval tasks, we propose FUTGA-MIR (Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation with Music Information Retrieval) to enhance the existing music LLMs by augmenting them with MIR features and aligning with human feedback. We calibrate the original positional distribution of music clips in the created pre-training synthetic music captions, conditioned on MIR features. The redistributed synthetic dataset better respects realistic MIR distributions of music and is tightly aligned with realistic music captions as well. In addition, We incorporate human-annotated music captions and MIR features from the Harmonixset dataset for an additional fine-tuning step, which addresses sim-to-real domain gaps and improves performance on realistic music tracks. We further evaluate FUTGA-MIR on several downstream tasks, including music classification, retrieval, and generation, and demonstrate the versatile capacities of FUTGA-MIR and better generation quality compared with previous music captioning models. Generated temporal-aware music descriptions are illustrated in our demonstration https://namburiamit.github.io/futga-music.github.io/.