MotionComposer: Enhancing Rhythmic Music Generation with Adaptive Retrieval Reference

Abstract

With the rise of the AIGC era, rhythmic music generation has extensive applications, particularly with the surge in motion video creation. However, generating music that is rhythmically synchronized and stylistically aligned with motion video presents significant challenges. Although existing methods have made progress, they still face difficulties in producing high-quality long-term music, particularly when addressing complex rhythmic patterns and maintaining style-consistent musical chords. In this work, we present MotionComposer, a novel retrieval-augmented, easy-to-hard training approach designed to enhance rhythmic music generation. By leveraging the inherent alignment between motion rhythms and music beats, we first tackle the simpler task of beat prediction with BeatNet, which predicts music beats by analyzing motion patterns. To address the complex musical chord generation, we propose ChordNet, a retrieval-augmented network that integrates external data to enrich chord generation. Additionally, to minimize the impact of irrelevant retrievals, we design RAGate, a retrieval adaptive module that selectively filters out low-relevance retrieval references during the retrieval process. Extensive experiments across three scenarios (i.e., dance, figure skating, and floor exercise) demonstrate that our approach significantly enhances video soundtrack generation, achieving new state-of-the-art performance. Our project is available at https://beria-moon.github.io/Soundtrackyour-Motion/.