MoMusic: A Motion-Driven Human-AI Collaborative Music Composition and Performing System
Weizhen Bian, Yijin Song, Nianzhen Gu, Tin Yan Chan, Tsz To Lo, Tsun Sun Li, King Chak Wong, Wei Xue
Abstract
The significant development of artificial neural network architectures has facilitated the increasing adoption of automated music composition models over the past few years. However, most existing systems feature algorithmic generative structures based on hard code and predefined rules, generally excluding interactive or improvised behaviors. We propose a motion based music system, MoMusic, as a AI real time music generation system. MoMusic features a partially randomized harmonic sequencing model based on a probabilistic analysis of tonal chord progressions, mathematically abstracted through musical set theory. This model is presented against a dual dimension grid that produces resulting sounds through a posture recognition mechanism. A camera captures the users' fingers' movement and trajectories, creating coherent, partially improvised harmonic progressions. MoMusic integrates several timbrical registers, from traditional classical instruments such as the piano to a new ''human voice instrument'' created using a voice conversion technique. Our research demonstrates MoMusic's interactiveness, ability to inspire musicians, and ability to generate coherent musical material with various timbrical registers. MoMusic's capabilities could be easily expanded to incorporate different forms of posture controlled timbrical transformation, rhythmic transformation, dynamic transformation, or even digital sound processing techniques.
BibTeX
@article{Bian_Song_Gu_Chan_Lo_Li_Wong_Xue_Alonso Trillo_2024, title={MoMusic: A Motion-Driven Human-AI Collaborative Music Composition and Performing System}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/26907}, DOI={10.1609/aaai.v37i13.26907}, abstractNote={The significant development of artificial neural network architectures has facilitated the increasing adoption of automated music composition models over the past few years. However, most existing systems feature algorithmic generative structures based on hard code and predefined rules, generally excluding interactive or improvised behaviors. We propose a motion based music system, MoMusic, as a AI real time music generation system. MoMusic features a partially randomized harmonic sequencing model based on a probabilistic analysis of tonal chord progressions, mathematically abstracted through musical set theory. This model is presented against a dual dimension grid that produces resulting sounds through a posture recognition mechanism. A camera captures the users’ fingers’ movement and trajectories, creating coherent, partially improvised harmonic progressions. MoMusic integrates several timbrical registers, from traditional classical instruments such as the piano to a new ’’human voice instrument’’ created using a voice conversion technique. Our research demonstrates MoMusic’s interactiveness, ability to inspire musicians, and ability to generate coherent musical material with various timbrical registers. MoMusic’s capabilities could be easily expanded to incorporate different forms of posture controlled timbrical transformation, rhythmic transformation, dynamic transformation, or even digital sound processing techniques.}, number={13}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Bian, Weizhen and Song, Yijin and Gu, Nianzhen and Chan, Tin Yan and Lo, Tsz To and Li, Tsun Sun and Wong, King Chak and Xue, Wei and Alonso Trillo, Roberto}, year={2024}, month={Jul.}, pages={16057-16062} }