ChatMusician: Understanding and Generating Music Intrinsically with LLM
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu
Abstract
While LLMs demonstrate impressive capabilities in musical knowledge, we find that music reasoning is still an unsolved task.We introduce ChatMusician, an open-source large language model (LLM) that integrates intrinsic musical abilities. It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.ChatMusician can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. Interestingly, endowing musical abilities does not harm language abilities, even achieving a slightly higher MMLU score.ChatMusician is capable of composing well-structured, full-length music, condition on texts, chords, melodies, motifs, musical forms, etc.On our meticulously curated college-level music understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and GPT-3.5 by a noticeable margin. We show that ChatMusician preserves or even surpasses the original LLaMA2 7B’s language abilities by evaluating on MMLU benchmark.Our work reveals that LLMs can be an excellent compressor for music, which can be seen as humanity’s creative language, but there remains significant territory to be conquered.We release our 5B token music-language corpora MusicPiles, the collected MusicTheoryBench, code, model and demo.
BibTeX
@inproceedings{yuan-etal-2024-chatmusician,
title = "{C}hat{M}usician: Understanding and Generating Music Intrinsically with {LLM}",
author = "Yuan, Ruibin and
Lin, Hanfeng and
Wang, Yi and
Tian, Zeyue and
Wu, Shangda and
Shen, Tianhao and
Zhang, Ge and
Wu, Yuhang and
Liu, Cong and
Zhou, Ziya and
Xue, Liumeng and
Ma, Ziyang and
Liu, Qin and
Zheng, Tianyu and
Li, Yizhi and
Ma, Yinghao and
Liang, Yiming and
Chi, Xiaowei and
Liu, Ruibo and
Wang, Zili and
Lin, Chenghua and
Liu, Qifeng and
Jiang, Tao and
Huang, Wenhao and
Chen, Wenhu and
Fu, Jie and
Benetos, Emmanouil and
Xia, Gus and
Dannenberg, Roger and
Xue, Wei and
Kang, Shiyin and
Guo, Yike",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-acl.373/",
doi = "10.18653/v1/2024.findings-acl.373",
pages = "6252--6271"
}