MuPT: A Generative Symbolic Music Pretrained Transformer
Xingwei Qu, yuelin bai, Yinghao Ma, Ziya Zhou, Ka Man Lo, Jiaheng Liu, Ruibin Yuan, Lejun Min
Abstract
In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music. While the prevalent use of MIDI in music modeling is well-established, our findings suggest that LLMs are inherently more compatible with ABC Notation, which aligns more closely with their design and strengths, thereby enhancing the model's performance in musical composition. To address the challenges associated with misaligned measures from different tracks during generation, we propose the development of a $\underline{S}$ynchronized $\underline{M}$ulti-$\underline{T}$rack ABC Notation ($\textbf{SMT-ABC Notation}$), which aims to preserve coherence across multiple musical tracks. Our contributions include a series of models capable of handling up to 8192 tokens, covering 90\% of the symbolic music data in our training set. Furthermore, we explore the implications of the $\underline{S}$ymbolic $\underline{M}$usic $\underline{S}$caling Law ($\textbf{SMS Law}$) on model performance. The results indicate a promising research direction in music generation, offering extensive resources for further research through our open-source contributions.
BibTeX
@inproceedings{
qu2025mupt,
title={Mu{PT}: A Generative Symbolic Music Pretrained Transformer},
author={Xingwei Qu and yuelin bai and Yinghao Ma and Ziya Zhou and Ka Man Lo and Jiaheng Liu and Ruibin Yuan and Lejun Min and Xueling Liu and Tianyu Zhang and Xeron Du and Shuyue Guo and Yiming Liang and Yizhi LI and Shangda Wu and Junting Zhou and Tianyu Zheng and Ziyang Ma and Fengze Han and Wei Xue and Gus Xia and Emmanouil Benetos and Xiang Yue and Chenghua Lin and Xu Tan and Wenhao Huang and Jie Fu and Ge Zhang},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=iAK9oHp4Zz}
}