InstructSpeech: Following Speech Editing Instructions via Large Language Models
Rongjie Huang, Ruofan Hu, Yongqi Wang, Zehan Wang, Xize Cheng, Ziyue Jiang, Zhenhui Ye, Dongchao Yang
Abstract
Instruction-guided speech editing aims to follow the user's natural language instruction to manipulate the semantic and acoustic attributes of a speech. In this work, we construct triplet paired data (instruction, input speech, output speech) to alleviate data scarcity and train a multi-task large language model named InstructSpeech. To mitigate the challenges of accurately executing user's instructions, we 1) introduce the learned task embeddings with a fine-tuned Flan-T5-XL to guide the generation process towards the correct generative task; 2) include an extensive and diverse set of speech editing and processing tasks to enhance model capabilities; 3) investigate chain-of-thought reasoning for free-form semantic content editing; and 4) propose a hierarchical adapter that effectively updates a small portion of parameters for generalization to new tasks. To assess instruction speech editing in greater depth, we introduce a benchmark evaluation with contrastive instruction-speech pre-training (CISP) to test the speech quality and instruction-speech alignment faithfulness. Experimental results demonstrate that InstructSpeech achieves state-of-the-art results in eleven tasks, for the first time unlocking the ability to edit speech's acoustic and semantic attributes following a user's instruction. Audio samples are available at https://InstructSpeech.github.io
BibTeX
@inproceedings{
huang2024instructspeech,
title={InstructSpeech: Following Speech Editing Instructions via Large Language Models},
author={Rongjie Huang and Ruofan Hu and Yongqi Wang and Zehan Wang and Xize Cheng and Ziyue Jiang and Zhenhui Ye and Dongchao Yang and Luping Liu and Peng Gao and Zhou Zhao},
booktitle={Forty-first International Conference on Machine Learning},
year={2024},
url={https://openreview.net/forum?id=xlWcdtCyOC}
}