DATA2VEC-SG: Improving Self-Supervised Learning Representations for Speech Generation Tasks
Heming Wang, Yao Qian, Hemin Yang, Naoyuki Kanda, Peidong Wang, Takuya Yoshioka, Xiaofei Wang, Yiming Wang
Abstract
Self-supervised learning has been successfully applied to various speech recognition and understanding tasks. However, for generative tasks such as speech enhancement and speech separation, most self-supervised speech representations did not show substantial improvements. To deal with this problem, in this paper, we propose data2vec-SG (Speech Generation), which is a teacher-student learning framework that addresses speech generation tasks. Our data2vec-SG introduces a reconstruction module into data2vec [1] and enforces the representations to contain not only the semantic information but also the acoustic knowledge to generate clean speech waveforms. Experimental results demonstrate that the proposed framework boosts the performance of various speech generation tasks including speech enhancement, speech separation, and packet loss concealment. Meanwhile, the learned representation is also capable of helping other downstream tasks, which is demonstrated by the good performance in the speech recognition task in both clean and noisy conditions.
BibTeX
@inproceedings{icassp2023_data2vecsgimprov,
title = {DATA2VEC-SG: Improving Self-Supervised Learning Representations for Speech Generation Tasks},
author = {Heming Wang and Yao Qian and Hemin Yang and Naoyuki Kanda and Peidong Wang and Takuya Yoshioka and Xiaofei Wang and Yiming Wang and Shujie Liu and Zhuo Chen and DeLiang Wang and Michael Zeng},
booktitle = {ICASSP 2023},
year = {2023}
}