AIConfPaper
← Search

Sanyuan Chen

14 accepted papers

2026

MR-FLOWDPO: MULTI-REWARD DIRECT PREFERENCE OPTIMIZATION FOR FLOW-MATCHING TEXT-TO-MUSIC GENERATION

Alon Ziv, The Hebrew University of Jerusalem, Sanyuan Chen, Yossi Adi, The Hebrew University of Jerusalem, Wei-Ning Hsu
ICASSP 2026oral

A key challenge in music generation models is their lack of direct alignment with human preferences, as music evaluation is inherently subjective and varies widely across individuals. We introduce MR-FlowDPO, a novel approach that enhances flow-matching-based music generation models - a major class…

Cited by 0Source ↗PDF ↗Scholar ↗
2026

Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning

Apoorv Vyas, Heng-Jui Chang, Cheng-Fu Yang, Po-Yao Huang, Luya Gao, Julius Richter +6
CVPR 2026

We introduce Perception Encoder-Audiovisual, PE-AV, a new family of encoders for audio and video understanding trained with scaled contrastive learning. Building on PE, PE-AV makes several key contributions to extend representations to audio, and natively support joint embeddings across audio-video,

Cited by 0Source ↗code ↗Scholar ↗
2025

Autoregressive Speech Synthesis without Vector Quantization

Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu +6
ACL 2025long

We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which is typically designed for audio co…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao +6
EMNLP 2024finding

Recent advancements in large language models (LLMs) have expanded their scope in natural language processing (NLP) to encompass multimodal functions. However, integrating listening capabilities effectively remains a significant challenge for generalization and complex auditory task execution. In thi…

Cited by 70Source ↗PDF ↗code ↗Scholar ↗
2023

BEATs: Audio Pre-Training with Acoustic Tokenizers

Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie LIU, Daniel Tompkins, Zhuo Chen +3
ICML 2023oral

We introduce a self-supervised learning (SSL) framework BEATs for general audio representation pre-training, where we optimize an acoustic tokenizer and an audio SSL model by iterations. Unlike the previous audio SSL models that employ reconstruction loss for pre-training, our audio SSL model is tra…

Cited by 341Source ↗PDF ↗code ↗Scholar ↗
2022

Improving Self-Supervised Learning for Speech Recognition with Intermediate Layer Supervision

Chengyi Wang, Yu Wu, Sanyuan Chen, Shujie Liu, Jinyu Li, Yao Qian +1
ICASSP 2022accepted

Recently, pioneer work finds that self-supervised pre-training methods can improve multiple downstream speech tasks, because the model utilizes bottom layers to learn speaker-related information and top layers to encode content-related information. Since the network capacity is limited, we believe t…

Cited by 0Source ↗Scholar ↗
2022

Large-Scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification

Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu +2
ICASSP 2022accepted

The speech representations learned from large-scale unlabeled data have shown better generalizability than those from supervised learning and thus attract a lot of interest to be applied for various downstream tasks. In this paper, we explore the limits of speech representations learned by different…

Cited by 0Source ↗Scholar ↗
2022

MPII: Multi-Level Mutual Promotion for Inference and Interpretation

Yan Liu, Sanyuan Chen, Yazheng Yang, Qi Dai
ACL 2022long

In order to better understand the rationale behind model behavior, recent works have exploited providing interpretation to support the inference prediction. However, existing methods tend to provide human-unfriendly interpretation, and are prone to sub-optimal performance due to one-side promotion,…

Cited by 6Source ↗PDF ↗code ↗Scholar ↗
2022

Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training

Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu +5
ICASSP 2022accepted

Self-supervised learning (SSL) is a long-standing goal for speech processing, since it utilizes large-scale unlabeled data and avoids extensive human labeling. Recent years have witnessed great successes in applying self-supervised learning in speech recognition, while limited exploration was attemp…

Cited by 0Source ↗Scholar ↗
2021

C2C-GenDA: Cluster-to-Cluster Generation for Data Augmentation of Slot Filling

Yutai Hou, Sanyuan Chen, Wanxiang Che, Cheng Chen, Ting Liu
AAAI 2021technical

Slot filling, a fundamental module of spoken language understanding, often suffers from insufficient quantity and diversity of training data. To remedy this, we propose a novel Cluster-to-Cluster generation framework for Data Augmentation (DA), named C2C-GenDA. It enlarges the training set by recons…

Cited by 21Source ↗PDF ↗code ↗Scholar ↗
2021

Don't Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer

Sanyuan Chen, Yu Wu, Zhuo Chen, Takuya Yoshioka, Shujie Liu, Jin-Yu Li +1
ICASSP 2021accepted

With its strong modeling capacity that comes from a multi-head and multi-layer structure, Transformer is a very powerful model for learning a sequential representation and has been successfully applied to speech separation recently. However, multi-channel speech separation sometimes does not necessa…

Cited by 0Source ↗Scholar ↗
2021

Microsoft Speaker Diarization System for the Voxceleb Speaker Recognition Challenge 2020

Xiong Xiao, Naoyuki Kanda, Zhuo Chen, Tianyan Zhou, Takuya Yoshioka, Sanyuan Chen +7
ICASSP 2021accepted

This paper describes the Microsoft speaker diarization system for monaural multi-talker recordings in the wild, evaluated at the diarization track of the VoxCeleb Speaker Recognition Challenge (VoxSRC) 2020. We will first explain our system design to address issues in handling real multi-talker reco…

Cited by 0Source ↗Scholar ↗