Yuchen Hu — accepted AI-conference papers

2026

ALIGNING GENERATIVE SPEECH ENHANCEMENT WITH PERCEPTUAL FEEDBACK

Haoyang Li, Nana Hou, Yuchen Hu, Jixun Yao, Sabato Marco Siniscalchi, Norwegian University of Science and Technology +4

ICASSP 2026oral

Language Model (LM)-based speech enhancement (SE) has recently emerged as a promising direction, but existing approaches predominantly rely on token-level likelihood objectives that weakly reflect human perception. This mismatch limits progress, as optimizing signal accuracy does not always improve…

Cited by 0 Source ↗PDF ↗Scholar ↗

2026

Attributing Response to Context: A Jensen–Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation

Ruizhe Li, Chen Chen, Yuchen Hu, Yanjun Gao, Xi Wang, Emine Yilmaz

ICLR 2026poster

Retrieval-Augmented Generation (RAG) leverages large language models (LLMs) combined with external contexts to enhance the accuracy and reliability of generated responses. However, reliably attributing generated content to specific context segments, context attribution, remains challenging due to th…

Cited by 0 Source ↗Scholar ↗

2026

HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction

Muyan Zhong, Erfei Cui, Sen Xing, Weiyun Wang, Wen Wu, Yuchen Hu +5

CVPR 2026

Multimodal large language models (MLLMs) have expanded from vision-language systems to include audio, unlocking new capabilities in cross-modal reasoning and interaction. To address the limitation that existing benchmarks focus mainly on perception tasks and lack a unified cognitive evaluation frame

Cited by 0 Source ↗Scholar ↗

2025

AnalyticKWS: Towards Exemplar-Free Analytic Class Incremental Learning for Small-footprint Keyword Spotting

Yang Xiao, Peng Tianyi, Rohan Kumar Das, Yuchen Hu, Huiping Zhuang

ACL 2025finding

Keyword spotting (KWS) offers a vital mechanism to identify spoken commands in voice-enabled systems, where user demands often shift, requiring models to learn new keywords continually over time. However, a major problem is catastrophic forgetting, where models lose their ability to recognize earlie…

Cited by 0 Source ↗PDF ↗Scholar ↗

2025

Audio Large Language Models Can Be Descriptive Speech Quality Evaluators

Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang +2

ICLR 2025poster

An ideal multimodal agent should be aware of the quality of its input modalities. Recent advances have enabled large language models (LLMs) to incorporate auditory systems for handling various speech-related tasks. However, most audio LLMs remain unaware of the quality of the speech they process. Th…

Cited by 1 Source ↗PDF ↗Scholar ↗

2025

Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning

Chengwei Qin, Wenhan Xia, Fangkai Jiao, Chen Chen, Yuchen Hu, Bosheng Ding +2

ACL 2025long

Large language models (LLMs) have shown impressive few-shot generalization on many tasks via in-context learning (ICL). Despite their success in showing such emergent abilities, the scale and complexity of larger models also lead to unprecedentedly high computational demands and deployment challenge…

Cited by 0 Source ↗PDF ↗Scholar ↗

2025

GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

Jixun Yao, Hexin Liu, Chen Chen, Yuchen Hu, EngSiong Chng, Lei Xie

ICLR 2025poster

Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in nois…

Cited by 1 Source ↗PDF ↗Scholar ↗

2025

Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?

Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, Bosheng Ding +2

ACL 2025finding

Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidenta…

Cited by 0 Source ↗PDF ↗Scholar ↗

2025

SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis

Helin Wang, Meng Yu, Jiarui Hai, Chen Chen, Yuchen Hu, Rilin Chen +2

ICASSP 2025accepted

In this paper, we introduce SSR-Speech, a neural codec autoregressive model designed for stable, safe, and robust zero-shot text-based speech editing and text-to-speech synthesis. SSR-Speech is built on a Transformer decoder and incorporates classifier-free guidance to enhance the stability of the g…

Cited by 0 Source ↗Scholar ↗

2024

An Experimental Comparison of Noise-Robust Text-To-Speech Synthesis Systems Based On Self-Supervised Representation

Xiao-Ying Zhao, Qiushi Zhu, Yuchen Hu

ICASSP 2024accepted

With the advance in deep learning, text-to-speech (TTS) using clean speech has witnessed significant performance improvements. As the data collected in real scenes often contain noise and thus needs to be denoised, TTS models trained on the enhanced speech suffer from distortions and residual noises…

Cited by 0 Source ↗Scholar ↗

2024

Cross-Modality and Within-Modality Regularization for Audio-Visual Deepfake Detection

Heqing Zou, Meng Shen, Yuchen Hu, Chen Chen, Eng Siong Chng, Deepu Rajan

ICASSP 2024accepted

Audio-visual deepfake detection scrutinizes manipulations in public video using complementary multimodal cues. Current methods, which train on fused multimodal data for multimodal targets face challenges due to uncertainties and inconsistencies in learned representations caused by independent modali…

Cited by 0 Source ↗Scholar ↗

2024

GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen +1

ACL 2024long

Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by its reduced representation errors and incorporated external knowledge. However, both translation tasks typically utilize beam search decoding and top-1 hypothesis se…

Cited by 17 Source ↗PDF ↗code ↗Scholar ↗

2024

It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

CHEN CHEN, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, EngSiong Chng +1

ICLR 2024poster

Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. Specifically, an LLM is utilized to carry out a direct mapping from the N-best hypotheses list generated by…

Cited by 27 Source ↗PDF ↗Scholar ↗

2024

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

Yuchen Hu, CHEN CHEN, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen +1

ICLR 2024spotlight

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recognition results. The latest work proposes a GER benchmark with "…

Cited by 29 Source ↗PDF ↗code ↗Scholar ↗

2024

Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models

Yuchen Hu, Chen Chen, Chengwei Qin, Qiushi Zhu, EngSiong Chng, Ruizhe Li

ACL 2024findings

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which aims to predict the ground-truth transcription from the decoded N-best hypotheses. Thanks to the strong language generation ability of LLMs and rich informati…

Cited by 6 Source ↗PDF ↗code ↗Scholar ↗

2024

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Qiushi Zhu, Jie Zhang, Yu Gu, Yuchen Hu, Lirong Dai

AAAI 2024technical

Self-supervised speech pre-training methods have developed rapidly in recent years, which show to be very effective for many near-field single-channel speech tasks. However, far-field multichannel speech processing is suffering from the scarcity of labeled multichannel data and complex ambient noise…

Cited by 13 Source ↗PDF ↗code ↗Scholar ↗

2024

Noise-Aware Speech Separation with Contrastive Learning

Zizheng Zhang, Chen Chen, Hsin-Hung Chen, Xiang Liu, Yuchen Hu, Eng Siong Chng

ICASSP 2024accepted

Recently, speech separation (SS) task has achieved remarkable progress driven by deep learning technique. However, it is still challenging to separate target speech from noisy mixture, as the neural model is vulnerable to assign background noise to each speaker. In this paper, we propose a noise-awa…

Cited by 0 Source ↗Scholar ↗

2024

Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System

Chen Chen, Ruizhe Li, Yuchen Hu, Yuanyuan Chen, Chengwei Qin, Qiang Zhang

ACL 2024findings

Intelligent task-oriented dialogue systems (ToDs) are expected to continuously acquire new knowledge, also known as Continual Learning (CL), which is crucial to fit ever-changing user needs. However, catastrophic forgetting dramatically degrades the model performance in face of a long streamed curri…

Cited by 0 Source ↗PDF ↗Scholar ↗

2024

Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Chengwei Qin, Pin-Yu Chen, EngSiong Chng +1

NeurIPS 2024poster

We propose an unsupervised adaptation framework, Self-TAught Recognizer (STAR), which leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains, such as noise and accents. STAR is developed for prevalent speech foundation models based…

Cited by 4 Source ↗PDF ↗code ↗Scholar ↗

2023

Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition

Yuchen Hu, Ruizhe Li, Chen Chen, Heqing Zou, Qiushi Zhu, Eng Siong Chng

IJCAI 2023poster

Audio-visual speech recognition (AVSR) research has gained a great success recently by improving the noise-robustness of audio-only automatic speech recognition (ASR) with noise-invariant visual information. However, most existing AVSR approaches simply fuse the audio and visual features by concaten…

Cited by 12 Source ↗PDF ↗code ↗Scholar ↗

2023

Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Qiushi Zhu, Eng Siong Chng

ICASSP 2023accepted

Speech enhancement (SE) is proved effective in reducing noise from noisy speech signals for downstream automatic speech recognition (ASR), where multi-task learning strategy is employed to jointly optimize these two tasks. However, the enhanced speech learned by SE objective may not always yield goo…

Cited by 0 Source ↗Scholar ↗

2023

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

Yuchen Hu, Ruizhe Li, Chen Chen, Chengwei Qin, Qiu-Shi Zhu, Eng Siong Chng

ACL 2023long

Audio-visual speech recognition (AVSR) provides a promising solution to ameliorate the noise-robustness of audio-only speech recognition with visual information. However, most existing efforts still focus on audio modality to improve robustness considering its dominance in AVSR task, with noise adap…

Cited by 11 Source ↗PDF ↗code ↗Scholar ↗

2023

HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

CHEN CHEN, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, EngSiong Chng

NeurIPS 2023poster

Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly available clean speech datasets. However, even state-of-the-art ASR systems experience performance degradation when confronted with adverse conditions, as a well-tr…

Cited by 59 Source ↗PDF ↗code ↗Scholar ↗

2023

Leveraging Modality-Specific Representations for Audio-Visual Speech Recognition via Reinforcement Learning

Chen Chen, Yuchen Hu, Qiang Zhang, Heqing Zou, Beier Zhu, Eng Siong Chng

AAAI 2023technical

Audio-visual speech recognition (AVSR) has gained remarkable success for ameliorating the noise-robustness of speech recognition. Mainstream methods focus on fusing audio and visual inputs to obtain modality-invariant representations. However, such representations are prone to over-reliance on audio…

Cited by 31 Source ↗PDF ↗Scholar ↗

2023

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Heqing Zou, Eng Siong Chng

ACL 2023long

Audio-visual speech recognition (AVSR) attracts a surge of research interest recently by leveraging multimodal signals to understand human speech. Mainstream approaches addressing this task have developed sophisticated architectures and techniques for multi-modality fusion and representation learnin…

Cited by 13 Source ↗PDF ↗code ↗Scholar ↗

2023

Metric-Oriented Speech Enhancement Using Diffusion Probabilistic Model

Chen Chen, Yuchen Hu, Weiwei Weng, Eng Siong Chng

ICASSP 2023accepted

Deep neural network based speech enhancement technique focuses on learning a noisy-to-clean transformation supervised by paired training data. However, the task-specific evaluation metric (e.g., PESQ) is usually non-differentiable and can not be directly constructed in the training criteria. This mi…

Cited by 0 Source ↗Scholar ↗

2023

UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning

Heqing Zou, Meng Shen, Chen Chen, Yuchen Hu, Deepu Rajan, Eng Siong Chng

ACL 2023findings

Multimodal learning aims to imitate human beings to acquire complementary information from multiple modalities for various downstream tasks. However, traditional aggregation-based multimodal fusion methods ignore the inter-modality relationship, treat each modality equally, suffer sensor noise, and…

Cited by 22 Source ↗PDF ↗code ↗Scholar ↗

2023

Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separation

Yuchen Hu, Chen Chen, Heqing Zou, Xionghu Zhong, Eng Siong Chng

ICASSP 2023accepted

Recent studies in neural network-based monaural speech separation (SS) have achieved a remarkable success thanks to increasing ability of long sequence modeling. However, they would degrade significantly when put under realistic noisy conditions, as the background noise could be mistaken for speaker…

Cited by 0 Source ↗Scholar ↗

2023

Unsupervised Noise Adaptation Using Data Simulation

Chen Chen, Yuchen Hu, Heqing Zou, Linhui Sun, Eng Siong Chng

ICASSP 2023accepted

Deep neural network based speech enhancement approaches aim to learn a noisy-to-clean transformation using a supervised learning paradigm. However, such a trained-well transformation is vulnerable to unseen noises that are not included in training set. In this work, we focus on the unsupervised nois…

Cited by 0 Source ↗Scholar ↗

2022

Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognition

Yuchen Hu, Nana Hou, Chen Chen, Eng Siong Chng

ICASSP 2022accepted

Speech enhancement (SE) aims to suppress the additive noise from noisy speech signals to improve the speech’s perceptual quality and intelligibility. However, the over-suppression phenomenon in the enhanced speech might degrade the performance of downstream automatic speech recognition (ASR) task du…

Cited by 0 Source ↗Scholar ↗

2022

Noise-Robust Speech Recognition With 10 Minutes Unparalleled In-Domain Data

Chen Chen, Nana Hou, Yuchen Hu, Shashank Shirol, Eng Siong Chng

ICASSP 2022accepted

Noise-robust speech recognition systems require large amounts of training data including noisy speech data and corresponding transcripts to achieve state-of-the-art performances in face of various practical environments. However, such plenty of in-domain data is not always available in the real-life…

Cited by 0 Source ↗Scholar ↗

2022

Self-Critical Sequence Training for Automatic Speech Recognition

Chen Chen, Yuchen Hu, Nana Hou, Xiaofeng Qi, Heqing Zou, Eng Siong Chng

ICASSP 2022accepted

Although automatic speech recognition (ASR) task has gained remarkable success by sequence-to-sequence models, there are two main mismatches between its training and testing that might lead to performance degradation: 1) The typically used cross-entropy criterion aims to maximize log-likelihood of t…

Cited by 0 Source ↗Scholar ↗