AIConfPaper
← Search

Wei-Ning Hsu

39 accepted papers

2026

MR-FLOWDPO: MULTI-REWARD DIRECT PREFERENCE OPTIMIZATION FOR FLOW-MATCHING TEXT-TO-MUSIC GENERATION

Alon Ziv, The Hebrew University of Jerusalem, Sanyuan Chen, Yossi Adi, The Hebrew University of Jerusalem, Wei-Ning Hsu
ICASSP 2026oral

A key challenge in music generation models is their lack of direct alignment with human preferences, as music evaluation is inherently subjective and varies widely across individuals. We introduce MR-FlowDPO, a novel approach that enhances flow-matching-based music generation models - a major class…

Cited by 0Source ↗PDF ↗Scholar ↗
2026

Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning

Apoorv Vyas, Heng-Jui Chang, Cheng-Fu Yang, Po-Yao Huang, Luya Gao, Julius Richter +6
CVPR 2026

We introduce Perception Encoder-Audiovisual, PE-AV, a new family of encoders for audio and video understanding trained with scaled contrastive learning. Building on PE, PE-AV makes several key contributions to extend representations to audio, and natively support joint embeddings across audio-video,

Cited by 0Source ↗code ↗Scholar ↗
2025

FlowDec: A flow-based full-band general audio codec with high perceptual quality

Simon Welker, Matthew Le, Ricky T. Q. Chen, Wei-Ning Hsu, Timo Gerkmann, Alexander Richard +1
ICLR 2025poster

We propose FlowDec, a neural full-band audio codec for general audio sampled at 48 kHz that combines non-adversarial codec training with a stochastic postfilter based on a novel conditional flow matching method. Compared to the prior work ScoreDec which is based on score matching, we generalize from…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos

Changan Chen*, Puyuan Peng, Ami Baid, Zihui Xue, Wei-Ning Hsu, David Harwath +1
ECCV 2024oral

"Generating realistic audio for human actions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak…

Cited by 6Source ↗PDF ↗code ↗Scholar ↗
2024

Generative Pre-training for Speech with Flow Matching

Alexander H. Liu, Matthew Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, Wei-Ning Hsu
ICLR 2024poster

Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data. In speech, text-to-speech synthesis and neural vocoder are good examples where generative mode…

Cited by 34Source ↗PDF ↗Scholar ↗
2024

M2BART: Multilingual and Multimodal Encoder-Decoder Pre-Training for Any-to-Any Machine Translation

Peng-Jen Chen, Bowen Shi, Kelvin Niu, Ann Lee, Wei-Ning Hsu
ICASSP 2024accepted

Speech and language models are advancing towards universality. A single model can now handle translations across 200 languages and transcriptions for over 100 languages. Universal models simplify development, deployment, and importantly, transfer knowledge to less-resourced languages or modes. This…

Cited by 0Source ↗Scholar ↗
2024

MusicFlow: Cascaded Flow Matching for Text Guided Music Generation

K R Prajwal, Bowen Shi, Matthew Le, Apoorv Vyas, Andros Tjandra, Mahi Luthra +5
ICML 2024poster

We introduce MusicFlow, a cascaded text-to-music generation model based on flow matching. Based on self-supervised representations to bridge between text descriptions and music audios, we construct two flow matching networks to model the conditional distribution of semantic and acoustic features. Ad…

Cited by 9Source ↗PDF ↗Scholar ↗
2024

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

HyoJung Han, Mohamed Anwar, Juan Pino, Wei-Ning Hsu, Marine Carpuat, Bowen Shi +1
ACL 2024long

Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. Augmenting these systems with visual signals has the potential to improve robustness to noise. However, audio-visual (AV) data is only available in limited amounts and for fewer l…

Cited by 6Source ↗PDF ↗Scholar ↗
2023

Cocktail Hubert: Generalized Self-Supervised Pre-Training for Mixture and Single-Source Speech

Maryam Fazel-Zarandi, Wei-Ning Hsu
ICASSP 2023accepted

Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic/linguistic domains, languages, and modalities, these investigations are limited to single-source…

Cited by 0Source ↗Scholar ↗
2023

Continual Learning for On-Device Speech Recognition Using Disentangled Conformers

Anuj Diwan, Ching-Feng Yeh, Wei-Ning Hsu, Paden Tomasello, Eunsol Choi, David Harwath +1
ICASSP 2023accepted

Automatic speech recognition research focuses on training and evaluating on static datasets. Yet, as speech models are increasingly deployed on personal devices, such models encounter user-specific distributional shifts. To simulate this real-world scenario, we introduce LibriContinual, a continual…

Cited by 0Source ↗Scholar ↗
2023

DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning

Alexander H. Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, James R. Glass
NeurIPS 2023poster

In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering. We show that these concepts complement each other and result in a strong representation l…

Cited by 23Source ↗PDF ↗code ↗Scholar ↗
2023

Do Coarser Units Benefit Cluster Prediction-Based Speech Pre-Training?

Ali Elkahky, Wei-Ning Hsu, Paden Tomasello, Tu Anh Nguyen, Robin Algayres, Yossi Adi +3
ICASSP 2023accepted

The research community has produced many successful self-supervised speech representation learning methods over the past few years. Discrete units have been utilized in various self-supervised learning frameworks, such as VQ-VAE [1], wav2vec 2.0 [2], Hu-BERT [3], and Wav2Seq [4]. This paper studies…

Cited by 0Source ↗Scholar ↗
2023

Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language

Alexei Baevski, Arun Babu, Wei-Ning Hsu, Michael Auli
ICML 2023oral

Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use…

Cited by 120Source ↗PDF ↗code ↗Scholar ↗
2023

ReVISE: Self-Supervised Speech Resynthesis With Visual Input for Universal and Generalized Speech Regeneration

Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, Yossi Adi
CVPR 2023poster

Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes to unify these subjects and study Generalized Speech Regeneration, where the goal…

Cited by 21Source ↗PDF ↗code ↗Scholar ↗
2023

Scaling Laws for Generative Mixed-Modal Language Models

Armen Aghajanyan, LILI YU, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang +4
ICML 2023poster

Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens from VQ-VAEs, speech tokens from HuBERT, BPE tokens for language or code, and so on). To better understand the scaling pro…

Cited by 104Source ↗PDF ↗Scholar ↗
2023

Simple and Effective Unsupervised Speech Translation

Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu +2
ACL 2023long

The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled data covering two different languages. To address this issue, we study a simple and effective approach to build speech tr…

Cited by 15Source ↗PDF ↗code ↗Scholar ↗
2023

Speech-to-Speech Translation for a Real-world Unwritten Language

Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung +10
ACL 2023findings

We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training d…

Cited by 22Source ↗PDF ↗code ↗Scholar ↗
2023

Toward Joint Language Modeling for Speech Units and Text

Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau +2
EMNLP 2023long findings

Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language mode…

Cited by 0Source ↗Scholar ↗
2023

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz +5
NeurIPS 2023poster

Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and…

Cited by 299Source ↗PDF ↗Scholar ↗
2022

Direct Speech-to-Speech Translation With Discrete Units

Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma +6
ACL 2022long

We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then traini…

Cited by 182Source ↗PDF ↗code ↗Scholar ↗
2022

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed
ICLR 2022poster

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker’s lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framewor…

Cited by 368Source ↗PDF ↗code ↗Scholar ↗
2022

Text-Free Prosody-Aware Generative Spoken Language Modeling

Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia +5
ACL 2022long

Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) (CITATION) is the only prior work addressing t…

Cited by 136Source ↗PDF ↗code ↗Scholar ↗
2022

Textless Speech Emotion Conversion using Discrete & Decomposed Representations

Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière +4
EMNLP 2022main

Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into d…

Cited by 70Source ↗PDF ↗code ↗Scholar ↗
2022

Textless Speech-to-Speech Translation on Real Data

Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang +5
NAACL 2022long

We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and t…

Cited by 157Source ↗PDF ↗Scholar ↗
2022

Unified Speech-Text Pre-training for Speech Translation and Recognition

Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu +5
ACL 2022long

In this work, we describe a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. The proposed method utilizes multi-task learning to integrate four self-supervised and supervised subtasks for cross modality learning. A self-supe…

Cited by 92Source ↗PDF ↗code ↗Scholar ↗
2022

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli
ICML 2022oral

While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same…

Cited by 1024Source ↗PDF ↗code ↗Scholar ↗
2022

textless-lib: a Library for Textless Spoken Language Processing

Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee +5
NAACL 2022system demonstrations

Textless spoken language processing is an exciting area of research that promises to extend applicability of the standard NLP toolset onto spoken language and languages with few or no textual resources. Here, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in the area…

Cited by 23Source ↗PDF ↗code ↗Scholar ↗
2022

u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality

Wei-Ning Hsu, Bowen Shi
NeurIPS 2022accept

While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality. In this paper, we present u-HuBERT, a self-s…

Cited by 45Source ↗PDF ↗code ↗Scholar ↗
2021

Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?

Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, Abdelrahman Mohamed
ICASSP 2021accepted

Compared to vision and language applications, self-supervised pre-training approaches for ASR are challenged by three unique problems: (1) There are multiple sound units in each input utterance, (2) With audio-only pre-training, there is no lexicon of sound units, and (3) Sound units have variable l…

Cited by 0Source ↗Scholar ↗
2021

Text-Free Image-to-Speech Synthesis Using Learned Segmental Units

Wei-Ning Hsu, David Harwath, Tyler Miller, Christopher Song, James Glass
ACL 2021long

In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. Instead, we connect the image captioning module and the speech synthesi…

Cited by 87Source ↗PDF ↗code ↗Scholar ↗
2021

fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit

Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen +2
EMNLP 2021system demonstrations

This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis. We implement a number of autoregressive (AR) and non-AR text-to-speech models, and their multi-speaker variants. To enable training speech synthesis models with less curated data, a number of preprocessing tools are built and…

Cited by 34Source ↗PDF ↗code ↗Scholar ↗
2019

Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization

Wei-Ning Hsu, Yu Zhang, Ron J. Weiss, Yu-An Chung, Yuxuan Wang, Yonghui Wu +1
ICASSP 2019accepted

To leverage crowd-sourced data to train multi-speaker text-to-speech (TTS) models that can synthesize clean speech for all speakers, it is essential to learn disentangled representations which can independently control the speaker identity and background noise in generated signals. However, learning…

Cited by 0Source ↗Scholar ↗
2019

Hierarchical Generative Modeling for Controllable Speech Synthesis

Wei-Ning Hsu, Yu Zhang, Ron J. Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang +6
ICLR 2019poster

This paper proposes a neural end-to-end text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generati…

Cited by 297Source ↗PDF ↗Scholar ↗
2019

Semi-supervised Training for Improving Data Efficiency in End-to-end Speech Synthesis

Yu-An Chung, Yuxuan Wang, Wei-Ning Hsu, Yu Zhang, R. J. Skerry-Ryan
ICASSP 2019accepted

Although end-to-end text-to-speech (TTS) models such as Tacotron have shown excellent results, they typically require a sizable set of high-quality <;text, audio> pairs for training, which are expensive to collect. In this paper, we propose a semi-supervised training framework to improve the data ef…

Cited by 0Source ↗Scholar ↗
2018

Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition

Wei-Ning Hsu, James R. Glass
ICASSP 2018accepted

The performance of automatic speech recognition (ASR) systems can be significantly compromised by previously unseen conditions, which is typically due to a mismatch between training and testing distributions. In this paper, we address robustness by studying domain invariant features, such that domai…

Cited by 0Source ↗Scholar ↗
2017

Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data

Wei-Ning Hsu, Yu Zhang, James Glass
NeurIPS 2017poster

We present a factorized hierarchical variational autoencoder, which learns disentangled and interpretable representations from sequential data without supervision. Specifically, we exploit the multi-scale nature of information in sequential data by formulating it explicitly within a factorized hiera…

Cited by 442Source ↗PDF ↗code ↗Scholar ↗
2015

Enhancing automatically discovered multi-level acoustic patterns considering context consistency with applications in spoken term detection

Cheng-Tao Chung, Wei-Ning Hsu, Cheng-Yi Lee, Lin-Shan Lee
ICASSP 2015accepted

This paper presents a novel approach for enhancing the multiple sets of acoustic patterns automatically discovered from a given corpus. In a previous work it was proposed that different HMM configurations (number of states per model, number of distinct models) for the acoustic patterns form a two-di…

Cited by 0Source ↗Scholar ↗