AIConfPaper
← Search

Hongyu Gong

19 accepted papers

2025

AV-Flow: Transforming Text to Audio-Visual Human-like Interactions

Aggelina Chatziagapi, Louis-Philippe Morency, Hongyu Gong, Michael Zollhöfer, Dimitris Samaras, Alexander Richard
ICCV 2025poster

We introduce AV-Flow, an audio-visual generative model that animates photo-realistic 4D talking avatars given only text input. In contrast to prior work that assumes an existing speech signal, we synthesize speech and vision jointly. We demonstrate human-like speech synthesis, synchronized lip motio…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents

Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, Shyamnath Gollakota
EMNLP 2024main

Despite broad interest in modeling spoken dialogue agents, most approaches are inherently “half-duplex” – restricted to turn-based interaction with responses requiring explicit prompting by the user or implicit tracking of interruption or silence events. Human dialogue, by contrast, is “full-duplex”…

Cited by 10Source ↗PDF ↗Scholar ↗
2024

CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models

Jianing Zhou, Ziheng Zeng, Hongyu Gong, Suma Bhat
ACL 2024findings

Recent advancements in joint speech-text pre-training have significantly advanced the processing of natural language. However, a key limitation is their reliance on parallel speech-text data, posing challenges due to data accessibility. Addressing this, our paper introduces an innovative framework f…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation

Min-Jae Hwang, Ilia Kulikov, Benjamin Peloquin, Hongyu Gong, Peng-Jen Chen, Ann Lee
ACL 2024findings

In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST).Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S)…

Cited by 1Source ↗PDF ↗code ↗Scholar ↗
2023

A Holistic Cascade System, Benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation

Wen-Chin Huang, Benjamin N. Peloquin, Justine Kao, Changhan Wang, Hongyu Gong, Elizabeth Salesky +3
ICASSP 2023accepted

Expressive speech-to-speech translation (S2ST) aims to transfer prosodic attributes of source speech to target speech while maintaining translation accuracy. Existing research in expressive S2ST is limited, typically focusing on a single expressivity aspect at a time. Likewise, this research area la…

Cited by 0Source ↗Scholar ↗
2023

Improving Speech-to-Speech Translation Through Unlabeled Text

Xuan-Phi Nguyen, Sravya Popuri, Changhan Wang, Yun Tang, Ilia Kulikov, Hongyu Gong
ICASSP 2023accepted

Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data. While effort has been made to increase the data size from unlabeled speech by cascading pretrained speech recognition (ASR), machine translation…

Cited by 0Source ↗Scholar ↗
2023

Named Entity Detection and Injection for Direct Speech Translation

Marco Gaido, Yun Tang, Ilia Kulikov, Rongqing Huang, Hongyu Gong, Hirofumi Inaguma
ICASSP 2023accepted

In a sentence, certain words are critical for its semantic. Among them, named entities (NEs) are notoriously challenging for neural models. Despite their importance, their accurate handling has been neglected in speech-to-text (S2T) translation research, and recent work has shown that S2T models per…

Cited by 0Source ↗Scholar ↗
2023

Non-compositional Expression Generation Based on Curriculum Learning and Continual Learning

Jianing Zhou, Ziheng Zeng, Hongyu Gong, Suma Bhat
EMNLP 2023long findings

Non-compositional expressions, by virtue of their non-compositionality, are a classic `pain in the neck' for NLP systems. Different from the general language modeling and generation tasks that are primarily compositional, generating non-compositional expressions is more challenging for current neura…

Cited by 0Source ↗code ↗Scholar ↗
2023

Pre-training for Speech Translation: CTC Meets Optimal Transport

Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, Didier Schwab
ICML 2023oral

The gap between speech and text modalities is a major challenge in speech-to-text translation (ST). Different methods have been proposed to reduce this gap, but most of them require architectural changes in ST training. In this work, we propose to mitigate this issue at the pre-training stage, requi…

Cited by 27Source ↗PDF ↗code ↗Scholar ↗
2023

Speech-to-Speech Translation for a Real-world Unwritten Language

Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung +10
ACL 2023findings

We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training d…

Cited by 22Source ↗PDF ↗code ↗Scholar ↗
2023

SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswami +4
ACL 2023long

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we…

Cited by 34Source ↗PDF ↗Scholar ↗
2022

Contrastive Clustering to Mine Pseudo Parallel Data for Unsupervised Translation

Xuan-Phi Nguyen, Hongyu Gong, Yun Tang, Changhan Wang, Philipp Koehn, Shafiq Joty
ICLR 2022poster

Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means, such as iterative back-translation. However, there may exist small amount of actual parallel…

Cited by 6Source ↗PDF ↗Scholar ↗
2022

Idiomatic Expression Paraphrasing without Strong Supervision

Jianing Zhou, Ziheng Zeng, Hongyu Gong, Suma Bhat
AAAI 2022technical

Idiomatic expressions (IEs) play an essential role in natural language. In this paper, we study the task of idiomatic sentence paraphrasing (ISP), which aims to paraphrase a sentence with an IE by replacing the IE with its literal paraphrase. The lack of large-scale corpora with idiomatic-literal pa…

Cited by 10Source ↗PDF ↗code ↗Scholar ↗
2022

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger Schwenk
EMNLP 2022main

We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare different approaches to decode these multimodal and multilingual fixed-size represe…

Cited by 21Source ↗PDF ↗Scholar ↗
2022

Textless Speech-to-Speech Translation on Real Data

Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang +5
NAACL 2022long

We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and t…

Cited by 157Source ↗PDF ↗Scholar ↗
2022

Unified Speech-Text Pre-training for Speech Translation and Recognition

Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu +5
ACL 2022long

In this work, we describe a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. The proposed method utilizes multi-task learning to integrate four self-supervised and supervised subtasks for cross modality learning. A self-supe…

Cited by 92Source ↗PDF ↗code ↗Scholar ↗
2021

Multimodal and Multilingual Embeddings for Large-Scale Speech Mining

Paul-Ambroise Duquenne, Hongyu Gong, Holger Schwenk
NeurIPS 2021spotlight

We present an approach to encode a speech signal into a fixed-size representation which minimizes the cosine loss with the existing massively multilingual LASER text embedding space. Sentences are close in this embedding space, independently of their language and modality, either text or audio. Usin…

Cited by 40Source ↗PDF ↗code ↗Scholar ↗
2021

Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling

Hongyu Gong, Yun Tang, Juan Pino, Xian Li
NeurIPS 2021poster

Multi-head attention has each of the attention heads collect salient information from different parts of an input sequence, making it a powerful mechanism for sequence modeling. Multilingual and multi-domain learning are common scenarios for sequence modeling, where the key challenge is to maximize…

Cited by 13Source ↗PDF ↗Scholar ↗