AIConfPaper
← Search

Karen Livescu

38 accepted papers

2026

CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition

Martijn Bartelds, Ananjan Nandi, Moussa Koulako Bala Doumbouya, Dan Jurafsky, Tatsunori Hashimoto, Karen Livescu
ICLR 2026poster

Modern deep learning models often achieve high overall performance, but consistently fail on specific subgroups. Group distributionally robust optimization (group DRO) addresses this problem by minimizing the worst-group loss, but it fails when group losses misrepresent performance differences betwe…

Cited by 0Source ↗code ↗Scholar ↗
2026

MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

Chung-Ming Chien, Manu Orsini, Eugene Kharitonov, Neil Zeghidour, Karen Livescu, Alexandre Défossez
ICML 2026poster

Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an …

Cited by 0Source ↗Scholar ↗
2026

On the Predictive Power of Representation Dispersion in Language Models

Yanhong Li, Ming Li, Karen Livescu, Jiawei Zhou
ICLR 2026poster

We show that a language model’s ability to predict text is tightly linked to the breadth of its embedding space: models that spread their contextual representations more widely tend to achieve lower perplexity. Concretely, we find that representation dispersion—the average pairwise cosine distance a…

Cited by 0Source ↗code ↗Scholar ↗
2025

Constructing Datasets From Public Police Body Camera Footage

Jamie Rosas-Smith, Martijn Bartelds, Ruizhe Huang, Leibny Paola García-Perera, Karen Livescu, Dan Jurafsky +1
ICASSP 2025accepted

The enormous potential of body-worn cameras to improve accountability in policing remains largely unrealized due to large volumes of unreviewed footage. Transcription and diarization tools could aid in reviewing footage, but lack of public data hinders their development. We develop a pipeline to con…

Cited by 0Source ↗Scholar ↗
2025

Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks

Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin +70
ICLR 2025poster

Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication…

Cited by 10Source ↗PDF ↗code ↗Scholar ↗
2025

SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction

Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu, Alexander H. Liu
ACL 2025long

Sign language processing has traditionally relied on task-specific models, limiting the potential for transfer learning across tasks. Pre-training methods for sign language have typically focused on either supervised pre-training, which cannot take advantage of unlabeled data, or context-independent…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale

Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu
ACL 2025finding

A persistent challenge in sign language video processing, including the task of sign language to written language translation, is how we train efficient model given the nature of videos. Informed by the nature and linguistics of signed languages, our proposed method focuses on just the most relevant…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

AV2WAV: Diffusion-Based Re-Synthesis from Continuous Self-Supervised Features for Audio-Visual Speech Enhancement

Ju-Chieh Chou, Chung-Ming Chien, Karen Livescu
ICASSP 2024accepted

Speech enhancement systems are typically trained using pairs of clean and noisy speech. In audio-visual speech enhancement (AVSE), there is not as much ground-truth clean data available; most audio-visual datasets are collected in real-world environments with background noise and reverberation, hamp…

Cited by 0Source ↗Scholar ↗
2024

Generative Context-Aware Fine-Tuning of Self-Supervised Speech Models

Suwon Shon, Kwangyoun Kim, Prashant Sridhar, Yi-Te Hsu, Shinji Watanabe, Karen Livescu
ICASSP 2024accepted

When performing tasks like automatic speech recognition or spoken language understanding for a given utterance, access to preceding text or audio provides contextual information that can improve performance. Considering the recent advances in generative large language models (LLM), we hypothesize th…

Cited by 0Source ↗Scholar ↗
2024

On the Evaluation of Speech Foundation Models for Spoken Language Understanding

Siddhant Arora, Ankita Pasad, Chung-Ming Chien, Jionghao Han, Roshan Sharma, Jee-weon Jung +6
ACL 2024findings

The Spoken Language Understanding Evaluation (SLUE) suite of benchmark tasks was recently introduced to address the need for openresources and benchmarking of complex spoken language understanding (SLU) tasks, including both classification and sequence generation tasks, on natural speech. The benchm…

Cited by 6Source ↗PDF ↗Scholar ↗
2024

Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing

Freda Shi, Kevin Gimpel, Karen Livescu
ACL 2024long

We present the structured average intersection-over-union ratio (STRUCT-IOU), an evaluation metric that compares a constituency parse tree over automatically recognized spoken word boundaries with the ground-truth parse tree over written words. To compute the metric, we (1) project the ground-truth…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

Towards Robust Speech Representation Learning for Thousands of Languages

William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi +4
EMNLP 2024main

Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. However, models are still far from supporting the world’s 7000+ languages. We propose XEUS, a Cross-lingual Encoder for Universal Speech, trained on over 1 million hours of d…

Cited by 6Source ↗PDF ↗code ↗Scholar ↗
2024

UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions

Siddhant Arora, Hayato Futami, Jee-weon Jung, Yifan Peng, Roshan Sharma, Yosuke Kashiwagi +3
NAACL 2024long

Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model’s behavior and surpassing performance of task-specific models. Motivated by this, we ask: can we build a single model that jointly performs various spoken language underst…

Cited by 8Source ↗PDF ↗code ↗Scholar ↗
2023

Context-Aware Fine-Tuning of Self-Supervised Speech Models

Suwon Shon, Felix Wu, Kwangyoun Kim, Prashant Sridhar, Karen Livescu, Shinji Watanabe
ICASSP 2023accepted

Self-supervised pre-trained transformers have improved the state of the art on a variety of speech tasks. Due to the quadratic time and space complexity of self-attention, they usually operate at the level of relatively short (e.g., utterance) segments. In this paper, we study the use of context, i.…

Cited by 0Source ↗Scholar ↗
2023

SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks

Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan Sharma +4
ACL 2023long

Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In this work, we introduce several new annotated SLU benchmark tasks based on freely availa…

Cited by 34Source ↗PDF ↗code ↗Scholar ↗
2023

Toward Joint Language Modeling for Speech Units and Text

Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau +2
EMNLP 2023long findings

Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language mode…

Cited by 0Source ↗Scholar ↗
2022

Chess as a Testbed for Language Model State Tracking

Shubham Toshniwal, Sam Wiseman, Karen Livescu, Kevin Gimpel
AAAI 2022technical

Transformer language models have made tremendous strides in natural language understanding tasks. However, the complexity of natural language makes it challenging to ascertain how accurately these models are tracking the world state underlying the text. Motivated by this issue, we consider the task…

Cited by 52Source ↗PDF ↗code ↗Scholar ↗
2022

On the Use of External Data for Spoken Named Entity Recognition

Ankita Pasad, Felix Wu, Suwon Shon, Karen Livescu, Kyu Han
NAACL 2022long

Spoken language understanding (SLU) tasks involve mapping from speech signals to semantic labels. Given the complexity of such tasks, good performance is expected to require large labeled datasets, which are difficult to collect for each new task and domain. However, recent advances in self-supervis…

Cited by 22Source ↗PDF ↗code ↗Scholar ↗
2022

Open-Domain Sign Language Translation Learned from Online Video

Bowen Shi, Diane Brentari, Gregory Shakhnarovich, Karen Livescu
EMNLP 2022main

Existing work on sign language translation – that is, translation from sign language videos into sentences in a written language – has focused mainly on (1) data collected in a controlled environment or (2) data in a specific domain, which limits the applicability to real-world settings. In this pap…

Cited by 61Source ↗PDF ↗code ↗Scholar ↗
2022

SLUE: New Benchmark Tasks For Spoken Language Understanding Evaluation on Natural Speech

Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu +1
ICASSP 2022accepted

Progress in speech processing has been facilitated by shared datasets and benchmarks. Historically these have focused on automatic speech recognition (ASR), speaker identification, or other lower-level tasks. Interest has been growing in higher-level spoken language understanding tasks, including us…

Cited by 0Source ↗Scholar ↗
2022

Searching for fingerspelled content in American Sign Language

Bowen Shi, Diane Brentari, Greg Shakhnarovich, Karen Livescu
ACL 2022long

Natural language processing for sign language video—including tasks like recognition, translation, and search—is crucial for making artificial intelligence technologies accessible to deaf individuals, and is gaining research interest in recent years. In this paper, we address the problem of searchin…

Cited by 6Source ↗PDF ↗Scholar ↗
2022

Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing

Freda Shi, Kevin Gimpel, Karen Livescu
ACL 2022long

We present substructure distribution projection (SubDP), a technique that projects a distribution over structures in one domain to another, by projecting substructure distributions separately. Models for the target domain can then be trained, using the projected distributions as soft silver labels.…

Cited by 11Source ↗PDF ↗Scholar ↗
2020

Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction

Weiran Wang, Qingming Tang, Karen Livescu
ICASSP 2020accepted

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amou…

Cited by 0Source ↗Scholar ↗
2019

Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition

Shane Settle, Kartik Audhkhasi, Karen Livescu, Michael Picheny
ICASSP 2019accepted

Direct acoustics-to-word (A2W) systems for end-to-end automatic speech recognition are simpler to train, and more efficient to decode with, than sub-word systems. However, A2W systems can have difficulties at training time when data is limited, and at decoding time when recognizing words outside the…

Cited by 0Source ↗Scholar ↗
2019

Fingerspelling Recognition in the Wild With Iterative Visual Attention

Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Diane Brentari, Greg Shakhnarovich, Karen Livescu
ICCV 2019poster

Sign language recognition is a challenging gesture sequence recognition problem, characterized by quick and highly coarticulated motion. In this paper we focus on recognition of fingerspelling sequences in American Sign Language (ASL) videos collected in the wild, mainly from YouTube and Deaf social…

Cited by 90PDF ↗code ↗Scholar ↗
2019

Semantic Query-by-example Speech Search Using Visual Grounding

Herman Kamper, Aristotelis Anastassiou, Karen Livescu
ICASSP 2019accepted

A number of recent studies have started to investigate how speech systems can be trained on untranscribed speech by leveraging accompanying images at training time. Examples of tasks include keyword prediction and within- and across-mode retrieval. Here we consider how such models can be used for qu…

Cited by 0Source ↗Scholar ↗
2018

A Study of All-Convolutional Encoders for Connectionist Temporal Classification

Kalpesh Krishna, Liang Lu, Kevin Gimpel, Karen Livescu
ICASSP 2018accepted

Connectionist temporal classification (CTC) is a popular sequence prediction approach for automatic speech recognition that is typically used with models based on recurrent neural networks (RNNs). We explore whether deep convolutional neural networks (CNNs) can be used effectively instead of RNNs as…

Cited by 0Source ↗Scholar ↗
2016

Deep convolutional acoustic word embeddings using word-pair side information

Herman Kamper, Weiran Wang, Karen Livescu
ICASSP 2016accepted

Recent studies have been revisiting whole words as the basic modelling unit in speech recognition and query applications, instead of phonetic units. Such whole-word segmental systems rely on a function that maps a variable-length speech segment to a vector in a fixed-dimensional space; the resulting…

Cited by 0Source ↗Scholar ↗
2016

Signer-independent fingerspelling recognition with deep neural network adaptation

Taehwan Kim, Weiran Wang, Hao Tang, Karen Livescu
ICASSP 2016accepted

We study the problem of recognition of fingerspelled letter sequences in American Sign Language in a signer-independent setting. Fingerspelled sequences are both challenging and important to recognize, as they are used for many content words such as proper nouns and technical terms. Previous work ha…

Cited by 0Source ↗Scholar ↗
2015

Unsupervised learning of acoustic features via deep canonical correlation analysis

Weiran Wang, Raman Arora, Karen Livescu, Jeff A. Bilmes
ICASSP 2015accepted

It has been previously shown that, when both acoustic and articulatory training data are available, it is possible to improve phonetic recognition accuracy by learning acoustic features from this multi-view data with canonical correlation analysis (CCA). In contrast with previous work based on linea…

Cited by 0Source ↗Scholar ↗