AIConfPaper
← Search

Dan Jurafsky

47 accepted papers

2026

CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition

Martijn Bartelds, Ananjan Nandi, Moussa Koulako Bala Doumbouya, Dan Jurafsky, Tatsunori Hashimoto, Karen Livescu
ICLR 2026poster

Modern deep learning models often achieve high overall performance, but consistently fail on specific subgroups. Group distributionally robust optimization (group DRO) addresses this problem by minimizing the worst-group loss, but it fails when group losses misrepresent performance differences betwe…

Cited by 0Source ↗code ↗Scholar ↗
2026

ELEPHANT: Measuring and understanding social sycophancy in LLMs

Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, Dan Jurafsky
ICLR 2026poster

LLMs are known to exhibit _sycophancy_: agreeing with and flattering users, even at the cost of correctness. Prior work measures sycophancy only as direct agreement with users' explicitly stated beliefs that can be compared to a ground truth. This fails to capture broader forms of sycophancy such as…

Cited by 0Source ↗code ↗Scholar ↗
2026

From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning

Chen Shani, Liron Soffer, Dan Jurafsky, Yann LeCun, Ravid Shwartz-Ziv
ICLR 2026poster

Humans organize knowledge into compact categories that balance compression with semantic meaning preservation. Large Language Models (LLMs) demonstrate striking linguistic abilities, yet whether they achieve this same balance remains unclear. We apply the Information Bottleneck principle to quantita…

Cited by 0Source ↗Scholar ↗
2026

Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similarity

Moussa Koulako Bala Doumbouya, Dan Jurafsky, Christopher D Manning
ICLR 2026poster

Work in psychology has highlighted that the geometric model of similarity standard in deep learning is not psychologically plausible because its metric properties such as symmetry do not align with human perception of similarity. In contrast, (Tversky,1977) proposed an axiomatic theory of simila…

Cited by 0Source ↗code ↗Scholar ↗
2025

AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders

Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky +2
ICML 2025spotlight

Fine-grained steering of language model outputs is essential for safety and reliability. Prompting and finetuning are widely used to achieve these goals, but interpretability researchers have proposed a variety of representation-based techniques as well, including sparse autoencoders (SAEs), linear…

Cited by 12Source ↗PDF ↗code ↗Scholar ↗
2025

Can Unconfident LLM Annotations Be Used for Confident Conclusions?

Kristina Gligoric, Tijana Zrnic, Cinoo Lee, Emmanuel Candes, Dan Jurafsky
NAACL 2025long

Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. In computational social science (CSS), researchers are increasingly leveraging LLM annotations to complement slow and expensive…

Cited by 12Source ↗PDF ↗code ↗Scholar ↗
2025

Constructing Datasets From Public Police Body Camera Footage

Jamie Rosas-Smith, Martijn Bartelds, Ruizhe Huang, Leibny Paola García-Perera, Karen Livescu, Dan Jurafsky +1
ICASSP 2025accepted

The enormous potential of body-worn cameras to improve accountability in policing remains largely unrealized due to large volumes of unreviewed footage. Transcription and diarization tools could aid in reviewing footage, but lack of public data hinders their development. We develop a pipeline to con…

Cited by 0Source ↗Scholar ↗
2025

False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models

Julie Kallini, Dan Jurafsky, Christopher Potts, Martijn Bartelds
EMNLP 2025

Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages. Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages? Prior work offers mixed evidence, partly due to varied setups and confounders, such as t

Cited by 0Source ↗PDF ↗Scholar ↗
2025

In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

Nathan Roll, Calbert Graham, Yuka Tatsumi, Kim Tien Nguyen, Meghan Sumner, Dan Jurafsky
EMNLP 2025

Human listeners readily adjust to unfamiliar speakers and language varieties through exposure, but do these adaptation benefits extend to state-of-the-art spoken language models (SLMs)? We introduce a scalable framework that allows for in-context learning (ICL) in Phi-4 Multimodal (Phi-4-MM) using i

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

REL-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance

Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Nouha Dziri, Dan Jurafsky, Maarten Sap
NAACL 2025long

The ability to communicate uncertainty and knowledge limitations is crucial for the safety of large language models (LLMs). Current evaluations of these abilities typically examine the correspondence between model accuracy and its internal probabilities or linguistic outputs. However, evaluation of…

Cited by 6Source ↗PDF ↗Scholar ↗
2025

Rethinking Word Similarity: Semantic Similarity through Classification Confusion

Kaitlyn Zhou, Haishan Gao, Sarah Li Chen, Dan Edelstein, Dan Jurafsky, Chen Shani
NAACL 2025long

Word similarity has many applications to social science and cultural analytics tasks like measuring meaning change over time and making sense of contested terms. Yet traditional similarity methods based on cosine similarity between word embeddings cannot capture the context-dependent, asymmetrical,…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

What can large language models do for sustainable food?

Anna Thomas, Adam Yee, Andrew Mayne, Maya B. Mathur, Dan Jurafsky, Kristina Gligorić
ICML 2025poster

Food systems are responsible for a third of human-caused greenhouse gas emissions. We investigate what Large Language Models (LLMs) can contribute to reducing the environmental impacts of food production. We define a typology of design and prediction tasks based on the sustainable food literature an…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

h4rm3l: A Language for Composable Jailbreak Attack Synthesis

Moussa Koulako Bala Doumbouya, Ananjan Nandi, Gabriel Poesia, Davide Ghilardi, Anna Goldie, Federico Bianchi +2
ICLR 2025poster

Despite their demonstrated valuable capabilities, state-of-the-art (SOTA) widely deployed large language models (LLMs) still have the potential to cause harm to society due to the ineffectiveness of their safety filters, which can be bypassed by prompt transformations called jailbreak attacks. Curre…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

A Benchmark for Learning to Translate a New Language from One Grammar Book

Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, Luke Melas-Kyriazi
ICLR 2024spotlight

Large language models (LLMs) can perform impressive feats with in-context learning or lightweight finetuning. It is natural to wonder how well these models adapt to genuinely new tasks, but how does one find tasks that are unseen in internet-scale training sets? We turn to a field that is explicitly…

Cited by 40Source ↗PDF ↗Scholar ↗
2024

CausalGym: Benchmarking causal interpretability methods on linguistic tasks

Aryaman Arora, Dan Jurafsky, Christopher Potts
ACL 2024long

Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e.g., surprisal comparisons). At the same time, research in model interpretability has begun to illuminate the abstract causal mechanisms shaping LM b…

Cited by 17Source ↗PDF ↗code ↗Scholar ↗
2024

Grounding Gaps in Language Model Generations

Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, Dan Jurafsky
NAACL 2024long

Effective conversation requires common ground: a shared understanding between the participants. Common ground, however, does not emerge spontaneously in conversation. Speakers and listeners work together to both identify and construct a shared basis while avoiding misunderstanding. To accomplish gro…

Cited by 28Source ↗PDF ↗code ↗Scholar ↗
2024

How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis

Federico Bianchi, Patrick John Chia, Mert Yuksekgonul, Jacopo Tagliabue, Dan Jurafsky, James Zou
ICML 2024poster

Negotiation is the basis of social interactions; humans negotiate everything from the price of cars to how to share common resources. With rapidly growing interest in using large language models (LLMs) to act as agents on behalf of human users, such LLM agents would also need to be able to negotiate…

Cited by 34Source ↗PDF ↗code ↗Scholar ↗
2024

Model Alignment as Prospect Theoretic Optimization

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela
ICML 2024spotlight

Kahneman & Tversky's $\textit{prospect theory}$ tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases---the suc…

Cited by 32Source ↗PDF ↗Scholar ↗
2024

NLP Systems That Can’t Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps

Kristina Gligoric, Myra Cheng, Lucia Zheng, Esin Durmus, Dan Jurafsky
NAACL 2024long

The use of words to convey speaker’s intent is traditionally distinguished from the ‘mention’ of words for quoting what someone said, or pointing out properties of a word. Here we show that computationally modeling this use-mention distinction is crucial for dealing with counterspeech online. Counte…

Cited by 11Source ↗PDF ↗code ↗Scholar ↗
2024

ReFT: Representation Finetuning for Language Models

Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning +1
NeurIPS 2024spotlight

Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of *weights*. However, much prior interpretability work has shown that *representations* encode rich semantic information, suggesting that editing representations might be a more powerful al…

Cited by 70Source ↗PDF ↗code ↗Scholar ↗
2024

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto +1
ICLR 2024poster

Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful. However, a perfectly helpful model will follow even the most malicious instructions and readily generate harmful content. In this paper, we raise concerns over…

Cited by 172Source ↗PDF ↗Scholar ↗
2024

string2string: A Modern Python Library for String-to-String Algorithms

Mirac Suzgun, Stuart Shieber, Dan Jurafsky
ACL 2024system demonstrations

We introduce **string2string**, an open-source library that offers a comprehensive suite of efficient algorithms for a broad range of string-to-string problems. It includes traditional algorithmic solutions as well as recent advanced neural approaches to tackle various problems in string alignment,…

Cited by 8Source ↗PDF ↗code ↗Scholar ↗
2023

Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes

Connor Toups, Rishi Bommasani, Kathleen Creel, Sarah H Bana, Dan Jurafsky, Percy Liang
NeurIPS 2023poster

Machine learning is traditionally studied at the model level: researchers measure and improve the accuracy, robustness, bias, efficiency, and other dimensions of specific models. In practice, however, the societal impact of any machine learning model is partially determined by the context into which…

Cited by 11Source ↗PDF ↗Scholar ↗
2023

Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding

Mirac Suzgun, Luke Melas-Kyriazi, Dan Jurafsky
ACL 2023findings

In open-ended natural-language generation, existing text decoding methods typically struggle to produce text which is both diverse and high-quality. Greedy and beam search are known to suffer from text degeneration and linguistic diversity issues, while temperature, top-k, and nucleus sampling yield…

Cited by 36Source ↗PDF ↗code ↗Scholar ↗
2023

Injecting structural hints: Using language models to study inductive biases in language learning

Isabel Papadimitriou, Dan Jurafsky
EMNLP 2023long findings

Both humans and transformer language models are able to learn language without explicit structural supervision. What cognitive inductive biases make this learning possible? Here, we examine the effect of different inductive learning biases by actively controlling the inductive biases of artificial l…

Cited by 0Source ↗code ↗Scholar ↗
2023

Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation

Martijn Bartelds, Nay San, Bradley McDonnell, Dan Jurafsky, Martijn Wieling
ACL 2023long

The performance of automatic speech recognition (ASR) systems has advanced substantially in recent years, particularly for languages for which a large amount of transcribed speech is available. Unfortunately, for low-resource languages, such as minority languages, regional languages or dialects, ASR…

Cited by 48Source ↗PDF ↗code ↗Scholar ↗
2023

Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

Myra Cheng, Esin Durmus, Dan Jurafsky
ACL 2023long

To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs. Toward this end, we present Marked Personas, a prompt-based method to measure stereotypes in LLMs for intersectional demographic groups without any lexic…

Cited by 180Source ↗PDF ↗code ↗Scholar ↗
2023

Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models

Kaitlyn Zhou, Dan Jurafsky, Tatsunori Hashimoto
EMNLP 2023long main

The increased deployment of LMs for real-world tasks involving knowledge and facts makes it important to understand model epistemology: what LMs think they know, and how their attitudes toward that knowledge are affected by language use in their inputs. Here, we study an aspect of model epistemology…

Cited by 0Source ↗Scholar ↗
2023

When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, James Zou
ICLR 2023top-5%

Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode the compositional relationships between objects and attributes. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the abili…

Cited by 430Source ↗PDF ↗code ↗Scholar ↗
2022

Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?

Rishi Bommasani, Kathleen Creel, Ananya Kumar, Dan Jurafsky, Percy Liang
NeurIPS 2022accept

As the scope of machine learning broadens, we observe a recurring theme of *algorithmic monoculture*: the same systems, or systems that share components (e.g. datasets, models), are deployed by multiple decision-makers. While sharing offers advantages like amortizing effort, it also has risks. We…

Cited by 100Source ↗PDF ↗Scholar ↗
2022

Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

Peter Henderson, Mark Simon Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky +1
NeurIPS 2022accept

One concern with the rise of large language models lies with their potential for significant harm, particularly from pretraining on biased, obscene, copyrighted, and private information. Emerging ethical approaches have attempted to filter pretraining material, but such approaches have been ad hoc a…

Cited by 114Source ↗PDF ↗code ↗Scholar ↗
2022

Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words

Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, Dan Jurafsky
ACL 2022short

Cosine similarity of contextual embeddings is used in many NLP tasks (e.g., QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in which word similarities estimated by cosine over BERT embeddings are understated and trace this effect to training data frequency. We find that r…

Cited by 63Source ↗PDF ↗code ↗Scholar ↗
2022

Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models

Mirac Suzgun, Luke Melas-Kyriazi, Dan Jurafsky
EMNLP 2022main

We propose a method for arbitrary textual style transfer (TST)—the task of transforming a text into any given style—utilizing general-purpose pre-trained language models. Our method, Prompt-and-Rerank, is based on a mathematical formulation of the TST task, decomposing it into three constituent comp…

Cited by 68Source ↗PDF ↗code ↗Scholar ↗
2021

Causal Effects of Linguistic Properties

Reid Pryzant, Dallas Card, Dan Jurafsky, Victor Veitch, Dhanya Sridhar
NAACL 2021long

We consider the problem of using observational data to estimate the causal effects of linguistic properties. For example, does writing a complaint politely lead to a faster response time? How much will a positive product review increase sales? This paper addresses two technical challenges related to…

Cited by 61Source ↗PDF ↗code ↗Scholar ↗
2021

Focus on what matters: Applying Discourse Coherence Theory to Cross Document Coreference

William Held, Dan Iter, Dan Jurafsky
EMNLP 2021main

Performing event and entity coreference resolution across documents vastly increases the number of candidate mentions, making it intractable to do the full n2 pairwise comparisons. Existing approaches simplify by considering coreference only within document clusters, but this fails to handle inter-c…

Cited by 20Source ↗PDF ↗code ↗Scholar ↗
2021

Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation

Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, Dan Jurafsky
NAACL 2021long

Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. However, existing report generation systems, despite achieving high performances on natural langu…

Cited by 195Source ↗PDF ↗code ↗Scholar ↗
2021

Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky +1
ACL 2021long

In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In education, teachers’ uptake of student contributions has been linked to higher student achievement. Yet measuring and imp…

Cited by 81Source ↗PDF ↗code ↗Scholar ↗
2021

Nearest Neighbor Machine Translation

Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis
ICLR 2021poster

We introduce $k$-nearest-neighbor machine translation ($k$NN-MT), which predicts tokens with a nearest-neighbor classifier over a large datastore of cached examples, using representations from a neural translation model for similarity search. This approach requires no additional training and scales…

Cited by 329Source ↗PDF ↗code ↗Scholar ↗
2020

Generalization through Memorization: Nearest Neighbor Language Models

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis
ICLR 2020poster

We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including…

Cited by 952Source ↗code ↗Scholar ↗
2020

Language Through a Prism: A Spectral Approach for Multiscale Language Representations

Alex Tamkin, Dan Jurafsky, Noah Goodman
NeurIPS 2020poster

Language exhibits structure at a wide range of scales, from subwords to words, sentences, paragraphs, and documents. We propose building models that isolate scale-specific information in deep representations, and develop methods for encouraging models during training to learn more about particular s…

Cited by 48Source ↗PDF ↗code ↗Scholar ↗
2018

Embedding Logical Queries on Knowledge Graphs

Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, Jure Leskovec
NeurIPS 2018poster

Learning low-dimensional embeddings of knowledge graphs is a powerful approach used to predict unobserved or missing edges between entities. However, an open challenge in this area is developing techniques that can go beyond simple edge prediction and handle more complex logical queries, which might…

Cited by 381Source ↗PDF ↗code ↗Scholar ↗
2017

Data Noising as Smoothing in Neural Network Language Models

Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky +1
ICLR 2017poster

Data noising is an effective technique for regularizing neural network models. While noising is widely adopted in application domains such as vision and speech, commonly used noising primitives have not been developed for discrete sequence-level settings such as language modeling. In this paper, we…

Cited by 321Source ↗Scholar ↗