AIConfPaper
← Search

Karthik R Narasimhan

27 accepted papers

2025

An Annotated Dataset of Errors in Premodern Greek and Baselines for Detecting Them

Creston Brooks, Johannes Haubold, Charlie Cowen-Breen, Jay White, Desmond DeVaul, Frederick Riemenschneider +2
NAACL 2025findings

As premodern texts are passed down over centuries, errors inevitably accrue. These errors can be challenging to identify, as some have survived undetected for so long precisely because they are so elusive. While prior work has evaluated error detection methods on artificially-generated errors, we in…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

Contextual Experience Replay for Self-Improvement of Language Agents

Yitao Liu, Chenglei Si, Karthik R Narasimhan, Shunyu Yao
ACL 2025long

Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in these complex tasks. Moreover, current LLM agents are not designed to continually learn from past experiences during in…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities

Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner +10
ICML 2025poster

Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present *EnIGMA*, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools an…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

PersonaGym: Evaluating Persona Agents and LLMs

Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit +3
EMNLP 2025

Persona agents, which are LLM agents conditioned to act according to an assigned persona, enable contextually rich and user-aligned interactions across domains like education and healthcare.However, evaluating how faithfully these agents adhere to their personas remains a significant challenge, part

Cited by 0Source ↗PDF ↗Scholar ↗
2025

SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?

John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu +7
ICLR 2025poster

Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositor…

Cited by 198Source ↗PDF ↗Scholar ↗
2025

When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration

Quan Shi, Carlos E Jimenez, Shunyu Yao, Nick Haber, Diyi Yang, Karthik R Narasimhan
NeurIPS 2025poster

As large language models (LLMs) increasingly serve as close collaborators for humans, it is crucial that they express their reasoning in ways that humans can understand and learn from. However, this capability remains relatively less understood and under-evaluated. To address this, we introduce a co…

Cited by 0Source ↗Scholar ↗
2025

{$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains

Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik R Narasimhan
ICLR 2025poster

Existing benchmarks for language agents do not set them up to interact with human users or follow domain-specific rules, both of which are vital to safe and realistic deployment. We propose $\tau$-bench, a benchmark with two domains (retail and airline) emulating dynamic conversations between a user…

Cited by 2Source ↗PDF ↗Scholar ↗
2024

COLLIE: Systematic Construction of Constrained Text Generation Tasks

Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, Karthik R Narasimhan
ICLR 2024poster

Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained generation usually focus on fixed constraint types (e.g. generate a sentence cont…

Cited by 29Source ↗PDF ↗code ↗Scholar ↗
2024

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan +1
NeurIPS 2024poster

Language model agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that language model agents represent…

Cited by 217Source ↗PDF ↗Scholar ↗
2024

SWE-bench: Can Language Models Resolve Real-world Github Issues?

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press +1
ICLR 2024oral

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of langu…

Cited by 500Source ↗PDF ↗Scholar ↗
2023

C-STS: Conditional Semantic Textual Similarity

Ameet Deshpande, Carlos E Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit +3
EMNLP 2023long main

Semantic textual similarity (STS) has been a cornerstone task in NLP that measures the degree of similarity between a pair of sentences, with applications in information retrieval, question answering, and embedding methods. However, it is an inherently ambiguous task, with the sentence similarity de…

Cited by 0Source ↗code ↗Scholar ↗
2023

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

John Yang, Akshara Prabhakar, Karthik R Narasimhan, Shunyu Yao
NeurIPS 2023poster

Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequen…

Cited by 104Source ↗PDF ↗code ↗Scholar ↗
2023

MUX-PLMs: Data Multiplexing for High-throughput Language Models

Vishvak Murahari, Ameet Deshpande, Carlos E Jimenez, Izhak Shafran, Mingqiu Wang, Yuan Cao +1
EMNLP 2023long findings

The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasing model sizes coupled with hardware shortages has limited affordable access and poses a pressing need for efficiency ap…

Cited by 0Source ↗code ↗Scholar ↗
2023

ReAct: Synergizing Reasoning and Acting in Language Models

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan +1
ICLR 2023top-5%

While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics…

Cited by 2772Source ↗PDF ↗code ↗Scholar ↗
2023

Reflexion: language agents with verbal reinforcement learning

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, Shunyu Yao
NeurIPS 2023poster

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning…

Cited by 1443Source ↗PDF ↗code ↗Scholar ↗
2023

SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification

Pranjal Aggarwal, Ameet Deshpande, Karthik R Narasimhan
ICML 2023poster

Extreme classification (XC) involves predicting over large numbers of classes (thousands to millions), with real-world applications like news article classification and e-commerce product tagging. The zero-shot version of this task requires generalization to novel classes without additional supervis…

Cited by 13Source ↗PDF ↗code ↗Scholar ↗
2023

Toxicity in chatgpt: Analyzing persona-assigned language models

Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik R Narasimhan
EMNLP 2023long findings

Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students…

Cited by 0Source ↗Scholar ↗
2023

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao +1
NeurIPS 2023oral

Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where i…

Cited by 2737Source ↗PDF ↗code ↗Scholar ↗
2022

DataMUX: Data Multiplexing for Neural Networks

Vishvak Murahari, Carlos E Jimenez, Runzhe Yang, Karthik R Narasimhan
NeurIPS 2022accept

In this paper, we introduce \emph{data multiplexing} (DataMUX), a technique that enables deep neural networks to process multiple inputs simultaneously using a single compact representation. DataMUX demonstrates that neural networks are capable of generating accurate predictions over \emph{mixtures…

Cited by 22Source ↗PDF ↗code ↗Scholar ↗
2022

Learning Physics Constrained Dynamics Using Autoencoders

Tsung-Yen Yang, Justinian P. Rosca, Karthik R Narasimhan, Peter Ramadge
NeurIPS 2022accept

We consider the problem of estimating states (e.g., position and velocity) and physical parameters (e.g., friction, elasticity) from a sequence of observations when provided a dynamic equation that describes the behavior of the system. The dynamic equation can arise from first principles (e.g., Newt…

Cited by 26Source ↗PDF ↗Scholar ↗
2022

Leveraging Language for Accelerated Learning of Tool Manipulation

Allen Z. Ren, Bharat Govil, Tsung-Yen Yang, Karthik R Narasimhan, Anirudha Majumdar
CoRL 2022poster

Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain divers…

Cited by 53Source ↗Scholar ↗
2022

Linking Emergent and Natural Languages via Corpus Transfer

Shunyu Yao, Mo Yu, Yang Zhang, Karthik R Narasimhan, Joshua B. Tenenbaum, Chuang Gan
ICLR 2022spotlight

The study of language emergence aims to understand how human languages are shaped by perceptual grounding and communicative intent. Computational approaches to emergent communication (EC) predominantly consider referential games in limited domains and analyze the learned protocol within the game fra…

Cited by 24Source ↗PDF ↗code ↗Scholar ↗
2022

Multi-Stage Episodic Control for Strategic Exploration in Text Games

Jens Tuyls, Shunyu Yao, Sham M. Kakade, Karthik R Narasimhan
ICLR 2022spotlight

Text adventure games present unique challenges to reinforcement learning methods due to their combinatorially large action spaces and sparse rewards. The interplay of these two factors is particularly demanding because large action spaces require extensive exploration, while sparse rewards provide l…

Cited by 26Source ↗PDF ↗code ↗Scholar ↗
2022

Using natural language and program abstractions to instill human inductive biases in machines

Sreejan Kumar, Carlos G Correa, Ishita Dasgupta, Raja Marjieh, Michael Hu, Robert D. Hawkins +4
NeurIPS 2022accept

Strong inductive biases give humans the ability to quickly learn to perform a variety of tasks. Although meta-learning is a method to endow neural networks with useful inductive biases, agents trained by meta-learning may sometimes acquire very different strategies from humans. We show that co-train…

Cited by 53Source ↗PDF ↗code ↗Scholar ↗
2022

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Shunyu Yao, Howard Chen, John Yang, Karthik R Narasimhan
NeurIPS 2022accept

Most existing benchmarks for grounding language in interactive environments either lack realistic linguistic elements, or prove difficult to scale up due to substantial human involvement in the collection of data or feedback signals. We develop WebShop – a simulated e-commerce website environment wi…

Cited by 432Source ↗PDF ↗code ↗Scholar ↗
2021

SILG: The Multi-domain Symbolic Interactive Language Grounding Benchmark

Victor Zhong, H. J. Austin Wang, Sida Wang, Karthik R Narasimhan, Luke Zettlemoyer
NeurIPS 2021poster

Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection of diverse grounded language learning…

Cited by 19Source ↗PDF ↗Scholar ↗
2021

Safe Reinforcement Learning with Natural Language Constraints

Tsung-Yen Yang, Michael Hu, Yinlam Chow, Peter Ramadge, Karthik R Narasimhan
NeurIPS 2021spotlight

While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propos…

Cited by 41Source ↗PDF ↗Scholar ↗