AIConfPaper
← Search

Dragomir Radev

46 accepted papers

2025

SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models

Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle +48
NAACL 2025long

Large Language Models (LLMs) reproduce and exacerbate the social biases present in their training data, and resources to quantify this issue are limited. While research has attempted to identify and mitigate such biases, most efforts have been concentrated around English, lagging the rapid advanceme…

Cited by 1Source ↗PDF ↗Scholar ↗
2024

A Call for Clarity in Beam Search: How It Works and When It Stops

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Dragomir Radev, Yejin Choi, Noah A. Smith
COLING 2024main

Text generation with beam search has proven successful in a wide range of applications. We point out that, though largely overlooked in the literature, the commonly-used implementation of beam decoding (e.g., Hugging Face Transformers and fairseq) uses a first come, first served heuristic: it keeps…

Cited by 1Source ↗PDF ↗code ↗Scholar ↗
2024

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization

Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty +4
NAACL 2024findings

While large language models (LLMs) can already achieve strong performance on standard generic summarization benchmarks, their performance on more complex summarization task settings is less studied. Therefore, we benchmark LLMs on instruction controllable text summarization, where the model input co…

Cited by 50Source ↗PDF ↗code ↗Scholar ↗
2024

FOLIO: Natural Language Reasoning with First-Order Logic

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou +29
EMNLP 2024main

Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse d…

Cited by 150Source ↗PDF ↗code ↗Scholar ↗
2024

Fair Abstractive Summarization of Diverse Perspectives

Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi +6
NAACL 2024long

People from different social and demographic groups express diverse perspectives and conflicting opinions on a broad set of topics such as product reviews, healthcare, law, and politics. A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain…

Cited by 13Source ↗PDF ↗code ↗Scholar ↗
2024

On Learning to Summarize with Large Language Models as References

Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu +2
NAACL 2024long

Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. Therefore, we study an LLM-as-reference learning setting for smaller text summarization models to investig…

Cited by 89Source ↗PDF ↗code ↗Scholar ↗
2024

P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains

Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou +10
EMNLP 2024finding

Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales, which are not sufficient for properly assessing model’s capabilities. We present P-FOLIO, a human-annotated dataset consisting of diverse and…

Cited by 2Source ↗PDF ↗Scholar ↗
2023

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak +9
ACL 2023long

The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In…

Cited by 66Source ↗PDF ↗code ↗Scholar ↗
2023

Binding Language Models in Symbolic Languages

Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu +6
ICLR 2023top-25%

Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of la…

Cited by 195Source ↗PDF ↗code ↗Scholar ↗
2023

Crosslingual Generalization through Multitask Finetuning

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao +13
ACL 2023long

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned varia…

Cited by 770Source ↗PDF ↗code ↗Scholar ↗
2023

Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies

Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang +2
EMNLP 2023long findings

In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context that has been supplemented with a few examples or task-specific instructions. In this paper, we aim to extend this method…

Cited by 0Source ↗Scholar ↗
2023

HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution

Ye Liu, Semih Yavuz, Rui Meng, Dragomir Radev, Caiming Xiong, Shafiq Joty +1
EMNLP 2023long findings

The dominant paradigm of textual question answering systems is based on end-to-end neural networks, which excels at answering natural language questions but falls short on complex ones. This stands in contrast to the broad adaptation of semantic parsing approaches over structured data sources (e.g.,…

Cited by 0Source ↗Scholar ↗
2023

HiPool: Modeling Long Documents Using Graph Neural Networks

Irene Li, Aosong Feng, Dragomir Radev, Rex Ying
ACL 2023short

Encoding long sequences in Natural Language Processing (NLP) is a challenging problem. Though recent pretraining language models achieve satisfying performances in many NLP tasks, they are still restricted by a pre-defined maximum length, making them challenging to be extended to longer sequences. S…

Cited by 7Source ↗PDF ↗code ↗Scholar ↗
2023

LEVER: Learning to Verify Language-to-Code Generation with Execution

Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang +1
ICML 2023poster

The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine LLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it i…

Cited by 227Source ↗PDF ↗code ↗Scholar ↗
2023

Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions

Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev +1
ICLR 2023poster

Pretrained language models have shown superior performance on many natural language processing tasks, yet they still struggle at multi-step formal reasoning tasks like grade school math problems. One key challenge of finetuning them to solve such math reasoning problems is that many existing dataset…

Cited by 35Source ↗PDF ↗code ↗Scholar ↗
2023

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron Halfaker, Liqun Shao, Daniel McDuff +3
ACL 2023findings

Natural language contains rich logical structures and logical information, and correctly detecting and accurately understanding these logical structures and information underlying natural language texts is very crucial for NLP models’ performance on many important NLU and NLG tasks. Existing pre-tra…

Cited by 1Source ↗PDF ↗Scholar ↗
2023

Molformer: Motif-Based Transformer on 3D Heterogeneous Molecular Graphs

Fang Wu, Dragomir Radev, Stan Z. Li
AAAI 2023technical

Procuring expressive molecular representations underpins AI-driven molecule design and scientific discovery. The research mainly focuses on atom-level homogeneous molecular graphs, ignoring the rich information in subgraphs or motifs. However, it has been widely accepted that substructures play a do…

Cited by 66Source ↗PDF ↗code ↗Scholar ↗
2023

On Improving Summarization Factual Consistency from Natural Language Feedback

Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, Ahmed Hassan Awadallah
ACL 2023long

Despite the recent progress in language generation models, their outputs may not always meet user expectations. In this work, we study whether informational feedback in natural language can be leveraged to improve generation quality and user preference alignment. To this end, we consider factual con…

Cited by 42Source ↗PDF ↗code ↗Scholar ↗
2023

QTSumm: Query-Focused Summarization over Tabular Data

Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou +6
EMNLP 2023long main

People primarily consult tables to conduct data analysis or answer specific questions. Text generation systems that can provide accurate table summaries tailored to users' information needs can facilitate more efficient access to relevant data insights. Motivated by this, we define a new query-focu…

Cited by 0Source ↗code ↗Scholar ↗
2023

RealTime QA: What's the Answer Right Now?

Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu +4
NeurIPS 2023poster

We introduce RealTime QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). RealTime QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore chall…

Cited by 136Source ↗PDF ↗code ↗Scholar ↗
2023

Rethinking Explaining Graph Neural Networks via Non-parametric Subgraph Matching

Fang Wu, Siyuan Li, Xurui Jin, Yinghui Jiang, Dragomir Radev, Zhangming Niu +1
ICML 2023poster

The success of graph neural networks (GNNs) provokes the question about explainability: ``Which fraction of the input graph is the most determinant of the prediction?'' Particularly, parametric explainers prevail in existing approaches because of their more robust capability to decipher the black-bo…

Cited by 22Source ↗PDF ↗code ↗Scholar ↗
2023

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation

Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han +5
ACL 2023long

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation…

Cited by 116Source ↗PDF ↗code ↗Scholar ↗
2023

RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations

Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang +2
ACL 2023long

Despite significant progress having been made in question answering on tabular data (Table QA), it’s unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the r…

Cited by 32Source ↗PDF ↗code ↗Scholar ↗
2023

bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark

Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova +4
ACL 2023long

We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural language inference, fact-checking, named…

Cited by 3Source ↗PDF ↗code ↗Scholar ↗
2022

BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization

Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, Dragomir Radev
EMNLP 2022finding

The majority of existing text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such datasets will offer limited challenges for future text summarization systems. We…

Cited by 151Source ↗PDF ↗code ↗Scholar ↗
2022

CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning

Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade +4
NAACL 2022long

Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. Although significant progress has been achieved by using pre-trained neural language models, substantial amounts of hallucinated content are found during the human evaluati…

Cited by 71Source ↗PDF ↗Scholar ↗
2022

DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization

Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu +4
ACL 2022long

Transformer-based models have achieved state-of-the-art performance on short-input summarization. However, they still struggle with summarizing longer text. In this paper, we present DYLE, a novel dynamic latent extraction approach for abstractive long-input summarization. DYLE jointly trains an ext…

Cited by 77Source ↗PDF ↗code ↗Scholar ↗
2022

Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries

Xiangru Tang, Alexander Fabbri, Haoran Li, Ziming Mao, Griffin Adams, Borui Wang +3
NAACL 2022long

Current pre-trained models applied for summarization are prone to factual inconsistencies that misrepresent the source text. Evaluating the factual consistency of summaries is thus necessary to develop better models. However, the human evaluation setup for evaluating factual consistency has not been…

Cited by 20Source ↗PDF ↗Scholar ↗
2022

Leveraging Locality in Abstractive Text Summarization

Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed Hassan Awadallah +1
EMNLP 2022main

Neural attention models have achieved significant improvements on many natural language processing tasks. However, the quadratic memory complexity of the self-attention module with respect to the input length hinders their applications in long text summarization. Instead of designing more efficient…

Cited by 32Source ↗PDF ↗code ↗Scholar ↗
2022

R2D2: Robust Data-to-Text with Replacement Detection

Linyong Nan, Lorenzo Jaime Flores, Yilun Zhao, Yixin Liu, Luke Benson, Weijin Zou +1
EMNLP 2022main

Unfaithful text generation is a common problem for text generation systems. In the case of Data-to-Text (D2T) systems, the factuality of the generated text is particularly crucial for any real-world applications. We introduce R2D2, a training framework that addresses unfaithful Data-to-Text generati…

Cited by 17Source ↗PDF ↗code ↗Scholar ↗
2022

ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples

Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, Dragomir Radev
EMNLP 2022main

Reasoning over tabular data requires both table structure understanding and a broad set of table reasoning skills. Current models with table-specific architectures and pre-training methods perform well on understanding table structures, but they still struggle with tasks that require various table r…

Cited by 31Source ↗PDF ↗code ↗Scholar ↗
2022

STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension

Borui Wang, Chengcheng Feng, Arjun Nair, Madelyn Mao, Jai Desai, Asli Celikyilmaz +3
EMNLP 2022main

Abstractive dialogue summarization has long been viewed as an important standalone task in natural language processing, but no previous work has explored the possibility of whether abstractive dialogue summarization can also be used as a means to boost an NLP system’s performance on other important…

Cited by 2Source ↗PDF ↗Scholar ↗
2022

SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents

Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb +3
ACL 2022long

Text summarization helps readers capture salient information from documents, news, interviews, and meetings. However, most state-of-the-art pretrained language models (LM) are unable to efficiently process long text for many summarization tasks. In this paper, we propose SummN, a simple, flexible, a…

Cited by 108Source ↗PDF ↗code ↗Scholar ↗
2022

Twist Decoding: Diverse Generators Guide Each Other

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev +2
EMNLP 2022main

Many language generation models are now available for a wide range of generation tasks, including machine translation and summarization. Combining such diverse models may lead to further progress, but ensembling generation models is challenging during inference: conventional ensembling methods (e.g.…

Cited by 4Source ↗PDF ↗code ↗Scholar ↗
2022

Uni-Parser: Unified Semantic Parser for Question Answering on Knowledge Base and Database

Ye Liu, Semih Yavuz, Rui Meng, Dragomir Radev, Caiming Xiong, Yingbo Zhou
EMNLP 2022main

Parsing natural language questions into executable logical forms is a useful and interpretable way to perform question answering on structured data such as knowledge bases (KB) or databases (DB). However, existing approaches on semantic parsing cannot adapt to both modalities, as they suffer from th…

Cited by 26Source ↗PDF ↗code ↗Scholar ↗
2022

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga +17
EMNLP 2022main

Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases. Since the inputs and outputs of SKG tasks are heterogeneous, they have been studied separately by different communities,…

Cited by 214Source ↗PDF ↗code ↗Scholar ↗
2021

An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next

Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb +3
EMNLP 2021finding

Dialogue summarization helps readers capture salient information from long conversations in meetings, interviews, and TV series. However, real-world dialogues pose a great challenge to current summarization models, as the dialogue length typically exceeds the input limits imposed by recent transform…

Cited by 60Source ↗PDF ↗code ↗Scholar ↗
2021

ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining

Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad +1
ACL 2021long

While online conversations can cover a vast amount of information in many different formats, abstractive text summarization has primarily focused on modeling solely news articles. This research gap is due, in part, to the lack of standardized datasets for summarizing online discussions. To address t…

Cited by 74Source ↗PDF ↗code ↗Scholar ↗
2021

DART: Open-Domain Structured Data Record to Text Generation

Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh +18
NAACL 2021long

We present DART, an open domain structured DAta Record to Text generation dataset with over 82k instances (DARTs). Data-to-text annotations can be a costly process, especially when dealing with tables which are the major source of structured data and contain nontrivial structures. To this end, we pr…

Cited by 200Source ↗PDF ↗code ↗Scholar ↗
2021

GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang +3
ICLR 2021poster

We present GraPPa, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG). We p…

Cited by 106Source ↗PDF ↗Scholar ↗
2021

Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation

Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty +2
NAACL 2021long

Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. However, these models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, nich…

Cited by 117Source ↗PDF ↗Scholar ↗
2021

QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha +5
NAACL 2021long

Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard t…

Cited by 341Source ↗PDF ↗code ↗Scholar ↗
2021

SummerTime: Text Summarization Toolkit for Non-experts

Ansong Ni, Zhangir Azerbayev, Mutethia Mutuma, Troy Feng, Yusen Zhang, Tao Yu +2
EMNLP 2021system demonstrations

Recent advances in summarization provide models that can generate summaries of higher quality. Such models now exist for a number of summarization tasks, including query-based summarization, dialogue summarization, and multi-document summarization. While such models and tasks are rapidly growing in…

Cited by 14Source ↗PDF ↗code ↗Scholar ↗
2021

Unsupervised Cross-Domain Prerequisite Chain Learning using Variational Graph Autoencoders

Irene Li, Vanessa Yan, Tianxiao Li, Rihao Qu, Dragomir Radev
ACL 2021short

Learning prerequisite chains is an important task for one to pick up knowledge efficiently in both known and unknown domains. For example, one may be an expert in the natural language processing (NLP) domain, but want to determine the best order in which to learn new concepts in an unfamiliar Comput…

Cited by 18Source ↗PDF ↗Scholar ↗
2020

R-VGAE: Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning

Irene Li, Alexander Fabbri, Swapnil Hingmire, Dragomir Radev
COLING 2020main

The task of concept prerequisite chain learning is to automatically determine the existence of prerequisite relationships among concept pairs. In this paper, we frame learning prerequisite relationships among concepts as an unsupervised task with no access to labeled concept pairs during training. W…

Cited by 37Source ↗PDF ↗code ↗Scholar ↗