← Search

Nikita Nangia

9 accepted papers

2025

SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models

NAACL 2025long

Large Language Models (LLMs) reproduce and exacerbate the social biases present in their training data, and resources to quantify this issue are limited. While research has attempted to identify and mitigate such biases, most efforts have been concentrated around English, lagging the rapid advanceme…

Cited by 1SourcePDFScholar
2023

Common Law Annotations: Investigating the Stability of Dialog System Output Annotations

ACL 2023findings

Metrics for Inter-Annotator Agreement (IAA), like Cohen’s Kappa, are crucial for validating annotated datasets. Although high agreement is often used to show the reliability of annotation procedures, it is insufficient to ensure or reproducibility. While researchers are encouraged to increase annota…

Cited by 5SourcePDFScholar
2023

What Do NLP Researchers Believe? Results of the NLP Community Metasurvey

ACL 2023long

We present the results of the NLP Community Metasurvey. Run from May to June 2022, it elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split in…

Cited by 39SourcePDFScholar
2022

BBQ: A hand-built bias benchmark for question answering

ACL 2022findings

It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA). We introduce the Bias Benchmark for QA (BBQ), a dataset of question-sets constructed by the authors that highlight at…

2022

QuALITY: Question Answering with Long Input Texts, Yes!

NAACL 2022long

To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our qu…

2022

What Makes Reading Comprehension Questions Difficult?

ACL 2022long

For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However, we do not yet know how best to select text sources to collect a variety of cha…

2021

Does Putting a Linguist in the Loop Improve NLU Data Collection?

EMNLP 2021finding

Many crowdsourced NLP datasets contain systematic artifacts that are identified only after data collection is complete. Earlier identification of these issues should make it easier to create high-quality training and evaluation data. We attempt this by evaluating protocols in which expert linguists…

Cited by 46SourcePDFScholar
2021

What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?

ACL 2021long

Crowdsourcing is widely used to create data for common natural language understanding tasks. Despite the importance of these datasets for measuring and refining model understanding of language, there has been little focus on the crowdsourcing methods used for collecting the datasets. In this paper,…

2019

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

NeurIPS 2019spotlight

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse…