AIConfPaper
← Search

Monojit Choudhury

31 accepted papers

2026

Measuring Meta-Cultural Competency: A Spectral Framework for LLM Knowledge Structures

Sougata Saha, Madhur Jindal, Saurabh Kumar Pandey, Mahardika Ihsani, Alham Fikri Aji, Monojit Choudhury
ICML 2026poster

Most existing cultural evaluation frameworks for large language models (LLMs) focus on matching model outputs to ground-truth answers, primarily measuring factual cultural awareness. This overlooks whether models internalize broader cultural structure and pluralism. We introduce a spectral-analysis-…

Cited by 0Source ↗Scholar ↗
2025

All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages

Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar +63
CVPR 2025highlight

Existing Large Multimodal Models (LMMs) generally focus on only a few regions and languages. As LMMs continue to improve, it is increasingly important to ensure they understand cultural contexts, respect local sensitivities, and support low-resource languages, all while effectively integrating corr…

Cited by 14Source ↗PDF ↗code ↗Scholar ↗
2025

An Interdisciplinary Approach to Human-Centered Machine Translation

Marine Carpuat, Omri Asscher, Kalika Bali, Luisa Bentivogli, Fr{\'e}d{\'e}ric Blain, Lynne Bowker +14
EMNLP 2025

Machine Translation (MT) tools are widely used today, often in contexts where professional translators are not present. Despite progress in MT technology, a gap persists between system development and real-world usage, particularly for non-expert users who may struggle to assess translation reliabil

Cited by 0Source ↗PDF ↗Scholar ↗
2025

CULTURALLY YOURS: A Reading Assistant for Cross-Cultural Content

Saurabh Kumar Pandey, Harshit Budhiraja, Sougata Saha, Monojit Choudhury
COLING 2025system demonstrations

Users from diverse cultural backgrounds frequently face challenges in understanding content from various online sources that are written by people from a different culture. This paper presents CULTURALLY YOURS (CY), a first-of-its-kind cultural reading assistant tool designed to identify culture-spe…

Cited by 2Source ↗PDF ↗code ↗Scholar ↗
2025

Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability

Haonan Li, Xudong Han, Zenan Zhai, Honglin Mu, Hao Wang, Zhenxuan Zhang +29
NAACL 2025system demonstrations

As large language models (LLMs) continue to evolve, leaderboards play a significant role in steering their development. Existing leaderboards often prioritize model capabilities while overlooking safety concerns, leaving a significant gap in responsible AI development. To address this gap, we introd…

Cited by 4Source ↗PDF ↗code ↗Scholar ↗
2025

Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models

Hanin Atwany, Abdul Waheed, Rita Singh, Monojit Choudhury, Bhiksha Raj
ACL 2025finding

Speech foundation models trained at a massive scale, both in terms of model and data size, result in robust systems capable of performing multiple speech tasks, including automatic speech recognition (ASR). These models transcend language and domain barriers, yet effectively measuring their performa…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness

Sougata Saha, Saurabh Kumar Pandey, Monojit Choudhury
NAACL 2025long

Numerous recent studies have shown that Large Language Models (LLMs) are biased towards a Western and Anglo-centric worldview, which compromises their usefulness in non-Western cultural settings. However, “culture” is a complex, multifaceted topic, and its awareness, representation, and modeling in…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Music for All: Representational Bias and Cross-Cultural Adaptability of Music Generation Models

Atharva Mehta, Shivam Chauhan, Amirbek Djanibekov, Atharva Kulkarni, Gus Xia, Monojit Choudhury
NAACL 2025findings

The advent of Music-Language Models has greatly enhanced the automatic music generation capability of AI systems, but they are also limited in their coverage of the musical genres and cultures of the world. We present a study of the datasets and research papers for music generation and quantify the…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps?

Sougata Saha, Saurabh Kumar Pandey, Harshit Gupta, Monojit Choudhury
NAACL 2025long

In a rapidly globalizing and digital world, content such as book and product reviews created by people from diverse cultures are read and consumed by others from different corners of the world. In this paper, we investigate the extent and patterns of gaps in understandability of book reviews due to…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation

Saurabh Kumar Pandey, Sachin Vashistha, Debrup Das, Somak Aditya, Monojit Choudhury
NAACL 2025long

To understand the complexity of sequence classification tasks, Hahn et al. (2021) proposed sensitivity as the number of disjoint subsets of the input sequence that can each be individually changed to change the output. Though effective, calculating sensitivity at scale using this framework is costly…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs

Sougata Saha, Monojit Choudhury
ACL 2025finding

Measuring the generalization ability of Large Language Models (LLMs) is challenging due to data contamination. As models grow and computation becomes cheaper, ensuring tasks and test cases are unseen during training phases will become nearly impossible. We argue that knowledge-retrieval and reasonin…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Viability of Machine Translation for Healthcare in Low-Resourced Languages

Hellina Hailu Nigatu, Nikita Mehandru, Negasi Haile Abadi, Blen Gebremeskel, Ahmed Alaa, Monojit Choudhury
EMNLP 2025

Machine Translation errors in high-stakes settings like healthcare pose unique risks that could lead to clinical harm. The challenges are even more pronounced for low-resourced languages where human translators are scarce and MT tools perform poorly. In this work, we provide a taxonomy of Machine Tr

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

Women, Infamous, and Exotic Beings: A Comparative Study of Honorific Usages in Wikipedia and LLMs for Bengali and Hindi

Sourabrata Mukherjee, Atharva Mehta, Sougata Saha, Akhil Arora, Monojit Choudhury
EMNLP 2025

The obligatory use of third-person honorifics is a distinctive feature of several South Asian languages, encoding nuanced socio-pragmatic cues such as power, age, gender, fame, and social distance.In this work, (i) We present the first large-scale study of third-person honorific pronoun and verb usa

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting

Sagnik Mukherjee, Muhammad Farid Adilazuarda, Sunayana Sitaram, Kalika Bali, Alham Fikri Aji, Monojit Choudhury
EMNLP 2024main

Socio-demographic prompting is a commonly employed approach to study cultural biases in LLMs as well as for aligning models to certain cultures. In this paper, we systematically probe four LLMs (Llama 3, Mistral v0.2, GPT-3.5 Turbo and GPT4) with prompts that are conditioned on culturally sensitive…

Cited by 7Source ↗PDF ↗Scholar ↗
2024

Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language We Prompt Them in

Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, Monojit Choudhury
COLING 2024main

Ethical reasoning is a crucial skill for Large Language Models (LLMs). However, moral values are not universal, but rather influenced by language and culture. This paper explores how three prominent LLMs – GPT-4, ChatGPT, and Llama2Chat-70B – perform ethical reasoning in different languages and if t…

Cited by 20Source ↗PDF ↗Scholar ↗
2024

Evaluating Large Language Models for Health-related Queries with Presuppositions

Navreet Kaur, Monojit Choudhury, Danish Pruthi
ACL 2024findings

As corporations rush to integrate large language models (LLMs) it is critical that they provide factually accurate information, that is robust to any presuppositions that a user may express. In this work, we introduce UPHILL, a dataset consisting of health-related queries with varying degrees of pre…

Cited by 9Source ↗PDF ↗code ↗Scholar ↗
2024

INMT-Lite: Accelerating Low-Resource Language Data Collection via Offline Interactive Neural Machine Translation

Harshita Diddee, Anurag Shukla, Tanuja Ganu, Vivek Seshadri, Sandipan Dandapat, Monojit Choudhury +1
COLING 2024main

A steady increase in the performance of Massively Multilingual Models (MMLMs) has contributed to their rapidly increasing use in data collection pipelines. Interactive Neural Machine Translation (INMT) systems are one class of tools that can utilize MMLMs to promote such data collection in several u…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

The Zeno’s Paradox of ‘Low-Resource’ Languages

Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, Monojit Choudhury
EMNLP 2024main

The disparity in the languages commonly studied in Natural Language Processing (NLP) is typically reflected by referring to languages as low vs high-resourced. However, there is limited consensus on what exactly qualifies as a ‘low-resource language.’ To understand how NLP papers define and study ‘l…

Cited by 10Source ↗PDF ↗code ↗Scholar ↗
2024

Towards Measuring and Modeling “Culture” in LLMs: A Survey

Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O’Neill +2
EMNLP 2024main

We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs). We observe that none of the studies explicitly define “culture, which is a complex, multifaceted concept; instead, they probe the models on some specially design…

Cited by 65Source ↗PDF ↗code ↗Scholar ↗
2024

Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks

Abhinav Sukumar Rao, Atharva Roshan Naik, Sachin Vashistha, Somak Aditya, Monojit Choudhury
COLING 2024main

Recent explorations with commercial Large Language Models (LLMs) have shown that non-expert users can jailbreak LLMs by simply manipulating their prompts; resulting in degenerate output behavior, privacy and security breaches, offensive outputs, and violations of content regulator policies. Limited…

Cited by 85Source ↗PDF ↗code ↗Scholar ↗
2024

“They are uncultured”: Unveiling Covert Harms and Social Threats in LLM Generated Conversations

Preetam Prabhu Srikar Dammu, Hayoung Jung, Anjali Singh, Monojit Choudhury, Tanu Mitra
EMNLP 2024main

Large language models (LLMs) have emerged as an integral part of modern societies, powering user-facing applications such as personal assistants and enterprise applications like recruitment tools. Despite their utility, research indicates that LLMs perpetuate systemic biases. Yet, prior works on LLM…

Cited by 12Source ↗PDF ↗code ↗Scholar ↗
2023

Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs

Abhinav Sukumar Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, Monojit Choudhury
EMNLP 2023long findings

In this position paper, we argue that instead of morally aligning LLMs to specific set of ethical principles, we should infuse generic ethical reasoning capabilities into them so that they can handle value pluralism at a global scale. When provided with an ethical policy, an LLM should be capable of…

Cited by 0Source ↗Scholar ↗
2023

LLM-powered Data Augmentation for Enhanced Cross-lingual Performance

Chenxi Whitehouse, Monojit Choudhury, Alham Fikri Aji
EMNLP 2023long main

This paper explores the potential of leveraging Large Language Models (LLMs) for data augmentation in multilingual commonsense reasoning datasets where the available training data is extremely limited. To achieve this, we utilise several LLMs, namely Dolly-v2, StableVicuna, ChatGPT, and GPT-4, to au…

Cited by 0Source ↗code ↗Scholar ↗
2023

X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents

Mehrad Moradshahi, Tianhao Shen, Kalika Bali, Monojit Choudhury, Gael de Chalendar, Anmol Goel +13
ACL 2023findings

Task-oriented dialogue research has mainly focused on a few popular languages like English and Chinese, due to the high dataset creation cost for a new language. To reduce the cost, we apply manual editing to automatically translated data. We create a new multilingual benchmark, X-RiSAWOZ, by transl…

Cited by 9Source ↗PDF ↗code ↗Scholar ↗
2022

Global Readiness of Language Technology for Healthcare: What Would It Take to Combat the Next Pandemic?

Ishani Mondal, Kabir Ahuja, Mohit Jain, Jacki O’Neill, Kalika Bali, Monojit Choudhury
COLING 2022main

The COVID-19 pandemic has brought out both the best and worst of language technology (LT). On one hand, conversational agents for information dissemination and basic diagnosis have seen widespread use, and arguably, had an important role in fighting against the pandemic. On the other hand, it has al…

Cited by 7Source ↗PDF ↗Scholar ↗
2022

Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, Monojit Choudhury
ACL 2022long

Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the ex…

Cited by 22Source ↗PDF ↗code ↗Scholar ↗
2022

On the Calibration of Massively Multilingual Language Models

Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, Monojit Choudhury
EMNLP 2022main

Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. While there has been much work in evaluating these models for their performance on a variety of tasks and languages, little attention has been paid on how w…

Cited by 19Source ↗PDF ↗code ↗Scholar ↗
2022

On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data

Kabir Ahuja, Monojit Choudhury, Sandipan Dandapat
NAACL 2022long

Borrowing ideas from Production functions in micro-economics, in this paper we introduce a framework to systematically evaluate the performance and cost trade-offs between machine-translated and manually-created labelled data for task-specific fine-tuning of massively multilingual language models. W…

Cited by 1Source ↗PDF ↗code ↗Scholar ↗
2022

SyMCoM - Syntactic Measure of Code Mixing A Study Of English-Hindi Code-Mixing

Prashant Kodali, Anmol Goel, Monojit Choudhury, Manish Shrivastava, Ponnurangam Kumaraguru
ACL 2022findings

Code mixing is the linguistic phenomenon where bilingual speakers tend to switch between two or more languages in conversations. Recent work on code-mixing in computational settings has leveraged social media code mixed texts to train NLP models. For capturing the variety of code mixing in, and acro…

Cited by 8Source ↗PDF ↗Scholar ↗
2022

”Diversity and Uncertainty in Moderation” are the Key to Data Selection for Multilingual Few-shot Transfer

Shanu Kumar, Sandipan Dandapat, Monojit Choudhury
NAACL 2022findings

Few-shot transfer often shows substantial gain over zero-shot transfer (CITATION), which is a practically useful trade-off between fully supervised and unsupervised learning approaches for multilingual pretained model-based systems. This paper explores various strategies for selecting data for annot…

Cited by 6Source ↗PDF ↗Scholar ↗