Rishabh Maheshwary — accepted AI-conference papers

2026

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam, Arshia Soltani Moakhar, Shivalika Singh, Fabian Farestam +39

ICLR 2026poster

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and language, many rely on translations of English datasets, failing t…

Cited by 0 Source ↗code ↗Scholar ↗

2025

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary +51

ICLR 2025spotlight

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., mult…

Cited by 9 Source ↗PDF ↗Scholar ↗

2025

M-RewardBench: Evaluating Reward Models in Multilingual Settings

Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata +4

ACL 2025long

Reward models (RMs) have driven the state-of-the-art performance of LLMs today by enabling the integration of human feedback into the language modeling process. However, RMs are primarily trained and evaluated in English, and their capabilities in multilingual settings remain largely understudied. I…

Cited by 0 Source ↗PDF ↗code ↗Scholar ↗

2025

M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Models

Rishabh Maheshwary, Vikas Yadav, Hoang H Nguyen, Khyati Mahajan, Sathwik Tejaswi Madhusudhan

NAACL 2025long

Collecting instruction fine-tuning (IFT) data is a resource and time intensive task especially in multilingual setting where finding proficient native speakers is challenging. Moreover, traditional data collection is prone to privacy risks, toxicity and lacks scalability. While, fully synthetic data…

Cited by 5 Source ↗PDF ↗code ↗Scholar ↗

2025

Prompting with Phonemes: Enhancing LLMs’ Multilinguality for Non-Latin Script Languages

Hoang H Nguyen, Khyati Mahajan, Vikas Yadav, Julian Salazar, Philip S. Yu, Masoud Hashemi +1

NAACL 2025long

Multilingual LLMs have achieved remarkable benchmark performance, but we find they continue to underperform on non-Latin script languages across contemporary LLM families. This discrepancy arises from the fact that LLMs are pretrained with orthographic scripts, which are dominated by Latin character…

Cited by 0 Source ↗PDF ↗Scholar ↗

2025

Variable Layerwise Quantization: A Simple and Effective Approach to Quantize LLMs

Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul Ioan Clotan, Sathwik Tejaswi Madhusudhan, Mihai Surdeanu

ACL 2025finding

We present a simple meta quantization approach that quantizes different layers of a large language model (LLM) at different bit levels, and is independent of the underlying quantization technique. Specifically, we quantize the most important layers to higher bit precision and less important layers t…

Cited by 0 Source ↗PDF ↗code ↗Scholar ↗

2024

Enhancing Alignment using Curriculum Learning & Ranked Preferences

Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, Sathwik Tejaswi Madhusudhan

EMNLP 2024finding

Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data (one chosen and rejected response per prompt) to align LLMs to human preferences. In practice, multiple responses could exist for a given prompt with varying quality relative to each other. We prop…

Cited by 4 Source ↗PDF ↗Scholar ↗

2023

Improving Selective Visual Question Answering by Learning From Your Peers

Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen +2

CVPR 2023poster

Despite advances in Visual Question Answering (VQA), the ability of models to assess their own correctness remains underexplored. Recent work has shown that VQA models, out-of-the-box, can have difficulties abstaining from answering when they are wrong. The option to abstain, also called Selective P…

Cited by 20 Source ↗PDF ↗code ↗Scholar ↗

2022

Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers

Vivek Kumar, Rishabh Maheshwary, Vikram Pudi

NAACL 2022long

Existing Math Word Problem (MWP) solvers have achieved high accuracy on benchmark datasets. However, prior works have shown that such solvers do not generalize well and rely on superficial cues to achieve high performance. In this paper, we first conduct experiments to showcase that this behaviour i…

Cited by 13 Source ↗PDF ↗code ↗Scholar ↗

2021

A Strong Baseline for Query Efficient Attacks in a Black Box Setting

Rishabh Maheshwary, Saket Maheshwary, Vikram Pudi

EMNLP 2021main

Existing black box search methods have achieved high success rate in generating adversarial attacks against NLP models. However, such search methods are inefficient as they do not consider the amount of queries required to generate adversarial attacks. Also, prior attacks do not maintain a consisten…

Cited by 40 Source ↗PDF ↗code ↗Scholar ↗

2021

Adversarial Examples for Evaluating Math Word Problem Solvers

Vivek Kumar, Rishabh Maheshwary, Vikram Pudi

EMNLP 2021finding

Standard accuracy metrics have shown that Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. However, the extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. In this paper, we generate adversarial attacks t…

Cited by 30 Source ↗PDF ↗code ↗Scholar ↗

2021

Generating Natural Language Attacks in a Hard Label Black Box Setting

Rishabh Maheshwary, Saket Maheshwary, Vikram Pudi

AAAI 2021technical

We study an important and challenging task of attacking natural language processing models in a hard label black box setting. We propose a decision-based attack strategy that crafts high quality adversarial examples on text classification and entailment tasks. Our proposed attack strategy leverages…

Cited by 120 Source ↗PDF ↗code ↗Scholar ↗