Mahir Shah — accepted AI-conference papers

2025

MixEval-X: Any-to-any Evaluations from Real-world Data Mixture

Jinjie Ni, Yifan Song, Deepanway Ghosal, Bo Li, David Junhao Zhang, Xiang Yue +8

ICLR 2025spotlight

Perceiving and generating diverse modalities are crucial for AI models to effectively learn from and engage with real-world signals, necessitating reliable evaluations for their development. We identify two major issues in current evaluations: (1) inconsistent standards, shaped by different communit…

Cited by 3 Source ↗PDF ↗Scholar ↗

2024

MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain +2

NeurIPS 2024poster

Evaluating large language models (LLMs) is challenging. Traditional ground-truth- based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated ov…

Cited by 38 Source ↗PDF ↗Scholar ↗