2025
Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation
ICML 2025oral
In this position paper, we observe that empirical evaluation in Generative AI is at a crisis point since traditional ML evaluation and benchmarking strategies are insufficient to meet the needs of evaluating modern GenAI models and systems. There are many reasons for this, including the fact that th…