2026
Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
ICML 2026poster
While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce…