2024
Finding Replicable Human Evaluations via Stable Ranking Probability
NAACL 2024long
Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, b…