2022
k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations
ACL 2022short
Since the inception of crowdsourcing, aggregation has been a common strategy for dealing with unreliable data. Aggregate ratings are more reliable than individual ones. However, many Natural Language Processing (NLP) applications that rely on aggregate ratings only report the reliability of individu…