Lucy Cheke

Not a Number: Identifying Instance Features for Capability-Oriented Evaluation

In AI evaluation, performance is often calculated by averaging across various instances. But to fully understand the capabilities of an AI system, we need to understand the factors that cause its pattern of success and failure. In this paper, we present a new methodology to identify and build inform…