When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty
Fernando Martínez-Plumed, David Castellano, Carlos Monserrat-Aranda, José Hernández-Orallo
Abstract
One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications.
BibTeX
@inproceedings{aaai2022_whenaidifficulty,
title = {When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty},
author = {Fernando Martínez-Plumed and David Castellano and Carlos Monserrat-Aranda and José Hernández-Orallo},
booktitle = {AAAI 2022},
year = {2022}
}