But Are You Sure? An Uncertainty-Aware Perspective on Explainable AI
Charles Marx, Youngsuk Park, Hilaf Hasson, Yuyang Wang, Stefano Ermon, Luke Huan
Abstract
Although black-box models can accurately predict outcomes such as weather patterns, they often lack transparency, making it challenging to extract meaningful insights (such as which atmospheric conditions signal future rainfall). Model explanations attempt to identify the essential features of a model, but these explanations can be inconsistent: two near-optimal models may admit vastly different explanations. In this paper, we propose a solution to this problem by constructing uncertainty sets for explanations of the optimal model(s) in both frequentist and Bayesian settings. Our uncertainty sets are guaranteed to include the explanation of the optimal model with high probability, even though this model is unknown. We demonstrate the effectiveness of our approach in both synthetic and real-world experiments, illustrating how our uncertainty sets can be used to calibrate trust in model explanations.
BibTeX
@InProceedings{pmlr-v206-marx23a,
title = {But Are You Sure? An Uncertainty-Aware Perspective on Explainable AI},
author = {Marx, Charles and Park, Youngsuk and Hasson, Hilaf and Wang, Yuyang and Ermon, Stefano and Huan, Luke},
booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics},
pages = {7375--7391},
year = {2023},
editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem},
volume = {206},
series = {Proceedings of Machine Learning Research},
month = {25--27 Apr},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v206/marx23a/marx23a.pdf},
url = {https://proceedings.mlr.press/v206/marx23a.html},
abstract = {Although black-box models can accurately predict outcomes such as weather patterns, they often lack transparency, making it challenging to extract meaningful insights (such as which atmospheric conditions signal future rainfall). Model explanations attempt to identify the essential features of a model, but these explanations can be inconsistent: two near-optimal models may admit vastly different explanations. In this paper, we propose a solution to this problem by constructing uncertainty sets for explanations of the optimal model(s) in both frequentist and Bayesian settings. Our uncertainty sets are guaranteed to include the explanation of the optimal model with high probability, even though this model is unknown. We demonstrate the effectiveness of our approach in both synthetic and real-world experiments, illustrating how our uncertainty sets can be used to calibrate trust in model explanations.}
}