Uncertainty Estimation of Transformer Predictions for Misclassification Detection
Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko
Abstract
Uncertainty estimation (UE) of model predictions is a crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, out-of-distribution detection, etc. Most of the works on modeling the uncertainty of deep neural networks evaluate these methods on image classification tasks. Little attention has been paid to UE in natural language processing. To fill this gap, we perform a vast empirical investigation of state-of-the-art UE methods for Transformer models on misclassification detection in named entity recognition and text classification tasks and propose two computationally efficient modifications, one of which approaches or even outperforms computationally intensive methods.
BibTeX
@inproceedings{vazhentsev-etal-2022-uncertainty,
title = "Uncertainty Estimation of Transformer Predictions for Misclassification Detection",
author = "Vazhentsev, Artem and
Kuzmin, Gleb and
Shelmanov, Artem and
Tsvigun, Akim and
Tsymbalov, Evgenii and
Fedyanin, Kirill and
Panov, Maxim and
Panchenko, Alexander and
Gusev, Gleb and
Burtsev, Mikhail and
Avetisian, Manvel and
Zhukov, Leonid",
editor = "Muresan, Smaranda and
Nakov, Preslav and
Villavicencio, Aline",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-long.566/",
doi = "10.18653/v1/2022.acl-long.566",
pages = "8237--8252"
}