A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference
Abstract
Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility.First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations.We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods.Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, to support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.
BibTeX
@inproceedings{zaman-belinkov-2022-multilingual,
title = "A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference",
author = "Zaman, Kerem and
Belinkov, Yonatan",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.101/",
doi = "10.18653/v1/2022.emnlp-main.101",
pages = "1556--1576"
}