White-box Testing of NLP models with Mask Neuron Coverage
Arshdeep Sekhon, Yangfeng Ji, Matthew Dwyer, Yanjun Qi
Abstract
Recent literature has seen growing interest in using black-box strategies like for testing the behavior of NLP models. Research on white-box testing has developed a number of methods for evaluatinghow thoroughly the internal behavior of deep models is tested, but they are not applicableto NLP models. We propose a set of white-box testing methods that are customized for transformer-based NLP models. These include MASK NEURON COVERAGE (MNCOVER) that measures how thoroughlythe attention layers in models are exercised during testing. We show that MNCOVER can refine testing suites generated by CheckList by substantiallyreduce them in size, for more than 60% on average, while retaining failing tests – thereby concentrating the faultdetection power of the test suite. Further we show how can be used to guide CheckList input generation,evaluate alternative NLP testing methods, and drive data augmentation to improve accuracy.
BibTeX
@inproceedings{sekhon-etal-2022-white,
title = "White-box Testing of {NLP} models with Mask Neuron Coverage",
author = "Sekhon, Arshdeep and
Ji, Yangfeng and
Dwyer, Matthew and
Qi, Yanjun",
editor = "Carpuat, Marine and
de Marneffe, Marie-Catherine and
Meza Ruiz, Ivan Vladimir",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-naacl.116/",
doi = "10.18653/v1/2022.findings-naacl.116",
pages = "1547--1558"
}