ACL 2022long92 citations
Adaptive Testing and Debugging of NLP Models
Marco Tulio Ribeiro, Scott Lundberg
Abstract
Current approaches to testing and debugging NLP models rely on highly variable human creativity and extensive labor, or only work for a very restrictive class of bugs. We present AdaTest, a process which uses large scale language models (LMs) in partnership with human feedback to automatically write unit tests highlighting bugs in a target model. Such bugs are then addressed through an iterative text-fix-retest loop, inspired by traditional software development. In experiments with expert and non-expert users and commercial / research models for 8 different tasks, AdaTest makes users 5-10x more effective at finding bugs than current approaches, and helps users effectively fix bugs without adding new bugs.
BibTeX
@inproceedings{ribeiro-lundberg-2022-adaptive,
title = "Adaptive Testing and Debugging of {NLP} Models",
author = "Ribeiro, Marco Tulio and
Lundberg, Scott",
editor = "Muresan, Smaranda and
Nakov, Preslav and
Villavicencio, Aline",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-long.230/",
doi = "10.18653/v1/2022.acl-long.230",
pages = "3253--3267"
}