OpenVNA: A Framework for Analyzing the Behavior of Multimodal Language Understanding System under Noisy Scenarios
Ziqi Yuan, Baozheng Zhang, Hua Xu, Zhiyun Liang, Kai Gao
Abstract
We present OpenVNA, an open-source framework designed for analyzing the behavior of multimodal language understanding systems under noisy conditions. OpenVNA serves as an intuitive toolkit tailored for researchers, facilitating convenience batch-level robustness evaluation and on-the-fly instance-level demonstration. It primarily features a benchmark Python library for assessing global model robustness, offering high flexibility and extensibility, thereby enabling customization with user-defined noise types and models. Additionally, a GUI-based interface has been developed to intuitively analyze local model behavior. In this paper, we delineate the design principles and utilization of the created library and GUI-based web platform. Currently, OpenVNA is publicly accessible at https://github.com/thuiar/OpenVNA, with a demonstration video available at https://youtu.be/0Z9cW7RGct4.
BibTeX
@inproceedings{yuan-etal-2024-openvna,
title = "{O}pen{VNA}: A Framework for Analyzing the Behavior of Multimodal Language Understanding System under Noisy Scenarios",
author = "Yuan, Ziqi and
Zhang, Baozheng and
Xu, Hua and
Liang, Zhiyun and
Gao, Kai",
editor = "Cao, Yixin and
Feng, Yang and
Xiong, Deyi",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-demos.2/",
doi = "10.18653/v1/2024.acl-demos.2",
pages = "9--18"
}