Assessing Robustness of Multi-Modal Large Language Models in Image Classification through Hierarchical WordNet-Based Evaluation

Abstract

The advancement of multi-modal large language models (MLLMs) has significantly enhanced their capability to process and understand diverse data types, integrating text, images, and other modalities. Despite their impressive performance, evaluating the robustness of these models remains challenging due to the difficulty of aligning their text-based responses with image classification labels. Traditional approaches rely on CLIP scores or other large language models as judges, but these methods lack scientific rigor and fail to capture robustness across different semantic levels. In this paper, we propose a novel evaluation metric that systematically assesses the robustness of MLLMs in image classification using WordNet’s hierarchical structure. Specifically, we parse the text descriptions generated by MLLMs to extract all nouns, then calculate their distances to the groundtruth label in WordNet as a similarity metric. The minimum distance among all nouns is used as the similarity score between the text description and the label. By using WordNet, we can also evaluate classification performance at different semantic levels. Through extensive experiments, we demonstrate that our WordNet-based evaluation metric offers a deeper understanding of MLLMs’ robustness, paving the way for more resilient and reliable models in real-world applications.