Exploring the Role of CLIP Global Visual Features in Multimodal Large Language Models

Abstract

The next recognized development direction of large language models (LLMs) is to integrate and enhance multimodal capability. Although current multimodal large language models (MLLMs) have achieved impressive performance by combining the pre-trained visual encoder CLIP and LLM, these works mainly focus on using the CLIP patch visual features. In practice, we find that CLIP global visual features are more efficient than patch visual features in some scenarios, especially in multimodal reasoning tasks. Therefore, we explore the role of CLIP global visual features and propose a new MLLM with the usage of full global visual features in this paper. Our model adopts the parameter-efficient transfer learning (PETL) method Adapter to fine-tune the pre-trained models and a simple MLP-based network to connect the visual encoder and LLM. To validate the performance, we evaluate our model on the first large-scale multimodal science question dataset, ScienceQA. Our model achieves a new state-of-the-art (SoTA) result of 93.96% on ScienceQA, which is higher than the previous SoTA result of 92.53%.