Improving Multimodal Large Language Models through Combining Resampler and MLP Projections

Abstract

Current multimodal large language models (LLMs) achieve impressive performance through connecting visual encoders and LLMs by resampler or MLP projections. Although the MLP projections are effective and widely used in recent works when compared to resampler projections, it has the flaw of the inability to change the number of tokens, which limits the application scenarios. In this paper, we propose a new projection structure that inherits the efficiency of traditional MLP projections and has the ability to flexibly change the number of tokens. Furthermore, we propose a new multimodal LLM with the proposed projection applied. To validate the performance, we evaluate the proposed model on the first large-scale multimodal science question dataset ScienceQA, and the first multimodal LLM evaluation benchmark MME. Our model achieves a new state-of-the-art (SoTA) result of 94.27% on ScienceQA, which is higher than the previous SoTA result of 92.53% achieved by the LLaVA+GPT-4 judge model. On the MME benchmark, our model achieves better performance when compared to representative chatbot models BLIP-2 and InstructBLIP, with significantly lower training costs.