MixSense : Mixture of Vision Sense

Abstract

It is a new trend to fine-tune Large Multimodal Models (LMMs) to adapt to specific visual tasks through task-related conversation data. This approach provides a new paradigm for solving various vision-language tasks, however, it still faces two problems: (1) the global visual features input to the backbone Large Language Model (LLM) lack the focus on task-specific information, which may lead a sub-optimal performance on specific visual tasks; (2) The hallucinations on specific tasks are underestimated, which makes the model prone to factual errors during inference. In order to solve these two problems, we propose MixSense, a new LMM paradigm that aligns the visual features of specilized models and general models to the backbone LLM together, increasing the focus on specific task information while maintaining the vanilla capabilities of the model; evaluated and fine-tuned with the constructed task-specific negative samples, we can further assess the degree of hallucinations of the model on specific visual tasks and reduce it. We validate our method on a popular Object Detection (OD) task, Referring Expression Comprehension (REC), and extensive experiments demonstrate the effectiveness of our proposal.