Enhancing 3D Medical Image Understanding with 2D Multimodal Large Language Models

Abstract

Understanding medical image volumes is crucial in healthcare, yet most current models for classification and segmentation often focus narrowly on task-specific features without capturing the broader medical context. To address this, we introduce Med3DInsight, a pre-training framework that enhances 3D image understanding by leveraging 2D multimodal large language models (MLLMs) through a Plane-Slice-Aware Transformer (PSAT) module. Med3DInsight connects 3D image encoders with 2D MLLMs, enhancing representation learning for downstream tasks. Extensive experiments on CT and MRI datasets demonstrate that Med3DInsight achieves state-of-the-art performance, surpassing 19 baseline methods and proving effective across diverse imaging modalities and anatomical structures. This framework can be seamlessly integrated into existing 3D medical imaging networks, significantly boosting their performance and adaptability. Our source code is publicly available at https://github.com/Qybc/Med3DInsight.