A MEDICAL MULTIMODAL DIAGNOSTIC FRAMEWORK INTEGRATING VISION-LANGUAGE MODELS AND LOGIC TREE REASONING
Zelin Zang, Dan Yang, Siqi Ma, Zhu Zhang, Wing-Kuen Ling, Yue Shen, Fuji Yang
Abstract
With the rapid growth of large language models (LLMs) and vision-language models (VLMs) in medicine, simply integrating clinical text and medical imaging does not guarantee reliable reasoning. Existing multimodal models often produce hallucinations or inconsistent chains of thought, limiting clinical trust. We propose a diagnostic framework built upon LLaVA that combines vision-language alignment with logic-regularized reasoning. The system includes an input encoder for text and images, a projection module for cross-modal alignment, a reasoning controller that decomposes diagnostic tasks into steps, and a logic tree generator that assembles stepwise premises into verifiable conclusions. Evaluations on MedXpertQA and other benchmarks show that our method improves diagnostic accuracy and yields more interpretable reasoning traces on multimodal tasks, while remaining competitive on text-only settings. These results suggest a promising step toward trustworthy multimodal medical AI.
BibTeX
@inproceedings{icassp2026_amedicalmultimod,
title = {A MEDICAL MULTIMODAL DIAGNOSTIC FRAMEWORK INTEGRATING VISION-LANGUAGE MODELS AND LOGIC TREE REASONING},
author = {Zelin Zang and Dan Yang and Siqi Ma and Zhu Zhang and Wing-Kuen Ling and Yue Shen and Fuji Yang},
booktitle = {ICASSP 2026},
year = {2026}
}