ArogyaSutra: A Multi-Agent Framework for Multimodal Medical Reasoning in Indic Languages

Abstract

Multimodal Large Language Models(MLLMs) have shown promising reasoning capabilities in general domains, yet their performance remains limited in specialized settings such as healthcare, particularly for multilingual and low-resource scenarios. This gap is critical in regions like rural India, where patients often express complex medical queries in native Indic languages and rely on multimodal inputs such as medical images. Existing MLLMs, predominantly trained on English-centric data, struggle to support such use cases, limiting equitable access to AI-driven healthcare assistance. To address this challenge, we construct a large-scale multilingual multimodal medical question–answer dataset named \textbf{ArogyaBodha} from eight heterogeneous sources, covering 31 body systems, six imaging modalities, and 21 clinical domains across English and seven major Indian languages. We further propose \textbf{\textit{ArogyaSutra}}, an actor–critic–based multi-agent framework that combines tool grounding with dual-memory mechanisms to support step-wise, reasoning-aware decision making while explicitly retaining past mistakes to prevent their repeated occurrence. The Actor predicts the answer to the multimodal query from visual and memory states, whereas the Critic evaluates actor outcomes and delivers corrective feedback, enabling iterative refinement of the reasoning process. Experiments show that our dataset and framework improve the multilingual medical reasoning accuracy of an MLLM across all Indic languages, with ablation studies validating the effectiveness of each component. Our work supports UN SDGs~3,~4, and~10 by enabling reliable multilingual medical decision support, reducing healthcare inequities, and strengthening inclusive clinical education for underserved communities.