Modality Modulation and Dual Consistency for Multi-Modality Semi-Supervised Medical Image Segmentation

Abstract

Multi-modality (MM) semi-supervised learning (SSL) based medical image segmentation has recently gained increasing attention due to its ability to utilize MM data and low dependency on labeled images. However, current MM-SSL methods face two major challenges: (1) Complex network designs make it difficult to apply these methods to scenarios involving more than two modalities. (2) The use of generative methods to leverage unlabeled data may not be reliable for SSL learning. To address these challenges, we propose Modality Modulation Dual Consistency, dubbed MM-DC. Specifically, we design a modality all-in-one network to process data from all modalities, with learnable plug-in Modality Modulation Layers (MML) to gradually modulate features from different modalities into a modality-invariant feature space, enabling unified segmentation. Additionally, we propose a dual-consistency strategy that enforces consistency at both the image and feature levels, which eliminates the requirements for generative methods. Extensive experiments demonstrate that MM-DC outperforms other state-of-the-art methods on open-source datasets with 2- and 4-modalities. The code is available<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.