Spatio-Temporal Structure Consistency for Semi-Supervised Medical Image Classification

Abstract

Intelligent medical diagnosis has shown remarkable progress on the large-scale datasets with full annotations. However, very few labeled images are available due to significantly expensive annotations by experts. To efficiently leverage abundant unlabeled data, we propose a novel Spatio-Temporal Structure Consistent (STSC) learning framework to combine both spatial and temporal structure consistency. Specifically, a gram matrix is derived to capture the structural similarity among different training samples in the representation space. At the spatial level, our framework explicitly enforces the consistency of structural similarity among different samples under perturbations. At the temporal level, we desire to maintain the consistency of the structural similarity in different training iterations by digging out the stable sub-structures in a relation graph. Experiments on two medical image datasets (i.e., ISIC 2018 and ChestX-ray14) show that our method outperforms state-of-the-art Semi-Supervised Learning (SSL) methods. Furthermore, extensive qualitative analysis on the Gram matrices and heatmaps by Grad-CAM are presented to validate the effectiveness of our method.