Spatially-Aware Cross-Modal Contrastive Learning for Low-Shot HSI Classification

Abstract

Classifying hyperspectral images (HSI) with limited supervision is challenging due to their high dimensionality and complex spectral features, which frequently result in overfitting, especially under extremely low supervision. Existing self-supervised methods for HSI data focus predominantly on spectral attributes, neglecting the spatial details crucial for effective HSI classification. To address this, we introduce the Cross-Modal Spatial Contrastive (CM-SCON) framework, a novel self-supervised approach that employs co-registered, unlabeled HSI and LiDAR data. CM-SCON leverages LiDAR’s spatial context to enhance the spectral discriminability of the HSI encoder. Central to our method is a pair of cross-modal pretext tasks that merges cross-modal patch reconstruction with a contrastive learning objective, significantly boosting the HSI encoder’s effectiveness, which adeptly handles downstream land-cover classification tasks, even with minimal labeled data. Extensive evaluations on the Houston-13, 18, and Trento benchmark datasets show that CM-SCON outperforms existing baselines for within-dataset and cross-dataset evaluation scenarios.