Two-Stage Transfer Learning for Fusion and Classification of Airborne Hyperspectral Imagery

Abstract

In this work, we introduce a novel fusion and training strategy aimed at facilitating transfer learning to enhance classification in hyperspectral airborne imagery. Our training strategy has two stages: first we train separate convolutional neural network (CNN) models, one for the bands for which pretraining is available (e.g. visual bands), and a second model trained from scratch on all available wavelengths. These models are then integrated into a new fully-connected layer, which is fine-tuned to fuse the features from both modalities. We use the BigEarthNet and EuroSAT datasets, containing Sentinel-2 satellite imagery in visual and infrared wavelengths. Our approach provides significant performance improvements across all evaluation metrics on the aforementioned data sets, exemplifying the efficacy of our two-stage transfer learning strategy in handling multi-modal data.