Self-supervised Hyperspectral and Multispectral Fusion via Deep Low-Rank Prior and Learnable Degradation Networks

Abstract

Model-based shallow machine-learning methods and data-driven deep-learning (DL) methods have been advanced to address hyperspectral and multispectral image fusion (HS–MS fusion). Nonetheless, model-based approaches, which meticulously craft regularization terms within optimization models using hand-engineered priors, often struggle to pinpoint the optimal solution efficiently. DL-based methods, which train on extensive datasets to learn a non-linear mapping for generating a high spatial and spectral resolution image (HS<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>I), exhibit limited generalization capabilities when applied to novel and diverse test datasets. To improve the generalization ability and optimization efficiency of the existing HS–MS fusion methods, a novel deep low-rank prior (DLRP)-based self-supervised HS–MS fusion approach is devised. It incorporates a low-rank learning paradigm to produce the fused HS<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>I, subject to the constraints imposed by loss functions. Instead of deriving the solution via solving a low-rank approximation optimization problem, deep image prior (DIP) learned by a two-dimensional CNN and a one-dimensional CNN are integrated as the low-rank prior.