VLIMNet: A Visible Light And Infrared Image Matching Network Based On Segment Anything Model And SuperPoint

Abstract

This paper introduces a novel method for matching visible light and infrared images, termed the Visible Light and Infrared Image Matching Network (VLIMNet). In the image encoding stage, we incorporate a generative architecture-based modality transformation network after the SuperPoint encoder, enabling the local geometric features extracted from infrared images to more closely resemble those of visible light images. This reduces the impact of modality differences. Simultaneously, we utilize the image encoder of the Segment Anything Model to obtain global semantic descriptors. During the decoding stage, we fuse the global semantic descriptors with the local geometric descriptors and perform joint decoding to obtain keypoints and new feature vectors. The matching process is computed by solving a differentiable optimal transport problem based on the LightGlue network, a graph neural network with an attention mechanism. Compared to other matching models, our approach demonstrates improvements across various metrics in the domain of infrared and visible light image matching, particularly excelling in matching images with significant pose differences. Specifically, our method achieves approximately a 5% improvement in matching accuracy compared to the highest accuracy matching methods.