Optimizing Multimodal Image Fusion: A Novel Approach with Nystrom Attention Mechanisms in Transformer Models

Abstract

This work proposed a new model based on transformers for multimodal image fusion, with explicit attention paid to fusing infrared and visible images toward enhanced detail and information content. This method, which incorporates Nystrom attention and skip connections, optimizes the fusion process so that important features of both modalities are preserved. Traditional models find it hard to keep details in the content and introduce artifacts. On the contrary, this model can efficiently cope with these challenges, as it makes use of an attention mechanism that is computationally highly effective. This will efficiently fuse large-scale image data and may be useful in those fields where very detailed and comprehensive visual information is demanded. The detailed evaluations prove that our model has superior performance over the current state-of-the-art methods in texture detail preservation, thermal differentiation, and overall image quality, which possesses great potential for advanced imaging applications.