UniIVFT: Towards a Unified Framework for Infrared-Visible Fusion and Translation

Abstract

Infrared-visible image fusion (IVF) and infrared-to-visible image translation (I2V) are two closely related tasks in multimodal image processing, both aimed at combining or transforming infrared and visible modalities to enhance image information content. Existing methods typically focus on either fusion or translation, often requiring redundant construction of similar components for each task, which limits the effective utilization of cross-modal interactions and feature encoding capabilities. Furthermore, these approaches are often hindered by their reliance on complex feature extract models, limiting their overall effectiveness and adaptability. In this paper, we introduce the Unified Multimodal Infrared-Visible Image Fusion and Translation (UniIVFT) framework, which integrates both fusion and translation tasks within a single architecture. We employ a vision transformer (ViT) encoder-decoder structure augmented with task-specific tokens and introduce a contrastive loss to effectively align infrared and visible image features before multimodal encoding. This alignment enhances the encoder’s ability to capture cross-modal interactions. In UniIVFT, both IVF and I2V tasks share a unified encoder architecture and use task-specific tokens to control model outputs, reducing redundant model construction and training. Extensive experiments demonstrate that UniIVFT achieves performance on par with that of SOTAs across multiple tasks while maintaining a lightweight architecture with fewer model parameters.