MT-Fusion: Multi-Task Learning for Degradation-Aware Infrared and Visible Image Fusion

Abstract

The effective fusion of infrared and visible images could enhance environment perception during robot rescue mission by combining complementary information from both sensors. However, most existing fusion methods are developed for images captured under normal conditions, which limits their performance in real-world rescue scenarios where images often suffer from diverse degradation such as haze, low-light, noise, low-contrast, and so on. To address this challenge, we propose MT-Fusion, a novel deep learning workflow based on multi-task learning (MTL) that targets multiple specific degradation scenarios. MT-Fusion incorporates specific encoder modules for processing various degraded images, a degradation attention mechanism for fusion, and a shared decoder for image reconstruction. Extensive experiments demonstrate that our proposed specific scenario-guided image fusion strategy has obvious advantages in robot perception in the image fusion performance and degradation treatment. For the inference stage, we propose a selector that could automatically categorize the degradation of input and activate the specific encoder. The MT-Fusion also provides a more practical solution for enhancing robot rescue operations in challenging environments.