SSL-RGB2IR: Semi-supervised RGB-to-IR Image-to-Image Translation for Enhancing Visual Task Training in Semantic Segmentation and Object Detection

Abstract

The scarcity of annotated infrared (IR) image datasets limits deep learning networks from achieving performances comparable to those achieved with RGB data. To address this, we introduce a novel semi-supervised RGB-to-IR Image-to-Image Translation model (SSL-RGB2IR) that generates synthetic IR data from RGB images. Our model effectively preserves the IR characteristics in the generated images from both synthetic and real-world data. Compared to existing image-to-image translation techniques, training models on this generated IR data significantly improves performance in downstream tasks like segmentation and detection. Notably, in sim-to-real transfer, the segmentation model trained on SSL-RGB2IR generated IR images outperforms baselines and other Image-to-Image (I2I) models. Furthermore, for real-world applications utilizing EO/IR fusion images, this approach solves the well-known challenge of co-registering EO and IR images, which often have inherent misalignment’s due to differing sensor characteristics. Our code is available at https://github.com/prahlad-anand/ssl-rgb2ir https://github.com/prahlad-anand/ssl-rgb2ir.