Joint Coupled Transform Learning Framework for Multimodal Image Super-Resolution

Abstract

Insights from multiple imaging modalities have recently been applied in solving many computer vision related applications. In this paper, we model the cross-modal dependencies between different modalities for Multimodal Image Super-Resolution (MISR), i.e., enhance the Low Resolution (LR) image of target modality with the guidance of a High Resolution (HR) image from another modality. We introduce a joint optimization based transform learning frame-work referred to as Joint Coupled Transform Learning (JCTL) to combine the information from multiple modalities to generate the HR image of the target modality. All the necessary intermediate steps and the corresponding closed form solution updates are pro-vided. The performance of the proposed JCTL is benchmarked against the state-of-the-art MISR approaches on different multi-modal datasets with different upscaling factors. The results show better performance with the proposed JCTL approach compared to other state-of-the-art techniques both in terms of PSNR and SSIM.