Multi-scale Feature Interaction and Adaptive Experts for Panoptic Segmentation in Remote Sensing Images

Abstract

Panoptic segmentation unifies the traditional tasks of instance and semantic segmentation. It plays a crucial role in the field of remote sensing; however, it encounters challenges in recognizing small objects and in the model’s ability to generalize across complex scenes. In this paper, we introduce the MFIAE framework to address two specific challenges: a multi-scale interactive attention fusion (MSIAF) module and an adaptive disturbance sparse mixture-of-experts (ADSMoE) module based on Transformer. The MSIAF module is designed to fully utilize the rich contextual information captured by low-resolution features while simultaneously utilizing the advantages of high-resolution features to enhance small object segmentation. In the ADSMoE module, adaptive noise is introduced to disturb the expert selection in order to enhance the randomness and exploration of the model when it comes to selecting experts, thereby improving its capacity for generalization and robustness. Additionally, it also reduces the computational overhead and the model complexity. The experimental results demonstrate that our approach achieves state-of-the-art performance on the BSB Aerial dataset.