Multi-Modality Test-Time Adaptation for Semantic Segmentation in Robotic Perception

Abstract

Test-Time Adaptation (TTA) adjusts pre-trained models in unlabeled unseen environments during the test phase, making it more practical for robotic applications. However, the constant changes of the physical world create significant domain gaps between the received data during robot deployment and the source data used for training. In addition, existing methods mainly focus on a single modality, e.g., RGB images, limiting the application of these methods in multi-modality input scenarios. In this work, we propose a Deep Multi-modality Aggregation Test-time Adaptation (DMATA) method to address the above mentioned issues. To prevent the domain shifts from disrupting the adaptation process, we first propose a Momentum-based Teacher-Student (MTS) framework. Since the teacher model and the student model contain complementary information, we design an Uncertainty-Guide (UG) feature fusion block to fuse the representation of the teacher model and student model of each modality. Finally, we introduce a 3D-Guide-2D (3G2) feature fusion block to leverage spatial information for enhancing 2D feature representation. Extensive experiments across three scenarios, including sensor-to-sensor, day-to-night, and city-to-city, demonstrate the effectiveness of our method in TTA multi-modality semantic segmentation tasks. Notably, under the scenario of sensor-to-sensor adaptation, our proposed DMATA obtains an <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$m$</tex> IoU of 54.2%, which is superior to the state-of-the-art test-time adaptation method.