OVSG-SLAM: Open-Vocabulary Semantic Gaussian Splatting SLAM

Abstract

Most conventional semantic SLAM approaches concentrate on maintaining 3D semantic consistency while overlooking their reliance on predefined semantic categories, ultimately limiting flexibility in scene understanding. We propose Open-Vocabulary Semantic Gaussian Splatting SLAM (OVSG-SLAM), an approach that integrates multi-modal perception and 3D Gaussian splatting into a semantic SLAM framework. By combining the advantages of Segment Anything (SAM) for open-vocabulary 2D scene understanding with the powerful feature extraction capabilities of vision-language models, our method eliminates the reliance on predefined closed-set categories. Although Vision-Language Models (VLMs) provide open-vocabulary reasoning, integrating them with 3D semantic SLAM poses challenges such as embedding ambiguity and computational overhead. To address these challenges, we present a feature embedding strategy called differentiable identity-aware encoding, which reduces computational cost while ensuring accurate semantic mapping. Furthermore, instead of using a traditional semantic loss, we optimize the scene representation through an identity loss. Extensive experimental evaluations on the Replica and ScanNet datasets demonstrate that the proposed method achieves state-of-the-art performance in mapping, tracking and 3D semantic segmentation tasks.