Sensing Differently: Unifying Vision, Language, Posture and Tactile in Robotic Perception

Abstract

Multi-modal fusion perception enhances robotic performance in complex tasks by providing more comprehensive information than single modality. While tactile and proprioceptive sensing are effective for direct contact tasks like grasping, current research mainly focuses on vision-language fusion, neglecting other embodied modalities. The primary challenges of this limitation are the difficulty in generating natural language labels for embodied information like tactile and proprioception and aligning them with vision and language. To address this, we introduce VLaPT, a novel multi-modal grasping dataset that aligns vision and language (VL) with posture and tactile (PT), enabling robots to sense differently from environment to self. VLaPT includes 75 objects, 1,533 grasps, and over 78K synchronized vision-language-posturetactile pairs. The dataset incorporates structured, rich-text descriptions generated using modality-level language annotation templates, ensuring effective cross-modality alignment. Leveraging this dataset, we trained a lightweight multi-modal alignment framework, CLIP-ME, which enhances the performance of several downstream tasks with only a 5% increase in parameters. The VLaPT is publicly available in https://huggingface.co/datasets/xsdfasfgsa/VLaPT.