Exploring the Domain-Invariant Flow Representation in Vision-Based Tactile Sensors for Omni-Hardness Perception

Abstract

Vision-based tactile sensors have recently gained prominence due to their superior resolution and ability to capture multi-dimensional contact information. However, even when sensors share the same sensing principle, variations in production factors can lead to differences in the color patterns of tactile signals. Unlike common vision tasks, vision-based tactile perception depends on tracking light variation in colorful signals, making it more susceptible to lighting conditions and thus more prone to domain gaps. In this paper, we propose an Omni-hardness perception framework that enables adaptation across various vision-based tactile sensors. Firstly, in-depth analyses of the factors influencing the generalization of hardness perception are presented. Furthermore, the light balance module and the force scale module are coupled to regulate network learning of generalized representations. Experimental results across multiple sensors demonstrate the transferability of learned representations. Additionally, downstream tasks in natural object perception, tumor detection, and grasping stability prediction, are proposed to evaluate the potential applications. The framework's performance shows promise for advancing general tactile sensing and embodied tactile perception.