Tool-Mediated Robot Perception of Granular Substances Using Multiple Sensory Modalities

Abstract

People use tools to interact with and perceive the world, with multimodal sensory inputs forming the basis of how we understand our environment. For example, a blind person uses a walking cane to tap the road and detect obstacles, and a builder uses a hammer to strike a wall to assess its structural integrity. Using tools extends our sensory capabilities during exploratory behaviors, enabling us to perceive object properties that are otherwise inaccessible. Inspired by this cognitive process, we propose a framework in which a multisensory robot employs exploratory behaviors using various tools to recognize granular substances. Our framework effectively integrates multiple non-visual sensory inputs (e.g., audio, haptic, and tactile) gathered through multiple tools (e.g., spoon, fork) and behaviors (e.g., stirring, poking) to perceive object properties. The framework segments interactions into time windows and aligns different modalities, enhancing data efficiency and interactive perception. Additionally, we conducted tool-transfer experiments to evaluate similarities between tools. Our experiments demonstrate that combining multiple tools and behaviors outperforms single-tool and singlebehavior approaches. While the audio modality dominates the non-visual multimodal system, other modalities contribute. We further demonstrate that tool similarities vary depending on the behavior, and notably, the robot does not need to complete entire interactions to achieve optimal recognition accuracy.