Contrastive Auditory Knowledge Transfer for Tool-Mediated Robot Interaction with Granular Objects

Abstract

Tool-mediated interactions enable robotics to manipulate and explore granular objects, producing informative auditory signals. A central challenge is transferring this perceptual knowledge across different tools and behaviors without costly data collection for each new context. We address this problem in the domain of audio-based recognition of granular and liquid-like objects. In this work, we leverage audio signals from tool-mediated interactions and learn context-agnostic representations for object recognition. We propose two contrastive learning approaches: a shared-object transfer method that performs supervised contrastive learning using audio data, and a zero-shot transfer method that integrates both audio and natural language descriptions of interaction contexts. Experiments on real-world data show that both methods achieve strong object recognition performance in unseen contexts, sometimes matching or exceeding a supervised baseline despite limited target context data. Furthermore, the learned latent spaces exhibit clearly separable clusters by object identity, and the zero-shot method successfully recognizes novel objects, offering a practical solution for robot perception in data-scarce scenarios. The code for this paper is available at: https://github.com/siliu6487/AuditoryKnowledgeTransfer.