Cross-modal interpretation of multi-modal sensor streams in interactive perception based on coupled recursion

Abstract

We present an online system to perceive kinematic properties of articulated objects from multi-modal sensor streams. The novelty of our system is that it leverages multi-modal information in a cross-modal manner: instead of simply fusing information from different modalities, sensor streams are interpreted by leveraging information from another modality. We realize each cross-modal information extraction process using recursive estimation, with each process addressing a perceptual subproblem. Several estimators are then coupled in a cross-modal network, leading to efficient and robust online perception. We demonstrate experimentally that our cross-modal system improves over its uni-modal counterparts, increasing the variability of environments and task conditions in which the robot can robustly perceive the articulated objects. We further demonstrate that these perceptual abilities are sufficiently fast to provide feedback during manipulation actions and sufficiently comprehensive to allow the generation of new manipulation actions.