Leveraging Saliency-Aware Gaze Heatmaps for Multiperspective Teaching of Unknown Objects

Abstract

As robots become increasingly prevalent amidst diverse environments, their ability to adapt to novel scenarios and objects is essential. Advances in modern object detection have also paved the way for robots to identify interaction entities within their immediate vicinity. One drawback is that the robot's operational domain must be known at the time of training, which hinders the robot's ability to adapt to unexpected environments outside the preselected classes. However, when encountering such challenges a human can provide support to a robot by teaching it about the new, yet unknown objects on an ad hoc basis. In this work, we merge augmented reality and human gaze in the context of multimodal human-robot interaction to compose saliency-aware gaze heatmaps leveraged by a robot to learn emerging objects of interest. Our results show that our proposed method exceeds the capabilities of the current state of the art and outperforms it in terms of commonly used object detection metrics.