Evaluating Human-Robot Collaboration through Online Video: Perspective Matters
Leimin Tian, Shiyu Xu, Kerry He, Rachel Love, Akansel Cosgun, Dana Kulic
Abstract
Online evaluation is increasingly adopted in robotics research, providing an efficient approach to collect data from large and diverse populations. However, there have been ongoing debates about online studies as a proxy for in-person studies, especially where a participant passively observes video of robot behaviours or interaction. We conduct an online video comparison study (N=178) evaluating three robot handover policies in a collaborative assembly task, namely an adaptive autonomous policy, a non-adaptive scripted policy, and teleoperation. Participants watched three sets of videos in third-person view, each consisting of 9 sequential handovers executing one of the policies. Compared to in-person participants in two previous studies who evaluated handovers as users, online participants were observant of different robot behaviours and human-robot collaboration contexts, with 76.4% and 71.9% recognising the adaptive handovers exhibited by the teleoperated and autonomous robot, respectively. However, as observers, online participants showed more critical subjective perceptions compared to the in-person participants with a user’s perspective. They valued efficiency over adaptation with twice more autonomous handovers rated as being too late compared to scripted handovers. Our work highlights the need to consider user contexts when evaluating human-robot collaboration.
BibTeX
@inproceedings{iros2025_evaluatinghumanr,
title = {Evaluating Human-Robot Collaboration through Online Video: Perspective Matters},
author = {Leimin Tian and Shiyu Xu and Kerry He and Rachel Love and Akansel Cosgun and Dana Kulic},
booktitle = {IROS 2025},
year = {2025}
}