Steering Visuomotor Policy in Open Worlds via Cross-View Goal Alignment

Shaofei Cai; Zhancun Mu; Anji Liu; Yitao Liang

AAAI 2026technical0 citations

Steering Visuomotor Policy in Open Worlds via Cross-View Goal Alignment

Shaofei Cai, Zhancun Mu, Anji Liu, Yitao Liang

Abstract

We aim to develop a goal specification method that is semantically clear, spatially sensitive, domain-agnostic, and intuitive for human users to guide agent interactions in 3D environments. Specifically, we propose a novel cross-view goal alignment framework that allows users to specify target objects using segmentation masks from their camera views rather than the agent’s observations. We highlight that behavior cloning alone fails to align the agent’s behavior with human intent when the human and agent camera views differ significantly. To address this, we introduce two auxiliary objectives: cross-view consistency loss and target visibility loss, which explicitly enhance the agent

BibTeX

@inproceedings{aaai2026_steeringvisuomot,
  title = {Steering Visuomotor Policy in Open Worlds via Cross-View Goal Alignment},
  author = {Shaofei Cai and Zhancun Mu and Anji Liu and Yitao Liang},
  booktitle = {AAAI 2026},
  year = {2026}
}