Human-In-The-Loop Gaussian Splatting for Robotic Teleoperation

Abstract

Safe, precise teleoperation demands a third-person 3D view that reveals collision clearances and task-critical geometry in full detail. Yet most systems still rely on live camera streams that offer tunnel-vision perspectives and weak depth cues, hiding hazards and denying operators the spatial context for precise manipulation. 3D Gaussian Splatting (GS) enables real-time photorealistic streaming, but acquiring the required multi-view imagery safely and efficiently remains a critical bottleneck in cluttered teleoperation environments. We propose Human-in-the-Loop Gaussian Splatting (HIL-GS) that delivers safe, robust, and efficient 3D scene reconstruction for challenging teleoperation environments. HIL-GS combines three modules in a tightly-coupled loop: (1) motion-aware GS reconstruction that fuses RGB-D and proprioceptive sensors for drift-free and robust mapping under aggressive motions; (2) VR-based informative display that renders the GS map with contextual overlays/feedback in real time to ensure situational awareness and reconstruction completeness; and (3) finger- based control interface to guide the robot toward informative viewpoints through safe, non-redundant motions. Through simulation, real-world experiments, and a user study, we demonstrate that HIL-GS outperforms traditional approaches in reconstruction quality, usability, and efficiency.