Towards Open-World Human Action Segmentation Using Graph Convolutional Networks

Abstract

Current methods for human-object interaction segmentation excel in closed-world settings but struggle to generalize to open-world scenarios where novel actions emerge. Since collecting exhaustive training data for all possible dynamic human activities is impractical, a model capable of detecting and segmenting novel, out-of-distribution (OOD) actions without manual annotation is needed. To address this, we formally define the open-world action segmentation problem and propose a novel framework featuring three key components: 1) an Enhanced Pyramid Graph Convolutional Network with a new decoder for robust spatiotemporal upsampling, 2) hybrid-based training synthesizing OOD data to eliminate reliance on manual labels, and 3) a temporal clustering loss that groups in-distribution actions while distancing OOD samplesWe evaluate our framework on two challenging human-object interaction recognition datasets: Bimanual Actions and Two Hands and Object datasets. Experimental results demonstrate significant improvements over state-of-the-art action segmentation models across multiple open-set evaluation metrics, achieving 16.9% and 34.6% relative gains in open-set segmentation (F1@50) and out-of-distribution detection performances (AUROC), respectively. Additionally, we conduct an in-depth ablation study to assess the impact of each proposed component, identifying the optimal framework configuration for open-world action segmentation.