Open-Vocabulary Video Relation Extraction
Wentao Tian, Zheng Wang, Yuqian Fu, Jingjing Chen, Lechao Cheng
Abstract
A comprehensive understanding of videos is inseparable from describing the action with its contextual action-object interactions. However, many current video understanding tasks prioritize general action classification and overlook the actors and relationships that shape the nature of the action, resulting in a superficial understanding of the action. Motivated by this, we introduce Open-vocabulary Video Relation Extraction (OVRE), a novel task that views action understanding through the lens of action-centric relation triplets. OVRE focuses on pairwise relations that take part in the action and describes these relation triplets with natural languages. Moreover, we curate the Moments-OVRE dataset, which comprises 180K videos with action-centric relation triplets, sourced from a multi-label action classification dataset. With Moments-OVRE, we further propose a cross-modal mapping model to generate relation triplets as a sequence. Finally, we benchmark existing cross-modal generation models on the new task of OVRE. Our code and dataset are available at https://github.com/Iriya99/OVRE.
BibTeX
@article{Tian_Wang_Fu_Chen_Cheng_2024, title={Open-Vocabulary Video Relation Extraction}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/28328}, DOI={10.1609/aaai.v38i6.28328}, abstractNote={A comprehensive understanding of videos is inseparable from describing the action with its contextual action-object interactions. However, many current video understanding tasks prioritize general action classification and overlook the actors and relationships that shape the nature of the action, resulting in a superficial understanding of the action. Motivated by this, we introduce Open-vocabulary Video Relation Extraction (OVRE), a novel task that views action understanding through the lens of action-centric relation triplets. OVRE focuses on pairwise relations that take part in the action and describes these relation triplets with natural languages. Moreover, we curate the Moments-OVRE dataset, which comprises 180K videos with action-centric relation triplets, sourced from a multi-label action classification dataset. With Moments-OVRE, we further propose a cross-modal mapping model to generate relation triplets as a sequence. Finally, we benchmark existing cross-modal generation models on the new task of OVRE. Our code and dataset are available at https://github.com/Iriya99/OVRE.}, number={6}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Tian, Wentao and Wang, Zheng and Fu, Yuqian and Chen, Jingjing and Cheng, Lechao}, year={2024}, month={Mar.}, pages={5215-5223} }