HENASY: Learning to Assemble Scene-Entities for Interpretable Egocentric Video-Language Model
Khoa Vo, Thinh Phan, Kashu Yamazaki, Minh Tran, Ngan Hoang Le
Abstract
Current video-language models (VLMs) rely extensively on instance-level alignment between video and language modalities, which presents two major limitations: (1) visual reasoning disobeys the natural perception that humans do in first-person perspective, leading to a lack of reasoning interpretation; and (2) learning is limited in capturing inherent fine-grained relationships between two modalities. In this paper, we take an inspiration from human perception and explore a compositional approach for egocentric video representation. We introduce HENASY (Hierarchical ENtities ASsemblY), which includes a spatiotemporal token grouping mechanism to explicitly assemble dynamically evolving scene entities through time and model their relationship for video representation. By leveraging compositional structure understanding, HENASY possesses strong interpretability via visual grounding with free-form text queries. We further explore a suite of multi-grained contrastive losses to facilitate entity-centric understandings. This comprises three alignment types: video-narration, noun-entity, verb-entities alignments. Our method demonstrates strong interpretability in both quantitative and qualitative experiments; while maintaining competitive performances on five downstream tasks via zero-shot transfer or as video/text representation, including video/text retrieval, action recognition, multi-choice query, natural language query, and moments query. Project page: https://uark-aicv.github.io/HENASY
BibTeX
@inproceedings{
vo2024henasy,
title={{HENASY}: Learning to Assemble Scene-Entities for Interpretable Egocentric Video-Language Model},
author={Khoa Vo and Thinh Phan and Kashu Yamazaki and Minh Tran and Ngan Hoang Le},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=7uWzoGn4kv}
}