Knowledge-Constrained Answer Generation for Open-Ended Video Question Answering
Yao Jin, Guocheng Niu, Xinyan Xiao, Jian Zhang, Xi Peng, Jun Yu
Abstract
Open-ended Video question answering (open-ended VideoQA) aims to understand video content and question semantics to generate the correct answers. Most of the best performing models define the problem as a discriminative task of multi-label classification. In real-world scenarios, however, it is difficult to define a candidate set that includes all possible answers. In this paper, we propose a Knowledge-constrained Generative VideoQA Algorithm (KcGA) with an encoder-decoder pipeline, which enables out-of-domain answer generation through an adaptive external knowledge module and a multi-stream information control mechanism. We use ClipBERT to extract the video-question features, extract framewise object-level external knowledge from a commonsense knowledge base and compute the contextual-aware episode memory units via an attention based GRU to form the external knowledge features, and exploit multi-stream information control mechanism to fuse video-question and external knowledge features such that the semantic complementation and alignment are well achieved. We evaluate our model on two open-ended benchmark datasets to demonstrate that we can effectively and robustly generate high-quality answers without restrictions of training data.
BibTeX
@article{Jin_Niu_Xiao_Zhang_Peng_Yu_2023, title={Knowledge-Constrained Answer Generation for Open-Ended Video Question Answering}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/25983}, DOI={10.1609/aaai.v37i7.25983}, abstractNote={Open-ended Video question answering (open-ended VideoQA) aims to understand video content and question semantics to generate the correct answers. Most of the best performing models define the problem as a discriminative task of multi-label classification. In real-world scenarios, however, it is difficult to define a candidate set that includes all possible answers. In this paper, we propose a Knowledge-constrained Generative VideoQA Algorithm (KcGA) with an encoder-decoder pipeline, which enables out-of-domain answer generation through an adaptive external knowledge module and a multi-stream information control mechanism. We use ClipBERT to extract the video-question features, extract framewise object-level external knowledge from a commonsense knowledge base and compute the contextual-aware episode memory units via an attention based GRU to form the external knowledge features, and exploit multi-stream information control mechanism to fuse video-question and external knowledge features such that the semantic complementation and alignment are well achieved. We evaluate our model on two open-ended benchmark datasets to demonstrate that we can effectively and robustly generate high-quality answers without restrictions of training data.}, number={7}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Jin, Yao and Niu, Guocheng and Xiao, Xinyan and Zhang, Jian and Peng, Xi and Yu, Jun}, year={2023}, month={Jun.}, pages={8141-8149} }