VU-BERT: A Unified Framework for Visual Dialog
Tong Ye, Shijing Si, Jianzong Wang, Rui Wang, Ning Cheng, Jing Xiao
Abstract
The visual dialog task attempts to train an agent to answer multi-turn questions given an image, which requires the deep understanding of interactions between the image and dialog history. Existing researches tend to employ the modality-specific modules to model the interactions, which might be troublesome to use. To fill in this gap, we propose a unified framework for image-text joint embedding, named VU-BERT, and apply patch projection to obtain vision embedding firstly in visual dialog tasks to simplify the model. The model is trained over two tasks: masked language modeling and next utterance retrieval. These tasks help in learning visual concepts, utterances dependence, and the relationships between these two modalities. Finally, our VU-BERT achieves competitive performance (0.7287 NDCG scores) on VisDial v1.0 Datasets.
BibTeX
@inproceedings{icassp2022_vubertaunifiedfr,
title = {VU-BERT: A Unified Framework for Visual Dialog},
author = {Tong Ye and Shijing Si and Jianzong Wang and Rui Wang and Ning Cheng and Jing Xiao},
booktitle = {ICASSP 2022},
year = {2022}
}