Exploring logical consistency and viewport sensitivity in compositional VQA models

Abstract

The most recent architectures for Visual Question Answering (VQA), such as TbD or DDRprog, have already outperformed human-level accuracy on benchmark datasets (e.g. CLEVR). We administered an advanced analysis of their performance based on novel metrics called consistency (sum of all object feature instances in the scene (e.g. shapes) equals the total number of the objects in the scene) and revealed only 56% consistency for the most accurate architecture (TbD). In respect to this finding, we propose a new method of the VQA training, which reaches 98% consistency. Furthermore, testing of the VQA model in real world brings out a problem with precise mimicking of the camera position from the original dataset. We therefore created a virtual environment along with its real-world counterpart with variable camera positions to test the accuracy and consistency from different viewports. Based on these errors, we were able to estimate optimal position of the camera. The proposed method thus allows us to find the optimal camera viewport in the real environment without knowing the geometry and the exact position of the camera in the synthetic training environment.