Patch Attention Excitation Based Vision Transformer for Small-Sized Datasets

Abstract

Vision Transformers have proven their mettle across a variety of computer vision problems, however, their reliance on pretraining with very large-scale datasets such as JFT-300M is also no secret, as large amounts of data is very conductive to effective feature learning. In this paper, however, we propose a novel Vision Transformer architecture PAEViT that aims to effectively learn generalized features from small-sized datasets, usually having only a few hundred images per class at most. By refining the attention scores based on patch-level interactions and modulating it to enhance the trained model’s ability to focus more on task relevant patches, PAEViT is able to significantly improve upon the performance of the regular ViT model, as well as other variants of the same, when the model is being trained solely on small-sized datasets. In data-constrained situations and visual recognition tasks that do not conform well with the existing large-scale datasets, PAEViT can be used to create effective and scalable solutions with all the features and attention scores being based only on relevant data. Our code is publicly available at https://github.com/AkashVermaIN/PAEViT.