Attention Augmented Structure-centric Bias Mitigation with Feature Disentanglement

Abstract

Image classification models often rely on superficial visual features, such as textures or colors, leading to undesired bias. This can compromise the robustness and reliability of deep models, particularly their performance on out-of-distribution (o.o.d.) datasets. Existing approaches, focusing on data-centric aspects, typically predefine specific bias types to mitigate the impact of these superficial features. However, such data-centric methods may lack extensibility due to their focus on predefined biases. In this paper, we propose an attention augmented structure-centric bias mitigation method, considering network architecture can be flexibly manipulated to address a variety of visual features. This method captures global semantic representations by integrating the strengths of both self-attention and convolution, introducing a global receptive field to Convolutional Neural Networks. By incorporating feature disentanglement and augmentation, our concise network demonstrates improved performance as feature diversity increases in the latent space. Our method achieves state-of-the-art results on synthetic datasets (Colored MNIST and Corrupted CIFAR10) and shows impressive performance on challenging real-world datasets (ImageNet and BFFHQ), with improvements of about 2%-5% across different subsets.