CIGMA: Causal Information-Gain Mechanistic Attribution of Attention Heads in Vision Transformers
Vision Transformers often rely on spurious background correlations rather than foreground object features. While prior model pruning approaches focus solely on improving accuracy, they lack interpretability and fail to verify whether predictions are actually made by focusing on the main foreground o