Anomaly Knowledge Learning for Patch-Agnostic Defense against Adversarial Patches

Abstract

Adversarial patch defense has made significant progress recently, but defending against natural-looking patches remains a challenge due to their content-agnostic nature. We hypothesize that these patches exhibit position-related anomalies and are inspired by anomaly detection techniques. However, directly applying existing anomaly detection methods to patch detection behaves poorly because patches are a subset of anomalies, and using general anomalous features may introduce irrelevant anomalies. Additionally, anomaly detection datasets are primarily derived from industrial scenes, leading to out-of-distribution issues. To address these challenges, we propose a patch-agnostic defense method based on anomaly knowledge learning. It fine-tunes the Segment Anything Model in a self-supervised manner using an anomaly dataset, enabling the model’s image encoder to generate embeddings with enhanced activation for anomalous regions. It also designs a Cross Attention Patch Decoder based on cross-modal attention mechanisms to compute the mutual information between patch prediction probability maps and anomaly activation maps for patch localization. Our method shows strong performance on public datasets, and achieves a 16.9% mIoU improvement over the state-of-the-art in removing natural-looking patches with patch-to-target ratios over 0.6 on our constructed dataset.