SEER: Backdoor Detection for Vision-Language Models through Searching Target Text and Image Trigger Jointly
Liuwan Zhu, Rui Ning, Jiang Li, Chunsheng Xin, Hongyi Wu
Abstract
This paper proposes SEER, a novel backdoor detection algorithm for vision-language models, addressing the gap in the literature on multi-modal backdoor detection. While backdoor detection in single-modal models has been well studied, the investigation of such defenses in multi-modal models remains limited. Existing backdoor defense mechanisms cannot be directly applied to multi-modal settings due to their increased complexity and search space explosion. In this paper, we propose to detect backdoors in vision-language models by jointly searching image triggers and malicious target texts in feature space shared by vision and language modalities. Our extensive experiments demonstrate that SEER can achieve over 92% detection rate on backdoor detection in vision-language models in various settings without accessing training data or knowledge of downstream tasks.
BibTeX
@article{Zhu_Ning_Li_Xin_Wu_2024, title={SEER: Backdoor Detection for Vision-Language Models through Searching Target Text and Image Trigger Jointly}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/28611}, DOI={10.1609/aaai.v38i7.28611}, abstractNote={This paper proposes SEER, a novel backdoor detection algorithm for vision-language models, addressing the gap in the literature on multi-modal backdoor detection. While backdoor detection in single-modal models has been well studied, the investigation of such defenses in multi-modal models remains limited. Existing backdoor defense mechanisms cannot be directly applied to multi-modal settings due to their increased complexity and search space explosion. In this paper, we propose to detect backdoors in vision-language models by jointly searching image triggers and malicious target texts in feature space shared by vision and language modalities. Our extensive experiments demonstrate that SEER can achieve over 92% detection rate on backdoor detection in vision-language models in various settings without accessing training data or knowledge of downstream tasks.}, number={7}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Zhu, Liuwan and Ning, Rui and Li, Jiang and Xin, Chunsheng and Wu, Hongyi}, year={2024}, month={Mar.}, pages={7766-7774} }