RobustPrompt: Learning to defend against adversarial attacks with adaptive visual prompts

doi:10.1016/j.patrec.2025.02.015

	RobustPrompt: Learning to defend against adversarial attacks with adaptive visual prompts
	Liu, Chang 1; Xiang, Wenzhao 2,3; Dong, Yinpeng 6; Zhang, Xingxing 4; Wang, Liyuan 4; Duan, Ranjie 4,5; Zheng, Shibao 1; Su, Hang 6
	2025-04-01
发表期刊	PATTERN RECOGNITION LETTERS
ISSN	0167-8655
卷号	190 页码:161-168
摘要	Adversarial training stands out as one of the most effective techniques for enhancing robustness by enriching the training data with adversarial examples. Nonetheless, when faced with various perturbation budgets, the model's performance can suffer notable degradation. This occurs because different perturbations induce distinct distribution shifts in adversarial examples. In order to enhance performance on specific perturbations, fine-tuning is commonly employed. However, this approach can lead to catastrophic forgetting, where improvements on specific tasks come at the cost of degrading performance on previously learned ones. We frame this challenge as an incremental domain learning problem in continual learning. Inspired by the application of prompt techniques in vision models, we introduce RobustPrompt, which integrates additional guidance information regarding perturbation characteristics into the adversarial training process. This approach enables the model to adaptively enhance its robustness under varying budget perturbations. Specifically, we define an adaptive prompt pool composed of a noise level predictor and corresponding prompts for different perturbations. During training, prompts are injected into different layers of the model, thereby guiding the model to focus on correct features. Experiments demonstrate that RobustPrompt enhances the adversarial robustness of the state-of-the-art Swin Transformer Base model, achieving an average improvement of 61.1% against PGD attacks and 56.9% against AutoAttack across five white-box settings; an average improvement of 76.1% against VMI-FGSM attacks across five black-box settings; an average improvement of 53.7% on five datasets with natural noise. Our results underscore the potential of RobustPrompt as a useful tool for enhancing the reliability and robustness of transformers in image classification tasks.
关键词	Adversarial robustness Prompt tuning Continual learning Adaptive prompt pool
DOI	10.1016/j.patrec.2025.02.015
收录类别	SCI
语种	英语
资助项目	NSFC[62076147] ; NSFC[U19A2081] ; NSFC[U19B2034] ; NSFC[U1811461] ; NSFC[62071292] ; NSFC[U21B2013] ; STCSM[22DZ2229005] ; Alibaba Group through Alibaba Innovative Research Program, a grant from Tsinghua Institute for Guo Qiang ; High Performance Computing Center, Tsinghua University
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001436909400001
出版者	ELSEVIER
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/40705
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zheng, Shibao; Su, Hang
作者单位	1.Shanghai Jiao Tong Univ, Inst Image Commun & Networks Engn, Dept Elect Engn EE, Shanghai 200240, Peoples R China 2.PengCheng Lab, Shenzhen 518055, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, CAS, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 4.Tsinghua Univ, Beijing 100084, Peoples R China 5.Alibaba Grp, Beijing 100102, Peoples R China 6.Tsinghua Univ, Inst AI, Dept Comp Sci & Technol, THBI Lab, Beijing 100084, Peoples R China
推荐引用方式 GB/T 7714	Liu, Chang,Xiang, Wenzhao,Dong, Yinpeng,et al. RobustPrompt: Learning to defend against adversarial attacks with adaptive visual prompts[J]. PATTERN RECOGNITION LETTERS,2025,190:161-168.
APA	Liu, Chang.,Xiang, Wenzhao.,Dong, Yinpeng.,Zhang, Xingxing.,Wang, Liyuan.,...&Su, Hang.(2025).RobustPrompt: Learning to defend against adversarial attacks with adaptive visual prompts.PATTERN RECOGNITION LETTERS,190,161-168.
MLA	Liu, Chang,et al."RobustPrompt: Learning to defend against adversarial attacks with adaptive visual prompts".PATTERN RECOGNITION LETTERS 190(2025):161-168.