FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers

doi:10.1109/TIP.2025.3587598

	FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers
	Yuan, Zheng 1,2; Zhang, Jie 1,2; Shan, Shiguang 1,2; Chen, Xilin 1,2
	2025
发表期刊	IEEE TRANSACTIONS ON IMAGE PROCESSING
ISSN	1057-7149
卷号	34 页码:4580-4590
摘要	In recent years, the Vision Transformer (ViT) model has gradually become mainstream in various computer vision tasks, and the robustness of the model has received increasing attention. However, existing large models tend to prioritize performance during training, potentially neglecting the robustness, which may lead to serious security concerns. In this paper, we establish a new challenge: exploring how to use a small number of additional parameters for adversarial finetuning to quickly and effectively enhance the adversarial robustness of a standardly trained model. To address this challenge, we develop novel LNLoRA module, incorporating a learnable layer normalization before the conventional LoRA module, which helps mitigate magnitude differences in parameters between the adversarial and standard training paradigms. Furthermore, we propose the FullLoRA framework by integrating the learnable LNLoRA modules into all key components of ViT-based models while keeping the pretrained model frozen, which can significantly improve the model robustness via adversarial finetuning in a parameter-efficient manner. Extensive experiments on several datasets demonstrate the superiority of our proposed FullLoRA framework. It achieves comparable robustness with full finetuning while only requiring about 5% of the learnable parameters. This also effectively addresses concerns regarding extra model storage space and enormous training time caused by adversarial finetuning.
关键词	Training Computational modeling Robustness Adaptation models Computer vision Transformers Visualization Natural language processing Image classification Head Adversarial training parameter-efficient pretrained model
DOI	10.1109/TIP.2025.3587598
收录类别	SCI
语种	英语
资助项目	Strategic Priority Research Program of Chinese Academy of Sciences[XDB0680202] ; Beijing Nova Program[20230484368] ; Suzhou Frontier Technology Research Project[SYG202325] ; Youth Innovation Promotion Association of Chinese Academy of Sciences (CAS)
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001534512500004
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/42045
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Jie
作者单位	1.Chinese Acad Sci, Inst Comp Technol, State Key Lab AI Safety, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Yuan, Zheng,Zhang, Jie,Shan, Shiguang,et al. FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2025,34:4580-4590.
APA	Yuan, Zheng,Zhang, Jie,Shan, Shiguang,&Chen, Xilin.(2025).FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers.IEEE TRANSACTIONS ON IMAGE PROCESSING,34,4580-4590.
MLA	Yuan, Zheng,et al."FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers".IEEE TRANSACTIONS ON IMAGE PROCESSING 34(2025):4580-4590.