A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking

doi:10.1007/s11263-024-02196-3

	A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking
	Liu, Chang 2; Dong, Yinpeng 1,5; Xiang, Wenzhao 3,7; Yang, Xiao 1; Su, Hang 1,6; Zhu, Jun 1,5; Chen, Yuefeng 4; He, Yuan 4; Xue, Hui 4; Zheng, Shibao 2
	2024-08-09
发表期刊	INTERNATIONAL JOURNAL OF COMPUTER VISION
ISSN	0920-5691
页码	23
摘要	The robustness of deep neural networks is frequently compromised when faced with adversarial examples, common corruptions, and distribution shifts, posing a significant research challenge in the advancement of deep learning. Although new deep learning methods and robustness improvement techniques have been constantly proposed, the robustness evaluations of existing methods are often inadequate due to their rapid development, diverse noise patterns, and simple evaluation metrics. Without thorough robustness evaluations, it is hard to understand the advances in the field and identify the effective methods. In this paper, we establish a comprehensive robustness benchmark called ARES-Bench on the image classification task. In our benchmark, we evaluate the robustness of 61 typical deep learning models on ImageNet with diverse architectures (e.g., CNNs, Transformers) and learning algorithms (e.g., normal supervised training, pre-training, adversarial training) under numerous adversarial attacks and out-of-distribution (OOD) datasets. Using robustness curves as the major evaluation criteria, we conduct large-scale experiments and draw several important findings, including: (1) there exists an intrinsic trade-off between the adversarial and natural robustness of specific noise types for the same model architecture; (2) adversarial training effectively improves adversarial robustness, especially when performed on Transformer architectures; (3) pre-training significantly enhances natural robustness by leveraging larger training datasets, incorporating multi-modal data, or employing self-supervised learning techniques. Based on ARES-Bench, we further analyze the training tricks in large-scale adversarial training on ImageNet. Through tailored training settings, we achieve a new state-of-the-art in adversarial robustness. We have made the benchmarking results and code platform publicly available.
关键词	Robustness benchmark Distribution shift Pre-training Adversarial training Image classification
DOI	10.1007/s11263-024-02196-3
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62076147] ; National Natural Science Foundation of China[92370124] ; National Natural Science Foundation of China[62276149] ; National Natural Science Foundation of China[92248303] ; National Natural Science Foundation of China[62071292] ; National Natural Science Foundation of China[U21B2013] ; National Natural Science Foundation of China[U2341228] ; Science and Technology Commission of Shanghai Municipality[22DZ2229005] ; Tsinghua-Alibaba Joint Research Program ; China National Postdoctoral Program for Innovative Talents ; CCF-BaiChuan-Ebtech Foundation Model Fund
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001287481400002
出版者	SPRINGER
引用统计	被引频次：2[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39662
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Su, Hang; Zheng, Shibao
作者单位	1.Chinese Acad Sci, Key Lab Intelligent Informat Proc, Inst Comp Technol ICT, Beijing 100190, Peoples R China 2.Shanghai Jiao Tong Univ, Inst Image Commun & Networks Engn, Dept Elect Engn EE, Shanghai 200240, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 4.Alibaba Grp, Hangzhou 310023, Zhejiang, Peoples R China 5.RealAI, Beijing 100085, Peoples R China 6.Zhongguancun Lab, Beijing 100080, Peoples R China 7.Peng Cheng Lab, Shenzhen 518000, Peoples R China
推荐引用方式 GB/T 7714	Liu, Chang,Dong, Yinpeng,Xiang, Wenzhao,et al. A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking[J]. INTERNATIONAL JOURNAL OF COMPUTER VISION,2024:23.
APA	Liu, Chang.,Dong, Yinpeng.,Xiang, Wenzhao.,Yang, Xiao.,Su, Hang.,...&Zheng, Shibao.(2024).A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking.INTERNATIONAL JOURNAL OF COMPUTER VISION,23.
MLA	Liu, Chang,et al."A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking".INTERNATIONAL JOURNAL OF COMPUTER VISION (2024):23.