Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search

doi:10.1016/j.neunet.2023.09.012

	Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search
	Yang, Longxing 1,3; Fu, Yanxin 2; Lu, Shun 1,3; Sun, Zihao 1,3; Mei, Jilin 1; Zhao, Wenxiao 2,4; Hu, Yu 1,3
	2023-11-01
发表期刊	NEURAL NETWORKS
ISSN	0893-6080
卷号	168 页码:237-255
摘要	Zero-shot Neural Architecture Search has garnered attention due to its training-free nature and rapid search speed. However, existing zero-shot estimators commonly suffer from low consistency, which hampers their practicality. In this work, we theoretically analyze that network generalization and convergence are highly correlated with Sweet Gradient of Parameter, i.e., the number of parameters whose gradient absolute values are within a certain interval. Empirical results indicate that Sweet Gradient of Parameter brings a higher consistency than the overall number of parameters. Additionally, we demonstrate a positive correlation between the network depth and the proportion of parameters with sweet gradients in each layer. Based on the analysis, we propose a training-free method to find the Sweet Gradient interval and obtain an estimator, named Sweetimator. Furthermore, Sweet Gradient can be an effective and general approach to promote the consistency of zero-shot estimators. Experiments show that Sweetimator and Sweet-enhanced estimators have significant consistency improvement in multiple benchmarks. Our method achieves state-of-the-art performance with 256x speedup in NAS-Bench-201 and maintains high competitiveness in DARTS, MobileNet, and Transformer search spaces. The source code is available at https://github.com/xingxing-123/SweetGradient.
关键词	Zero-shot Neural Architecture Search Low Consistency Sweet Gradient Sweetimator Sweet-enhanced estimators
DOI	10.1016/j.neunet.2023.09.012
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62176250] ; National Natural Science Foundation of China[62203424] ; National Key Research and Development Program of China[2018YFA0703800] ; Chinese Academy of Sciences (CAS) Project for Young Scientists in Basic Research[YSBR-008] ; Strategic Priority Research Program of Chinese Academy of Sciences[XDA27000000]
WOS研究方向	Computer Science ; Neurosciences & Neurology
WOS类目	Computer Science, Artificial Intelligence ; Neurosciences
WOS记录号	WOS:001086926100001
出版者	PERGAMON-ELSEVIER SCIENCE LTD
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21090
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhao, Wenxiao; Hu, Yu
作者单位	1.Chinese Acad Sci, Res Ctr Intelligent Comp Syst, Inst Comp Technol, Beijing 100190, Peoples R China 2.Chinese Acad Sci, Acad Math & Syst Sci, Key Lab Syst & Control, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 100049, Peoples R China 4.Univ Chinese Acad Sci, Sch Math Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Yang, Longxing,Fu, Yanxin,Lu, Shun,et al. Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search[J]. NEURAL NETWORKS,2023,168:237-255.
APA	Yang, Longxing.,Fu, Yanxin.,Lu, Shun.,Sun, Zihao.,Mei, Jilin.,...&Hu, Yu.(2023).Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search.NEURAL NETWORKS,168,237-255.
MLA	Yang, Longxing,et al."Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search".NEURAL NETWORKS 168(2023):237-255.