ETBench: Characterizing Hybrid Vision <underline>T</underline>ransformer Workloads Across <underline>E</underline>dge Devices

doi:10.1109/TC.2025.3543697

	ETBench: Characterizing Hybrid Vision Transformer Workloads Across Edge Devices
	Zhou, Yingkun 1,2; Tian, Zhengshuyuan 1,2; Yang, Wenhao 1,2; Zhang, Tingting 3,4; Ye, Jinpeng 1,2; Han, Chenji 1,2; Liu, Tianyi 5; Zhang, Fuxin 1
	2025-06-01
发表期刊	IEEE TRANSACTIONS ON COMPUTERS
ISSN	0018-9340
卷号	74 期号:6 页码:1857-1871
摘要	Lightweight Convolution and Vision Transformer hybrid models have increasingly dominated the frontiers of deep learning (DL) on edge devices; however, to the best of our knowledge, no prior work has provided comprehensive evaluation on hybrid models' performance and analyzed their characteristics by diving deep into the edge ecosystem with diversified modern DL inference engines and heterogeneous hardware. This paper proposes a comprehensive open-source benchmark suite, ETBench, to allow power-efficiency, performance and accuracy assessment for state-of-the-art (SOTA) hybrid models across 11 most widely-used DL engines deployed on diverse edge devices. After building ETBench that satisfies 6 design requirements proposed in our work, we conduct extensive experiments on 14 devices including 19 CPUs, 11 GPUs and 5 NPUs, and obtain benchmark results from all deployment scenarios (combinations of models, quantization formats, software engines, and hardware platforms). Valuable observations and insightful implications are finally summarized. For example, within current DL engines, the INT8 quantization is significantly underperformed in terms of accuracy and speed against FP16 for hybrid models. Overall, ETBench serves as a collaborative platform that assists model architects in better evaluating their models and makes it possible for future co-optimizations of DL engines and hardware accelerators.
关键词	Benchmark deep learning edge devices vision transformer hybrid model Benchmark deep learning edge devices vision transformer hybrid model
DOI	10.1109/TC.2025.3543697
收录类别	SCI
语种	英语
资助项目	Chinese Academy of Sciences[XDC05020100]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Hardware & Architecture ; Engineering, Electrical & Electronic
WOS记录号	WOS:001492675500026
出版者	IEEE COMPUTER SOC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/42332
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Tingting
作者单位	1.Chinese Acad Sci, Inst Comp Technol, State Key Lab Processors, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Loongson Technol Co Ltd, Beijing 100095, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 5.Univ Texas San Antonio, San Antonio, TX 78249 USA
推荐引用方式 GB/T 7714	Zhou, Yingkun,Tian, Zhengshuyuan,Yang, Wenhao,et al. ETBench: Characterizing Hybrid Vision Transformer Workloads Across Edge Devices[J]. IEEE TRANSACTIONS ON COMPUTERS,2025,74(6):1857-1871.
APA	Zhou, Yingkun.,Tian, Zhengshuyuan.,Yang, Wenhao.,Zhang, Tingting.,Ye, Jinpeng.,...&Zhang, Fuxin.(2025).ETBench: Characterizing Hybrid Vision Transformer Workloads Across Edge Devices.IEEE TRANSACTIONS ON COMPUTERS,74(6),1857-1871.
MLA	Zhou, Yingkun,et al."ETBench: Characterizing Hybrid Vision Transformer Workloads Across Edge Devices".IEEE TRANSACTIONS ON COMPUTERS 74.6(2025):1857-1871.