Toward Egocentric Compositional Action Anticipation with Adaptive Semantic Debiasing

doi:10.1145/3633333

	Toward Egocentric Compositional Action Anticipation with Adaptive Semantic Debiasing
	Zhang, Tianyu 1,2; Min, Weiqing 1,2; Liu, Tao 1,2; Jiang, Shuqiang 1,2; Rui, Yong 3
	2024-05-01
发表期刊	ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS
ISSN	1551-6857
卷号	20 期号:5 页码:21
摘要	Predicting the unknown from the first-person perspective is expected as a necessary step toward machine intelligence, which is essential for practical applications including autonomous driving and robotics. As a human-level task, egocentric action anticipation aims at predicting an unknown action seconds before it is performed from the first-person viewpoint. Egocentric actions are usually provided as verb-noun pairs; however, predicting the unknown action may be trapped in insufficient training data for all possible combinations. Therefore, it is crucial for intelligent systems to use limited known verb-noun pairs to predict new combinations of actions that have never appeared, which is known as compositional generalization. In this article, we are the first to explore the egocentric compositional action anticipation problem, which is more in line with real-world settings but neglected by existing studies. Whereas prediction results are prone to suffer from semantic bias considering the distinct difference between training and test distributions, we further introduce a general and flexible adaptive semantic debiasing framework that is compatible with different deep neural networks. To capture and mitigate semantic bias, we can imagine one counterfactual situation where no visual representations have been observed and only semantic patterns of observation are used to predict the next action. Instead of the traditional counterfactual analysis scheme that reduces semantic bias in a mindless way, we devise a novel counterfactual analysis scheme to adaptively amplify or penalize the effect of semantic experience by considering the discrepancy both among categories and among examples. We also demonstrate that the traditional counterfactual analysis scheme is a special case of the devised adaptive counterfactual analysis scheme. We conduct experiments on three large-scale egocentric video datasets. Experimental results verify the superiority and effectiveness of our proposed solution.
关键词	Egocentric video understanding compositional action anticipation semantic bias adaptive counterfactual analysis
DOI	10.1145/3633333
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Project of New Generation Artificial Intelligence of China[2018AAA0102500]
WOS研究方向	Computer Science
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods
WOS记录号	WOS:001192177900002
出版者	ASSOC COMPUTING MACHINERY
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/38747
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Tianyu
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, 6 Kexueyuan South Rd, Beijing, Peoples R China 2.Univ Chinese Acad Sci, 80 Zhongguancun East Rd, Beijing, Peoples R China 3.Lenovo Grp, 6 Shangdi West Rd, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Tianyu,Min, Weiqing,Liu, Tao,et al. Toward Egocentric Compositional Action Anticipation with Adaptive Semantic Debiasing[J]. ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS,2024,20(5):21.
APA	Zhang, Tianyu,Min, Weiqing,Liu, Tao,Jiang, Shuqiang,&Rui, Yong.(2024).Toward Egocentric Compositional Action Anticipation with Adaptive Semantic Debiasing.ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS,20(5),21.
MLA	Zhang, Tianyu,et al."Toward Egocentric Compositional Action Anticipation with Adaptive Semantic Debiasing".ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 20.5(2024):21.