Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism

doi:10.1109/TIP.2018.2886767

	Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
	Li, Yong 1,2; Zeng, Jiabei 1; Shan, Shiguang 3,4; Chen, Xilin 1,2
	2019-05-01
发表期刊	IEEE TRANSACTIONS ON IMAGE PROCESSING
ISSN	1057-7149
卷号	28 期号:5 页码:2439-2450
摘要	Facial expression recognition in the wild is challenging due to various unconstrained conditions. Although existing facial expression classifiers have been almost perfect on analyzing constrained frontal faces, they fail to perform well on partially occluded faces that are common in the wild. In this paper, we propose a convolution neutral network (CNN) with attention mechanism (ACNN) that can perceive the occlusion regions of the face and focus on the most discriminative un-occluded regions. ACNN is an end-to-end learning framework. It combines the multiple representations from facial regions of interest (ROIs). Each representation is weighed via a proposed gate unit that computes an adaptive weight from the region itself according to the unobstructedness and importance. Considering different RoIs, we introduce two versions of ACNN: patch-based ACNN (pACNN) and global-local-based ACNN (gACNN). pACNN only pays attention to local facial patches. gACNN integrates local representations at patch-level with global representation at image-level. The proposed ACNNs are evaluated on both real and synthetic occlusions, including a self-collected facial expression dataset with real-world occlusions, the two largest in-the-wild facial expression datasets (RAF-DB and AffectNet) and their modifications with synthesized facial occlusions. Experimental results show that ACNNs improve the recognition accuracy on both the non-occluded faces and occluded faces. Visualization results demonstrate that, compared with the CNN without Gate Unit, ACNNs are capable of shifting the attention from the occluded patches to other related but unobstructed ones. ACNNs also outperform other state-of-the-art methods on several widely used in-the-lab facial expression datasets under the cross-dataset evaluation protocol.
关键词	Facial expression recognition occlusion CNN with attention mechanism gate unit
DOI	10.1109/TIP.2018.2886767
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2017YFB1002802] ; Natural Science Foundation of China[61702481] ; Natural Science Foundation of China[61702486] ; External Cooperation Program of CAS[GJHZ1843]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000458850800004
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：427[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/3423
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zeng, Jiabei
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Ctr Excellence Brain Sci & Intelligence Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 4.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Li, Yong,Zeng, Jiabei,Shan, Shiguang,et al. Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2019,28(5):2439-2450.
APA	Li, Yong,Zeng, Jiabei,Shan, Shiguang,&Chen, Xilin.(2019).Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism.IEEE TRANSACTIONS ON IMAGE PROCESSING,28(5),2439-2450.
MLA	Li, Yong,et al."Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism".IEEE TRANSACTIONS ON IMAGE PROCESSING 28.5(2019):2439-2450.