Exp-VQA: Fine-grained facial expression analysis via visual question answering

doi:10.1016/j.patcog.2025.111783

	Exp-VQA: Fine-grained facial expression analysis via visual question answering
	Yuan, Yujian 1,2; Zeng, Jiabei 1,2; Shan, Shiguang 1,2
	2025-12-01
发表期刊	PATTERN RECOGNITION
ISSN	0031-3203
卷号	168 页码:13
摘要	This paper presents a novel task, facial expression visual question answering (VQA), for fine-grained facial expression analysis across multiple scales. Facial expression VQA interprets facial expressions in a more detailed and comprehensive way than traditional emotion categories or facial action units (AUs). To develop the facial expression VQA model, we finetuned an InstructBLIP using synthesized VQA pairs, forming the Exp-VQA model. These VQA pairs are synthesized leveraging the powerful descriptive ability of GPT-3.5 and a rule-based generator, based on existing annotations on both emotion classification and AU detection datasets. Exp-VQA can describe the facial status of the whole face as well as infer the indicated emotion, detail facial actions in specific regions, and detect individual AU occurrences. Experiments demonstrate the effectiveness of Exp-VQA in describing multi-scale facial expressions, as well as state-of-the-art zero-shot ability in detecting unseen AUs. Furthermore, the training of Exp-VQA enhances its intermediate visual features' performance on both AU detection and emotion classification tasks. The code and trained models are available at: https://github.com/Yujianyuan/Exp-VQA.
关键词	Fine-grained facial expression analysis Emotion classification Facial action unit detection Visual question answering
DOI	10.1016/j.patcog.2025.111783
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62176248] ; National Natural Science Foundation of China[U2336213]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001502075500001
出版者	ELSEVIER SCI LTD
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/42314
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zeng, Jiabei
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Yuan, Yujian,Zeng, Jiabei,Shan, Shiguang. Exp-VQA: Fine-grained facial expression analysis via visual question answering[J]. PATTERN RECOGNITION,2025,168:13.
APA	Yuan, Yujian,Zeng, Jiabei,&Shan, Shiguang.(2025).Exp-VQA: Fine-grained facial expression analysis via visual question answering.PATTERN RECOGNITION,168,13.
MLA	Yuan, Yujian,et al."Exp-VQA: Fine-grained facial expression analysis via visual question answering".PATTERN RECOGNITION 168(2025):13.