Fine-Grained Open-Set Deepfake Detection via Unsupervised Domain Adaptation

doi:10.1109/TIFS.2024.3435440

	Fine-Grained Open-Set Deepfake Detection via Unsupervised Domain Adaptation
	Zhou, Xinye 1,2; Han, Hu 1,2; Shan, Shiguang 2,3; Chen, Xilin 2,3
	2024
发表期刊	IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY
ISSN	1556-6013
卷号	19 页码:7536-7547
摘要	Deepfake represented by face swapping and face reenactment can transfer the appearance and behavioral expressions of a face in one video image to another face in a different video. In recent years, with the advancement of deep learning techniques, deepfake technology has developed rapidly, achieving increasingly realistic effects. Therefore, many researchers have begun to study deepfake detection research. However, most existing studies on deepfake detection are mainly limited to binary classification of real and fake images, rather than identifying different methods in an open-world scenario, leading to failures in dealing with unknown deepfake categories in practice. In this paper, we propose an unsupervised domain adaptation method for fine-grained open-set deepfake detection. Our method first uses labeled data from the source domain for model pre-training to establish the ability of recognizing different deepfake methods in the source domain. Then, the method uses a Network Memorization based Adaptive Clustering (NMAC) approach to cluster unlabeled images in the target domain and designs a Pseudo-Label Generation (PLG) to generate virtual class labels for unknown deepfake categories by matching the adaptive clustering results with the known deepfake categories in the source domain. Finally, we retrain the initial multi-class deepfake detection model using labeled data of the source domain and pseudo-labeled data of the target domain to improve its generalization ability to unknown deepfake classes presented in the target domain. We validate the effectiveness of the proposed method under multiple open-set fine-grained deepfake detection tasks based on three deepfake datasets (ForgerNet, FaceForensics++, and FakeAVCeleb). Experimental results show that our method has better domain generalization ability than the state-of-the-art methods, and achieves promising performance in fine-grained open-set deepfake detection.
关键词	Deepfakes Adaptation models Feature extraction Data models Face recognition Training Faces Deepfake detection domain adaptation unsupervised learning fine-grained classification
DOI	10.1109/TIFS.2024.3435440
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2021ZD0111901] ; National Natural Science Foundation of China[62176249]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
WOS记录号	WOS:001294288400003
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39652
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Han, Hu
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Key Lab AI Safety CAS, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zhou, Xinye,Han, Hu,Shan, Shiguang,et al. Fine-Grained Open-Set Deepfake Detection via Unsupervised Domain Adaptation[J]. IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY,2024,19:7536-7547.
APA	Zhou, Xinye,Han, Hu,Shan, Shiguang,&Chen, Xilin.(2024).Fine-Grained Open-Set Deepfake Detection via Unsupervised Domain Adaptation.IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY,19,7536-7547.
MLA	Zhou, Xinye,et al."Fine-Grained Open-Set Deepfake Detection via Unsupervised Domain Adaptation".IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY 19(2024):7536-7547.