Reference-Based Deep Line Art Video Colorization

doi:10.1109/TVCG.2022.3146000

	Reference-Based Deep Line Art Video Colorization
	Shi, Min 1; Zhang, Jia-Qi 2; Chen, Shu-Yu 3,4; Gao, Lin 3,4; Lai, Yu-Kun 5; Zhang, Fang-Lue 6
	2023-06-01
发表期刊	IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
ISSN	1077-2626
卷号	29 期号:6 页码:2965-2979
摘要	Coloring line art images based on the colors of reference images is a crucial stage in animation production, which is time-consuming and tedious. This paper proposes a deep architecture to automatically color line art videos with the same color style as the given reference images. Our framework consists of a color transform network and a temporal refinement network based on 3U-net. The color transform network takes the target line art images as well as the line art and color images of the reference images as input and generates corresponding target color images. To cope with the large differences between each target line art image and the reference color images, we propose a distance attention layer that utilizes non-local similarity matching to determine the region correspondences between the target image and the reference images and transforms the local color information from the references to the target. To ensure global color style consistency, we further incorporate Adaptive Instance Normalization (AdaIN) with the transformation parameters obtained from a multiple-layer AdaIN that describes the global color style of the references extracted by an embedder network. The temporal refinement network learns spatiotemporal features through 3D convolutions to ensure the temporal color consistency of the results. Our model can achieve even better coloring results by fine-tuning the parameters with only a small number of samples when dealing with an animation of a new style. To evaluate our method, we build a line art coloring dataset. Experiments show that our method achieves the best performance on line art video coloring compared to the current state-of-the-art methods.
关键词	Image color analysis Art Animation Feature extraction Three-dimensional displays Transforms Color Line art colorization color transform temporal coherence few shot learning
DOI	10.1109/TVCG.2022.3146000
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61972379] ; National Natural Science Foundation of China[62102403] ; National Natural Science Foundation of China[61872440] ; Science and Technology Service Network Initiative, Chinese Academy of Sciences[KFJ-STS-QYZD-2021-11-001] ; Royal Society Newton Advanced Fellowship[NAF\R2\192151] ; Royal Society[IES\R1\180126] ; Youth Innovation Promotion Association CAS ; Marsden Fund Council[MFP-20-VUW-180]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:000981880500011
出版者	IEEE COMPUTER SOC
引用统计	被引频次：3[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21426
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.North China Elect Power Univ, Beijing 102206, Peoples R China 2.Beihang Univ, Beijing 100191, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100190, Peoples R China 4.Univ Chinese Acad Sci, Beijing 100190, Peoples R China 5.Cardiff Univ, Sch Comp Sci & Informat, Cardiff CF10 3AT, Wales 6.Victoria Univ Wellington, Sch Engn & Comp Sci, Wellington 6012, New Zealand
推荐引用方式 GB/T 7714	Shi, Min,Zhang, Jia-Qi,Chen, Shu-Yu,et al. Reference-Based Deep Line Art Video Colorization[J]. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,2023,29(6):2965-2979.
APA	Shi, Min,Zhang, Jia-Qi,Chen, Shu-Yu,Gao, Lin,Lai, Yu-Kun,&Zhang, Fang-Lue.(2023).Reference-Based Deep Line Art Video Colorization.IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,29(6),2965-2979.
MLA	Shi, Min,et al."Reference-Based Deep Line Art Video Colorization".IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 29.6(2023):2965-2979.