SAC-GAN: Structure-Aware Image Composition

doi:10.1109/TVCG.2022.3226689

	SAC-GAN: Structure-Aware Image Composition
	Zhou, Hang 1; Ma, Rui 2,3; Zhang, Ling-Xiao 4; Gao, Lin 4; Mahdavi-Amiri, Ali 1; Zhang, Hao 1
	2024-07-01
发表期刊	IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
ISSN	1077-2626
卷号	30 期号:7 页码:3151-3165
摘要	We introduce an end-to-end learning framework for image-to-image composition, aiming to plausibly compose an object represented as a cropped patch from an object image into a background scene image. As our approach emphasizes more on semantic and structural coherence of the composed images, rather than their pixel-level RGB accuracies, we tailor the input and output of our network with structure-aware features and design our network losses accordingly, with ground truth established in a self-supervised setting through the object cropping. Specifically, our network takes the semantic layout features from the input scene image, features encoded from the edges and silhouette in the input object patch, as well as a latent code as inputs, and generates a 2D spatial affine transform defining the translation and scaling of the object patch. The learned parameters are further fed into a differentiable spatial transformer network to transform the object patch into the target image, where our model is trained adversarially using an affine transform discriminator and a layout discriminator. We evaluate our network, coined SAC-GAN, for various image composition scenarios in terms of quality, composability, and generalizability of the composite images. Comparisons are made to state-of-the-art alternatives, including Instance Insertion, ST-GAN, CompGAN and PlaceNet, confirming superiority of our method.
关键词	Layout Transforms Semantics Three-dimensional displays Image edge detection Codes Coherence Structure-aware image composition self-supervision GANs
DOI	10.1109/TVCG.2022.3226689
收录类别	SCI
语种	英语
资助项目	NSERC Discovery[611370] ; National Natural Science Funds of China[62202199]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001258936700035
出版者	IEEE COMPUTER SOC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39835
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Ma, Rui
作者单位	1.Simon Fraser Univ, Sch Comp Sci, Burnaby, BC V5A 1S6, Canada 2.Jilin Univ, Sch Artificial Intelligence, Changchun 130012, Peoples R China 3.Minist Educ, Engn Res Ctr Knowledge Driven Human Machine Intell, Changchun 130012, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China
推荐引用方式 GB/T 7714	Zhou, Hang,Ma, Rui,Zhang, Ling-Xiao,et al. SAC-GAN: Structure-Aware Image Composition[J]. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,2024,30(7):3151-3165.
APA	Zhou, Hang,Ma, Rui,Zhang, Ling-Xiao,Gao, Lin,Mahdavi-Amiri, Ali,&Zhang, Hao.(2024).SAC-GAN: Structure-Aware Image Composition.IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,30(7),3151-3165.
MLA	Zhou, Hang,et al."SAC-GAN: Structure-Aware Image Composition".IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 30.7(2024):3151-3165.