SketchDream: Sketch-based Text-to-3D Generation and Editing

doi:10.1145/3658120

	SketchDream: Sketch-based Text-to-3D Generation and Editing
	Liu, Feng-Lin 1,2; Fu, Hongbo 3,4; Lai, Yu-Kun 5; Gao, Lin 1,2
	2024-07-01
发表期刊	ACM TRANSACTIONS ON GRAPHICS
ISSN	0730-0301
卷号	43 期号:4 页码:13
摘要	Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketch-based local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.
关键词	sketch-based interaction diffusion models neural radiance fields 3D generation
DOI	10.1145/3658120
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62322210] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; Beijing Municipal Science and Technology Commission[Z231100005923031]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001289270900011
出版者	ASSOC COMPUTING MACHINERY
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39540
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China 3.City Univ Hong Kong, SCM, Hong Kong, Peoples R China 4.HKUST, EMIA, Hong Kong, Peoples R China 5.Cardiff Univ, Sch Comp Sci & Informat, Cardiff, Wales
推荐引用方式 GB/T 7714	Liu, Feng-Lin,Fu, Hongbo,Lai, Yu-Kun,et al. SketchDream: Sketch-based Text-to-3D Generation and Editing[J]. ACM TRANSACTIONS ON GRAPHICS,2024,43(4):13.
APA	Liu, Feng-Lin,Fu, Hongbo,Lai, Yu-Kun,&Gao, Lin.(2024).SketchDream: Sketch-based Text-to-3D Generation and Editing.ACM TRANSACTIONS ON GRAPHICS,43(4),13.
MLA	Liu, Feng-Lin,et al."SketchDream: Sketch-based Text-to-3D Generation and Editing".ACM TRANSACTIONS ON GRAPHICS 43.4(2024):13.