Translating Words to Worlds: Zero-Shot Synthesis of 3D Terrain from Textual Descriptions Using Large Language Models

doi:10.3390/app14083257

	Translating Words to Worlds: Zero-Shot Synthesis of 3D Terrain from Textual Descriptions Using Large Language Models
	Zhang, Guangzi 1; Chen, Lizhe 1; Zhang, Yu 1; Liu, Yan 1; Ge, Yuyao 1,2,3; Cai, Xingquan 1
	2024-04-01
发表期刊	APPLIED SCIENCES-BASEL
卷号	14 期号:8 页码:19
摘要	The current research on text-guided 3D synthesis predominantly utilizes complex diffusion models, posing significant challenges in tasks like terrain generation. This study ventures into the direct synthesis of text-to-3D terrain in a zero-shot fashion, circumventing the need for diffusion models. By exploiting the large language model's inherent spatial awareness, we innovatively formulate a method to update existing 3D models through text, thereby enhancing their accuracy. Specifically, we introduce a Gaussian-Voronoi map data structure that converts simplistic map summaries into detailed terrain heightmaps. Employing a chain-of-thought behavior tree approach, which combines action chains and thought trees, the model is guided to analyze a variety of textual inputs and extract relevant terrain data, effectively bridging the gap between textual descriptions and 3D models. Furthermore, we develop a text-terrain re-editing technique utilizing multiagent reasoning, allowing for the dynamic update of the terrain's representational structure. Our experimental results indicate that this method proficiently interprets the spatial information embedded in the text and generates controllable 3D terrains with superior visual quality.
关键词	text-to-3D multiagent chain-of-thought large language model LLM's spatial awareness terrain synthesis
DOI	10.3390/app14083257
收录类别	SCI
语种	英语
资助项目	Project of Humanities and Social Sciences of the Ministry of Education in China
WOS研究方向	Chemistry ; Engineering ; Materials Science ; Physics
WOS类目	Chemistry, Multidisciplinary ; Engineering, Multidisciplinary ; Materials Science, Multidisciplinary ; Physics, Applied
WOS记录号	WOS:001210735000001
出版者	MDPI
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/38999
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Guangzi; Chen, Lizhe
作者单位	1.North China Univ Technol, Coll Informat, Beijing 100144, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab AI Secur, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Beijing 101408, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Guangzi,Chen, Lizhe,Zhang, Yu,et al. Translating Words to Worlds: Zero-Shot Synthesis of 3D Terrain from Textual Descriptions Using Large Language Models[J]. APPLIED SCIENCES-BASEL,2024,14(8):19.
APA	Zhang, Guangzi,Chen, Lizhe,Zhang, Yu,Liu, Yan,Ge, Yuyao,&Cai, Xingquan.(2024).Translating Words to Worlds: Zero-Shot Synthesis of 3D Terrain from Textual Descriptions Using Large Language Models.APPLIED SCIENCES-BASEL,14(8),19.
MLA	Zhang, Guangzi,et al."Translating Words to Worlds: Zero-Shot Synthesis of 3D Terrain from Textual Descriptions Using Large Language Models".APPLIED SCIENCES-BASEL 14.8(2024):19.