SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo With Depth Restoration and Occlusion Constraint

doi:10.1109/TCSVT.2025.3574473

	SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo With Depth Restoration and Occlusion Constraint
	Yuan, Zhenlong 1,2; Yang, Zhidong 1,2; Cai, Yujun 3; Wu, Kuangxin ; Liu, Mufan 4; Zhang, Dapeng 5; Jiang, Hao 1,2; Li, Zhaoxin 6,7; Wang, Zhaoqi 1,2
	2025-11-01
发表期刊	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN	1051-8215
卷号	35 期号:11 页码:11244-11257
摘要	Recently, patch-deformation methods have exhibited significant effectiveness in multi-view stereo owing to the deformable and expandable patches in reconstructing textureless areas. However, existing approaches neglect to address the problem of deformation instability caused by easily overlooked edge-skipping, potentially leading to matching distortions, thus leaving room for further improvement. To fill this gap, we propose SED-MVS, which adopts panoptic segmentation and multi-trajectory diffusion strategy for segmentation-driven and edge-aligned patch deformation. Specifically, to prevent unanticipated edge-skipping, we first employ SAM2 for panoptic segmentation as depth-edge guidance to guide patch deformation, followed by multi-trajectory diffusion strategy to ensure patches are comprehensively aligned with depth edges. Moreover, to avoid potential inaccuracy of random initialization, we combine both sparse points from LoFTR and monocular depth map from DepthAnything V2 to restore reliable and realistic depth map for initialization and supervised guidance. Finally, we integrate the segmentation image with the monocular depth map to exploit inter-instance occlusion relationship, then further regard them as occlusion map to implement two distinct edge constraint, thereby facilitating occlusion-aware patch deformation. Extensive results on ETH3D, Tanks & Temples, BlendedMVS, Strecha and DL3DV-10K datasets validate the state-of-the-art performance and robust generalization capability of our proposed method.
关键词	Deformation Image reconstruction Costs Image edge detection Image segmentation Reliability Three-dimensional displays Image restoration Feature extraction Electronic mail Multi-view stereo 3D reconstruction segmentation patch deformation occlusion
DOI	10.1109/TCSVT.2025.3574473
收录类别	SCI
语种	英语
资助项目	Strategic Priority Research Program of the Chinese Academy of Sciences[XDA0450203] ; National Natural Science Foundation of China[62172392] ; Beijing Smart Agriculture innovation Consortium Project[BAlC10-2024] ; Basic Research Center, Innovation Program of Chinese Academy of Agricultural Sciences[CAAS-BRC-SAE-2025-01] ; Basic Research Center, Innovation Program of Chinese Academy of Agricultural Sciences[CAAS-ASTIP-2025-AII]
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:001606669700009
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41599
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Jiang, Hao
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Univ Queensland, Sch Elect Engn & Comp Sci, Brisbane, Qld 4072, Australia 4.Shanghai Jiao Tong Univ, Cooperat MediaNet Innovat Ctr, Shanghai 200240, Peoples R China 5.Lanzhou Univ, Sch Informat Sci & Engn, DSLAB, Lanzhou 730000, Peoples R China 6.Chinese Acad Agr Sci, Agr Informat Inst, Beijing 100081, Peoples R China 7.Minist Agr & Rural Affairs, Key Lab Agr Big Data, Beijing 100081, Peoples R China
推荐引用方式 GB/T 7714	Yuan, Zhenlong,Yang, Zhidong,Cai, Yujun,et al. SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo With Depth Restoration and Occlusion Constraint[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2025,35(11):11244-11257.
APA	Yuan, Zhenlong.,Yang, Zhidong.,Cai, Yujun.,Wu, Kuangxin.,Liu, Mufan.,...&Wang, Zhaoqi.(2025).SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo With Depth Restoration and Occlusion Constraint.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,35(11),11244-11257.
MLA	Yuan, Zhenlong,et al."SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo With Depth Restoration and Occlusion Constraint".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 35.11(2025):11244-11257.