Pano-SfMLearner: Self-Supervised Multi-Task Learning of Depth and Semantics in Panoramic Videos

doi:10.1109/LSP.2021.3073627

	Pano-SfMLearner: Self-Supervised Multi-Task Learning of Depth and Semantics in Panoramic Videos
	Liu, Mengyi 1; Wang, Shuhui 2; Guo, Yulan 3; He, Yuan 1; Xue, Hui 1
	2021
发表期刊	IEEE SIGNAL PROCESSING LETTERS
ISSN	1070-9908
卷号	28 页码:832-836
摘要	With the advent of virtual reality and augment reality applications, omnidirectional imaging and 360 degrees cameras become increasingly popular in many scenarios such as entertainment and autonomous systems. In this paper, we propose a self-supervised framework for multi-task learning on depth, camera motion and semantics frompanoramic videos. Specifically, our method is based on differentiable warping of adjacent views to the target. Two improvements are provided. First, we introduce a view synthesis module based on equirectangular projection to enable direct optimization on panoramic images. Second, we introduce a self-supervised segmentation branch to involve the constraint of semantic consistency for further improvement. Extensive experiments on two 360 degrees video and two 360 degrees image datasets demonstrate that ourmethod outperforms the state-of-the-art and achieves favorable cross-modality performance.
关键词	Depth estimation semantic segmentation pano-ramic video self-supervised learning
DOI	10.1109/LSP.2021.3073627
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[U20A20185] ; National Natural Science Foundation of China[61972435] ; Natural Science Foundation of Guangdong Province[2019A1515011271] ; Science and Technology Innovation Committee of Shenzhen Municipality[JCYJ20190807152209394]
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000648329700002
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：6[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/17768
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, Mengyi
作者单位	1.Alibaba Grp, Beijing 100102, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 3.Sun Yat Sen Univ, Sch Elect & Commun Engn, Guangzhou 510275, Peoples R China
推荐引用方式 GB/T 7714	Liu, Mengyi,Wang, Shuhui,Guo, Yulan,et al. Pano-SfMLearner: Self-Supervised Multi-Task Learning of Depth and Semantics in Panoramic Videos[J]. IEEE SIGNAL PROCESSING LETTERS,2021,28:832-836.
APA	Liu, Mengyi,Wang, Shuhui,Guo, Yulan,He, Yuan,&Xue, Hui.(2021).Pano-SfMLearner: Self-Supervised Multi-Task Learning of Depth and Semantics in Panoramic Videos.IEEE SIGNAL PROCESSING LETTERS,28,832-836.
MLA	Liu, Mengyi,et al."Pano-SfMLearner: Self-Supervised Multi-Task Learning of Depth and Semantics in Panoramic Videos".IEEE SIGNAL PROCESSING LETTERS 28(2021):832-836.