Adaptive Spatial Location With Balanced Loss for Video Captioning

doi:10.1109/TCSVT.2020.3045735

	Adaptive Spatial Location With Balanced Loss for Video Captioning
	Li, Linghui 1; Zhang, Yongdong 2; Tang, Sheng 2; Xie, Lingxi 3; Li, Xiaoyong 1; Tian, Qi 3
	2022
发表期刊	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN	1051-8215
卷号	32 期号:1 页码:17-30
摘要	Many pioneering approaches have verified the effectiveness of utilizing the global temporal and local object information for video understanding tasks and have achieved significant progress. However, existing methods utilize object detectors to extract all objects overall video frames. This may bring performance degradation due to the information redundancy both spatially and temporally. To address this problem, we propose an adaptive spatial location module for the video captioning task which dynamically predicts an important position of each video frame in the procedure of generating the description sentence. The proposed adaptive spatial location method not only makes our model focus on local object information, but also reduces time and memory consumption brought by the temporal redundancy in extensive video frames and improves the accuracy of generated description. Besides, we propose a balanced loss function to address the class imbalance problem existing in training data. The proposed balanced loss assigns different weight to each word of ground-truth sentence in the training process which can generate more diversified description sentences. Extensive experimental results on the MSVD and MSR-VTT dataset show that the proposed method achieves competitive performance compared to state-of-the-art methods.
关键词	Task analysis Redundancy Feature extraction Visualization Detectors Computer vision Training Convolutional neural network recurrent neural network video captioning adaptive spatial location balanced loss
DOI	10.1109/TCSVT.2020.3045735
收录类别	SCI
语种	英语
资助项目	National Nature Science Foundation of China[61525206] ; National Nature Science Foundation of China[61871004] ; 242 Project[2020A077] ; National Natural Science Foundation of China (NSFC)-General Technology Fundamental Research Joint Fund[U1836215] ; China Postdoctoral Science Foundation[2020TQ0055] ; Foundation of Key Laboratory of Trustworthy Distributed Computing and Service (BUPT), Ministry of Education
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000742183600006
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：13[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/18258
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Yongdong
作者单位	1.Beijing Univ Posts & Telecommun, Key Lab Trustworthy Distributed Comp & Serv, Minist Educ, Beijing 100876, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 3.Huawei Technol Inc, Shenzhen 518129, Peoples R China
推荐引用方式 GB/T 7714	Li, Linghui,Zhang, Yongdong,Tang, Sheng,et al. Adaptive Spatial Location With Balanced Loss for Video Captioning[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2022,32(1):17-30.
APA	Li, Linghui,Zhang, Yongdong,Tang, Sheng,Xie, Lingxi,Li, Xiaoyong,&Tian, Qi.(2022).Adaptive Spatial Location With Balanced Loss for Video Captioning.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,32(1),17-30.
MLA	Li, Linghui,et al."Adaptive Spatial Location With Balanced Loss for Video Captioning".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 32.1(2022):17-30.