Dual-Stream Recurrent Neural Network for Video Captioning

doi:10.1109/TCSVT.2018.2867286

	Dual-Stream Recurrent Neural Network for Video Captioning
	Xu, Ning 1; Liu, An-An 1; Wong, Yongkang 2; Zhang, Yongdong 3,4; Nie, Weizhi 1; Su, Yuting 1; Kankanhalli, Mohan 5
	2019-08-01
发表期刊	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN	1051-8215
卷号	29 期号:8 页码:2482-2493
摘要	Recent progress in using recurrent neural networks (RNNs) for video description has attracted an increasing interest, due to its capability to encode a sequence of frames for caption generation. While existing methods have studied various features (e.g., CNN, 3D CNN, and semantic attributes) for visual encoding, the representation and fusion of heterogeneous information from multi-modal spaces have not fully explored. Consider that different modalities are often asynchronous, frame-level multi-modal fusion (e.g., concatenation and linear fusion) will negatively influence each modality. In this paper, we propose a dual-stream RNN (DS-RNN) framework to jointly discover and integrate the hidden states of both visual and semantic streams for video caption generation. First, an encoding RNN is used for each stream to flexibly exploit the hidden states of respective modality. Specifically, we proposed an attentive multi-grained encoder module to enhance the local feature learning with global semantics feature. Then, a dual-stream decoder is deployed to integrate the asynchronous yet complementary sequential hidden states from both streams for caption generation. Extensive experiments on three benchmark datasets, namely, MSVD, MSR-VTT, and MPII-MD, show that DS-RNN achieves competitive performance against the state-of-the-art. Additional ablation studies were conducted on various variants of the proposed DS-RNN.
关键词	Video captioning hidden state fusion dual stream recurrent neural network attention module
DOI	10.1109/TCSVT.2018.2867286
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61772359] ; National Natural Science Foundation of China[61472275] ; National Natural Science Foundation of China[61525206] ; National Natural Science Foundation of China[61502337] ; National Key Research and Development Program of China[2017YFC0820600] ; National Defense Science and Technology Fund for Distinguished Young Scholars[2017-JCJQ-ZQ-022] ; National Research Foundation, Prime Minister's Office, Singapore, under its International Research Centre in Singapore Funding Initiative
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000480310500022
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：66[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/4435
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, An-An
作者单位	1.Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China 2.Natl Univ Singapore, Smart Syst Inst, Singapore 119077, Singapore 3.Univ Sci & Technol China, Sch Informat Sci & Technol, Hefei 230026, Anhui, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 5.Natl Univ Singapore, Sch Comp, Singapore 119077, Singapore
推荐引用方式 GB/T 7714	Xu, Ning,Liu, An-An,Wong, Yongkang,et al. Dual-Stream Recurrent Neural Network for Video Captioning[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2019,29(8):2482-2493.
APA	Xu, Ning.,Liu, An-An.,Wong, Yongkang.,Zhang, Yongdong.,Nie, Weizhi.,...&Kankanhalli, Mohan.(2019).Dual-Stream Recurrent Neural Network for Video Captioning.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,29(8),2482-2493.
MLA	Xu, Ning,et al."Dual-Stream Recurrent Neural Network for Video Captioning".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29.8(2019):2482-2493.