Refocused Attention: Long Short-Term Rewards Guided Video Captioning

doi:10.1007/s11063-019-10030-y

	Refocused Attention: Long Short-Term Rewards Guided Video Captioning
	Dong, Jiarong 1,2; Gao, Ke 1; Chen, Xiaokai 1,2; Cao, Juan 1
	2020-10-01
发表期刊	NEURAL PROCESSING LETTERS
ISSN	1370-4621
卷号	52 期号:2 页码:935-948
摘要	The adaptive cooperation of visual model and language model is essential for video captioning. However, due to the lack of proper guidance for each time step in end-to-end training, the over-dependence of language model often results in the invalidation of attention-based visual model, which is called 'Attention Defocus' problem in this paper. Based on an important observation that the recognition precision of entity word can reflect the effectiveness of the visual model, we propose a novel strategy called refocused attention to optimize the training and cooperating of visual model and language model, using ingenious guidance at appropriate time step. The strategy consists of a short-term-reward guided local entity recognition and a long-term-reward guided global relation understanding, neither requires any external training data. Moreover, a framework with hierarchical visual representations and hierarchical attention is established to fully exploit the potential strength of the proposed learning strategy. Extensive experiments demonstrate that the ingenious guidance strategy together with the optimized structure outperform state-of-the-art video captioning methods with relative improvements 7.7% in BLEU-4 and 5.0% in CIDEr-D on MSVD dataset, even without multi-modal features.
关键词	Video captioning Hierarchical attention Reinforcement learning Reward
DOI	10.1007/s11063-019-10030-y
收录类别	SCI
语种	英语
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000574296700001
出版者	SPRINGER
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/15638
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Ke
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Dong, Jiarong,Gao, Ke,Chen, Xiaokai,et al. Refocused Attention: Long Short-Term Rewards Guided Video Captioning[J]. NEURAL PROCESSING LETTERS,2020,52(2):935-948.
APA	Dong, Jiarong,Gao, Ke,Chen, Xiaokai,&Cao, Juan.(2020).Refocused Attention: Long Short-Term Rewards Guided Video Captioning.NEURAL PROCESSING LETTERS,52(2),935-948.
MLA	Dong, Jiarong,et al."Refocused Attention: Long Short-Term Rewards Guided Video Captioning".NEURAL PROCESSING LETTERS 52.2(2020):935-948.