Learning Hierarchical Modular Networks for Video Captioning

doi:10.1109/TPAMI.2023.3327677

CSpace

	Learning Hierarchical Modular Networks for Video Captioning
	Li, Guorong 1; Ye, Hanhua 1; Qi, Yuankai 2; Wang, Shuhui 3; Qing, Laiyun 1; Huang, Qingming 1; Yang, Ming-Hsuan 4,5,6
	2024-02-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	46 期号:2 页码:1049-1064
摘要	Video captioning aims to generate natural language descriptions for a given video clip. Existing methods mainly focus on end-to-end representation learning via word-by-word comparison between predicted captions and ground-truth texts. Although significant progress has been made, such supervised approaches neglect semantic alignment between visual and linguistic entities, which may negatively affect the generated captions. In this work, we propose a hierarchical modular network to bridge video representations and linguistic semantics at four granularities before generating captions: entity, verb, predicate, and sentence. Each level is implemented by one module to embed corresponding semantics into video representations. Additionally, we present a reinforcement learning module based on the scene graph of captions to better measure sentence similarity. Extensive experimental results show that the proposed method performs favorably against the state-of-the-art models on three widely-used benchmark datasets, including microsoft research video description corpus (MSVD), MSR-video to text (MSR-VTT), and video-and-TEXt (VATEX).
关键词	Video captioning hierarchical modular network scene-graph reward reinforcement learning
DOI	10.1109/TPAMI.2023.3327677
收录类别	SCI
语种	英语
资助项目	National Key R#x0026;D Program of China
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001140839000013
出版者	IEEE COMPUTER SOC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/38406
专题	中国科学院计算技术研究所
通讯作者	Qi, Yuankai; Qing, Laiyun; Huang, Qingming
作者单位	1.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Key Lab Big Data Min & Knowledge Management, Beijing 100049, Peoples R China 2.Univ Adelaide, Australian Inst Machine Learning, Adelaide, SA 5005, Australia 3.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100045, Peoples R China 4.Univ Calif Merced, Merced, CA 95343 USA 5.Yonsei Univ, Seoul 03722, South Korea 6.Google, Mountain View, CA 94043 USA
推荐引用方式 GB/T 7714	Li, Guorong,Ye, Hanhua,Qi, Yuankai,et al. Learning Hierarchical Modular Networks for Video Captioning[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2024,46(2):1049-1064.
APA	Li, Guorong.,Ye, Hanhua.,Qi, Yuankai.,Wang, Shuhui.,Qing, Laiyun.,...&Yang, Ming-Hsuan.(2024).Learning Hierarchical Modular Networks for Video Captioning.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,46(2),1049-1064.
MLA	Li, Guorong,et al."Learning Hierarchical Modular Networks for Video Captioning".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 46.2(2024):1049-1064.