Linguistic Hallucination for Text-Based Video Retrieval

doi:10.1109/TCSVT.2024.3393843

	Linguistic Hallucination for Text-Based Video Retrieval
	Fang, Sheng 1,2; Dang, Tiantian 1,2; Wang, Shuhui 1; Huang, Qingming 1,2,3
	2024-10-01
发表期刊	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN	1051-8215
卷号	34 期号:10 页码:9692-9705
摘要	Text-based video retrieval is a crucial technology for video and multimodal applications. Although in traditional Text-Video Retrieval caption-video pairs are supposed to be entirely relevant, there is still information missing in text when compared to the video content. In a specific application scenario of Text-Video Retrieval, where the given caption corresponds to only a segment of the target video, the challenge of aligning two modalities becomes particularly difficult. To address this issue, we introduce context information as an auxiliary to enrich text representation and enhance alignment. In this work, we propose an effective Linguistic Hallucination framework, which incorporates context captions during training and replaces them with hallucinated textual representations predicted from the source sentence at inference. Specific hallucination loss and consistency loss are designed to supervise the learning process. Besides, Curriculum Learning is introduced at both data-level and model-level, which makes the training procedure more stable and improves the retrieval performance simultaneously. Extensive comparison experiments and ablation studies on benchmark datasets demonstrate the effectiveness of our framework. Moreover, we also apply our proposed method to other cross-modal tasks and the promising experimental results prove its generalization ability. Our codes and datasets are available in https://github.com/silenceFS/Linguistic-Hallucination.
关键词	Linguistics Training Testing Encoding Context modeling Feature extraction Task analysis Text-video retrieval partially relevant video retrieval linguistic hallucination curriculum learning
DOI	10.1109/TCSVT.2024.3393843
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2023YFC2508704] ; National Natural Science Foundation of China[62022083] ; National Natural Science Foundation of China[62236008] ; National Natural Science Foundation of China[U21B2038] ; National Natural Science Foundation of China[61931008] ; Fundamental Research Funds for Central Universities
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:001346503100066
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39486
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Shuhui
作者单位	1.Chinese Acad Sci, Key Lab Intelligent Informat Proc, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 101408, Peoples R China 3.Peng Cheng Lab, Shenzhen 518066, Peoples R China
推荐引用方式 GB/T 7714	Fang, Sheng,Dang, Tiantian,Wang, Shuhui,et al. Linguistic Hallucination for Text-Based Video Retrieval[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2024,34(10):9692-9705.
APA	Fang, Sheng,Dang, Tiantian,Wang, Shuhui,&Huang, Qingming.(2024).Linguistic Hallucination for Text-Based Video Retrieval.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,34(10),9692-9705.
MLA	Fang, Sheng,et al."Linguistic Hallucination for Text-Based Video Retrieval".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 34.10(2024):9692-9705.