Cross Modal Compression With Variable Rate Prompt

doi:10.1109/TMM.2023.3310657

CSpace

	Cross Modal Compression With Variable Rate Prompt
	Gao, Junlong 1; Li, Jiguo 2,3; Jia, Chuanmin 4; Wang, Shanshe 1; Ma, Siwei 1; Gao, Wen 1
	2024
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
卷号	26 页码:3444-3456
摘要	Traditional image/video compression compresses the highly redundant visual data while preserving signal fidelity. Recently, cross modal compression (CMC) is proposed to compress the data into a compact, human-comprehensible domain (such as text) with an ultra-high compression ratio while preserving semantic fidelity for machine analysis and semantic monitoring. CMC is with a constant rate because the CMC encoder can only represent the data with a fixed grain. But in practice, variable rate is necessary due to the complicated and dynamic transmission conditions, the different storage mediums, and the diverse levels of application requirements. To deal with this problem, in this paper, we propose variable rate cross modal compression (VR-CMC), where we introduce variable rate prompt to represent the data with different grains. Variable rate prompt is composed of three strategies. Specifically, 1) target length prompt (TLP) introduces the target length into the language prompt to guide the generation of the text representation; 2) decaying EOS probability (DEP) exponentially decays the probability of the EOS token with regard to the decoding step and target length, where the EOS (end-of-sequence) token is a special token indicating the end of the text; 3) text augmentation (TA) enriches the training data and makes the text representation length more balanced when training. Experimental results show that our proposed VR-CMC can effectively control the rate in the CMC framework and achieve state-of-the-art performance on MSCOCO and IM2P datasets.
关键词	Cross modal compression semantic fidelity variable rate prompt
DOI	10.1109/TMM.2023.3310657
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:001165348200028
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/38739
专题	中国科学院计算技术研究所
通讯作者	Jia, Chuanmin
作者单位	1.Peking Univ, Natl Engn Res Ctr Visual Technol, Sch Comp Sci, Beijing 100871, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 4.Peking Univ, Wangxuan Inst Comp Technol, Beijing 100871, Peoples R China
推荐引用方式 GB/T 7714	Gao, Junlong,Li, Jiguo,Jia, Chuanmin,et al. Cross Modal Compression With Variable Rate Prompt[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2024,26:3444-3456.
APA	Gao, Junlong,Li, Jiguo,Jia, Chuanmin,Wang, Shanshe,Ma, Siwei,&Gao, Wen.(2024).Cross Modal Compression With Variable Rate Prompt.IEEE TRANSACTIONS ON MULTIMEDIA,26,3444-3456.
MLA	Gao, Junlong,et al."Cross Modal Compression With Variable Rate Prompt".IEEE TRANSACTIONS ON MULTIMEDIA 26(2024):3444-3456.