<i>A<SUP>2</SUP>Pt </i>: Anti-Associative Prompt Tuning for Open Set Visual Recognition

doi:10.1109/TMM.2023.3339387

	A²Pt : Anti-Associative Prompt Tuning for Open Set Visual Recognition
	Ren, Hairui 1; Tang, Fan 2; Pan, Xingjia 3; Cao, Juan 2; Dong, Weiming 4; Lin, Zhiwen 3; Yan, Ke 3; Xu, Changsheng
	2024
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
卷号	26 页码:8419-8431
摘要	Multi-modality pre-trained models (PTMs) have considerably boosted the performance on a broad range of computer vision topics. Still, they have not been explored purposefully in open set recognition (OSR) scenarios when applying PTMs to downstream recognition tasks. Directly fine/prompt tuning PTMs on closed-set classification tasks will inevitably suffer from data bias and always learn more or less target class-irrelevant co-occurring contextual information, which leads to over-confident predictions on unknown samples. In this paper, we propose a simple yet effective approach, termed Anti-Associative Prompt Tuning (A(2)Pt), toward learning compact and accurate class-related representation with few class-irrelevant associations from context using multi-modal priors. Specifically, a cross-modal guided activation module is adopted to refine the class-aware representation and suppress the associations from co-occurring contexts by involving text-modal information. We further design an anti-association calibration module to obtain compact class-aware and class-irrelevant representations, respectively, by introducing two additional object functions. Extensive experiments on publicly available benchmarks, including CIFAR series, TinyImageNet, and ImageNet-21K-P, show that the proposed A(2)Pt achieves substantial and consistent performance gains compared with both SOTA OSR and PTM prompt tuning approaches.
关键词	Tuning Neck Task analysis Image recognition Calibration Visualization Training Multi-modality Pre-trained models (PTMs) open set recognition (OSR) class-aware representation anti-associative prompt tuning (A(2)Pt)
DOI	10.1109/TMM.2023.3339387
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62102162] ; National Natural Science Foundation of China[U20B2070] ; National Natural Science Foundation of China[61832016] ; National Natural Science Foundation of China[61832002] ; Beijing Natural Science Foundation[L221013]
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:001283692500027
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39666
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Tang, Fan
作者单位	1.Jilin Univ, Sch Artificial Intelligence, Jilin 130012, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China 3.Youtu Lab, Tencent, Shanghai 201103, Peoples R China 4.Chinese Acad Sci, Inst Automat, Beijing 100045, Peoples R China
推荐引用方式 GB/T 7714	Ren, Hairui,Tang, Fan,Pan, Xingjia,et al. A²Pt : Anti-Associative Prompt Tuning for Open Set Visual Recognition[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2024,26:8419-8431.
APA	Ren, Hairui.,Tang, Fan.,Pan, Xingjia.,Cao, Juan.,Dong, Weiming.,...&Xu, Changsheng.(2024).A²Pt : Anti-Associative Prompt Tuning for Open Set Visual Recognition.IEEE TRANSACTIONS ON MULTIMEDIA,26,8419-8431.
MLA	Ren, Hairui,et al."A²Pt : Anti-Associative Prompt Tuning for Open Set Visual Recognition".IEEE TRANSACTIONS ON MULTIMEDIA 26(2024):8419-8431.