Semantic and Relation Modulation for Audio-Visual Event Localization

doi:10.1109/TPAMI.2022.3226328

	Semantic and Relation Modulation for Audio-Visual Event Localization
	Wang, Hao 1; Zha, Zheng-Jun 1; Li, Liang 2; Chen, Xuejin 1; Luo, Jiebo 3
	2023-06-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	45 期号:6 页码:7711-7725
摘要	We study the problem of localizing audio-visual events that are both audible and visible in a video. Existing works focus on encoding and aligning audio and visual features at the segment level while neglecting informative correlation between segments of the two modalities and between multi-scale event proposals. We propose a novel Semantic and Relation Modulation Network (SRMN) to learn the above correlation and leverage it to modulate the related auditory, visual, and fused features. In particular, for semantic modulation, we propose intra-modal normalization and cross-modal normalization. The former modulates features of a single modality with the event-relevant semantic guidance of the same modality. The latter modulates features of two modalities by establishing and exploiting the cross-modal relationship. For relation modulation, we propose a multi-scale proposal modulating module and a multi-alignment segment modulating module to introduce multi-scale event proposals and enable dense matching between cross-modal segments, which strengthen correlations between successive segments within one proposal and between all segments. With the features modulated by the correlation information regarding audio-visual events, SRMN performs accurate event localization. Extensive experiments conducted on the public AVE dataset demonstrate that our method outperforms the state-of-the-art methods in both supervised event localization and cross-modality localization tasks.
关键词	Visualization Location awareness Correlation Proposals Semantics Task analysis Modulation Audio-visual learning event localization normalization
DOI	10.1109/TPAMI.2022.3226328
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2020AAA0105702] ; National Natural Science Foundation of China (NSFC)[62225207] ; National Natural Science Foundation of China (NSFC)[U19B2038] ; University Synergy Innovation Program of Anhui Province[GXXT-2019-025]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000982475600073
出版者	IEEE COMPUTER SOC
引用统计	被引频次：5[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21240
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zha, Zheng-Jun
作者单位	1.Univ Sci & Technol China, Sch Informat Sci & Technol, Hefei 230052, Anhui, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China 3.Univ Rochester, Dept Comp Sci, Rochester, NY 14627 USA
推荐引用方式 GB/T 7714	Wang, Hao,Zha, Zheng-Jun,Li, Liang,et al. Semantic and Relation Modulation for Audio-Visual Event Localization[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(6):7711-7725.
APA	Wang, Hao,Zha, Zheng-Jun,Li, Liang,Chen, Xuejin,&Luo, Jiebo.(2023).Semantic and Relation Modulation for Audio-Visual Event Localization.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(6),7711-7725.
MLA	Wang, Hao,et al."Semantic and Relation Modulation for Audio-Visual Event Localization".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.6(2023):7711-7725.