Synthesizing Knowledge-Enhanced Features for Real-World Zero-Shot Food Detection

doi:10.1109/TIP.2024.3360899

	Synthesizing Knowledge-Enhanced Features for Real-World Zero-Shot Food Detection
	Zhou, Pengfei 1,2; Min, Weiqing 1,2,3; Song, Jiajun 1,2; Zhang, Yang 1,2; Jiang, Shuqiang 1,2,3
	2024
发表期刊	IEEE TRANSACTIONS ON IMAGE PROCESSING
ISSN	1057-7149
卷号	33 页码:1285-1298
摘要	Food computing brings various perspectives to computer vision like vision-based food analysis for nutrition and health. As a fundamental task in food computing, food detection needs Zero-Shot Detection (ZSD) on novel unseen food objects to support real-world scenarios, such as intelligent kitchens and smart restaurants. Therefore, we first benchmark the task of Zero-Shot Food Detection (ZSFD) by introducing FOWA dataset with rich attribute annotations. Unlike ZSD, fine-grained problems in ZSFD like inter-class similarity make synthesized features inseparable. The complexity of food semantic attributes further makes it more difficult for current ZSD methods to distinguish various food categories. To address these problems, we propose a novel framework ZSFDet to tackle fine-grained problems by exploiting the interaction between complex attributes. Specifically, we model the correlation between food categories and attributes in ZSFDet by multi-source graphs to provide prior knowledge for distinguishing fine-grained features. Within ZSFDet, Knowledge-Enhanced Feature Synthesizer (KEFS) learns knowledge representation from multiple sources (e.g., ingredients correlation from knowledge graph) via the multi-source graph fusion. Conditioned on the fusion of semantic knowledge representation, the region feature diffusion model in KEFS can generate fine-grained features for training the effective zero-shot detector. Extensive evaluations demonstrate the superior performance of our method ZSFDet on FOWA and the widely-used food dataset UECFOOD-256, with significant improvements by 1.8% and 3.7% ZSD mAP compared with the strong baseline RRFS. Further experiments on PASCAL VOC and MS COCO prove that enhancement of the semantic knowledge can also improve the performance on general ZSD. Code and dataset are available at https://github.com/LanceZPF/KEFS.
关键词	Semantics Feature extraction Visualization Annotations Correlation Training Task analysis Food detection zero-shot detection food computing object detection zero-shot learning
DOI	10.1109/TIP.2024.3360899
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001173850100003
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：3[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/38711
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Min, Weiqing
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Coll Comp Sci & Technol, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Inst Intelligent Comp Technol, Suzhou 215124, Peoples R China
推荐引用方式 GB/T 7714	Zhou, Pengfei,Min, Weiqing,Song, Jiajun,et al. Synthesizing Knowledge-Enhanced Features for Real-World Zero-Shot Food Detection[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2024,33:1285-1298.
APA	Zhou, Pengfei,Min, Weiqing,Song, Jiajun,Zhang, Yang,&Jiang, Shuqiang.(2024).Synthesizing Knowledge-Enhanced Features for Real-World Zero-Shot Food Detection.IEEE TRANSACTIONS ON IMAGE PROCESSING,33,1285-1298.
MLA	Zhou, Pengfei,et al."Synthesizing Knowledge-Enhanced Features for Real-World Zero-Shot Food Detection".IEEE TRANSACTIONS ON IMAGE PROCESSING 33(2024):1285-1298.