Semantic-Aware Visual Decomposition for Image Coding

doi:10.1007/s11263-023-01809-7

	Semantic-Aware Visual Decomposition for Image Coding
	Chang, Jianhui 1; Zhang, Jian 2; Li, Jiguo 3; Wang, Shiqi 4; Mao, Qi 5; Jia, Chuanmin 1; Ma, Siwei 1; Gao, Wen 1
	2023-06-02
发表期刊	INTERNATIONAL JOURNAL OF COMPUTER VISION
ISSN	0920-5691
页码	23
摘要	In this paper, we propose a novel image coding framework with semantic-aware visual decomposition towards extremely low bitrate compression. In particular, an input image is analyzed into a semantic map as structural representation and semantic-wise texture representation and further compressed into bitstreams at the encoder side. On the decoder side, the received bitstreams of dual-layer representations are decoded and reconstructed for target image synthesis with generative models. Moreover, the attention mechanism is introduced into the model architecture for texture representation modeling and a coherency regularization is proposed to further optimize the texture representation space by aligning the representation space with the source pixel space for higher synthesis quality. Besides, we also propose a cross-channel entropy module and control the quantization scale to facilitate rate-distortion optimization. Upon compressing the decomposed components into the bitstream, the simple yet effective representation philosophy benefits image compression in many aspects. First, in terms of compression performance, compact representations, and high visual synthesis quality can bring remarkable advantages. Second, the proposed framework yields a physically explainable bitstream composed of the structural segment and semantic-wise texture segments. Third and most importantly, subsequent vision tasks (e.g., content manipulation) can receive fundamental support from the semantic-aware visual decomposition and synthesis mechanism. Extensive experimental results demonstrate the superiority of the proposed framework towards efficient visual representation learning, high efficiency image compression (< 0.1 bpp), and intelligent visual applications (e.g., manipulation and analysis).
关键词	Image coding Semantic-aware visual decomposition Structure-texture Coherency regularization Extremely low bitrate
DOI	10.1007/s11263-023-01809-7
收录类别	SCI
语种	英语
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001000503000001
出版者	SPRINGER
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21464
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Jian; Ma, Siwei
作者单位	1.Peking Univ, Natl Engn Res Ctr Visual Technol, Sch Comp Sci, Beijing 100871, Peoples R China 2.Peking Univ, Sch Elect & Comp Engn, Shenzhen Grad Sch, Shenzhen 518055, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 4.City Univ Hong Kong, Dept Comp Sci, Hong Kong, Peoples R China 5.Commun Univ China, State Key Lab Media Convergence & Commun, Beijing 100024, Peoples R China
推荐引用方式 GB/T 7714	Chang, Jianhui,Zhang, Jian,Li, Jiguo,et al. Semantic-Aware Visual Decomposition for Image Coding[J]. INTERNATIONAL JOURNAL OF COMPUTER VISION,2023:23.
APA	Chang, Jianhui.,Zhang, Jian.,Li, Jiguo.,Wang, Shiqi.,Mao, Qi.,...&Gao, Wen.(2023).Semantic-Aware Visual Decomposition for Image Coding.INTERNATIONAL JOURNAL OF COMPUTER VISION,23.
MLA	Chang, Jianhui,et al."Semantic-Aware Visual Decomposition for Image Coding".INTERNATIONAL JOURNAL OF COMPUTER VISION (2023):23.