Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog

doi:10.1109/TASLP.2021.3065823

	Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog
	Li, Zekang 1; Li, Zongjia 2; Zhang, Jinchao 3; Feng, Yang 1; Zhou, Jie 3
	2021
发表期刊	IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
ISSN	2329-9290
卷号	29 页码:2476-2483
摘要	Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). There are two challenges in this task: 1) making effective interaction among different modalities; 2) better understanding dialogues and generating informative responses. To tackle the challenges, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses by leveraging the pre-trained language model. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task, which allows fine-tuning language models to capture information across both visual and textual modalities. Our system achieves the best performance in the objective evaluation in both DSTC7-AVSD and DSTC8-AVSD dataset and achieves an impressive 98.4% of the human performance based on human ratings in the DSTC8-AVSD challenge.
关键词	Task analysis Feature extraction Visualization Speech processing History Social networking (online) Pattern recognition Dialogue System Multimodal Natural Language Processing Video Understanding
DOI	10.1109/TASLP.2021.3065823
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2018AAA0102502]
WOS研究方向	Acoustics ; Engineering
WOS类目	Acoustics ; Engineering, Electrical & Electronic
WOS记录号	WOS:000683990800006
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：23[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/17228
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Feng, Yang
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Peking Univ, Sch EECS, Beijing 100080, Peoples R China 3.Tencent Inc, WeChat AI, Pattern Recognit Ctr, Beijing 100080, Peoples R China
推荐引用方式 GB/T 7714	Li, Zekang,Li, Zongjia,Zhang, Jinchao,et al. Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog[J]. IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING,2021,29:2476-2483.
APA	Li, Zekang,Li, Zongjia,Zhang, Jinchao,Feng, Yang,&Zhou, Jie.(2021).Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog.IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING,29,2476-2483.
MLA	Li, Zekang,et al."Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog".IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING 29(2021):2476-2483.