CSpace  > 中国科学院计算技术研究所期刊论文
Spatial-temporal transformer for end-to-end sign language recognition
Cui, Zhenchao1,2; Zhang, Wenbo1,2,3; Li, Zhaoxin3; Wang, Zhaoqi3
2023-02-03
发表期刊COMPLEX & INTELLIGENT SYSTEMS
ISSN2199-4536
页码12
摘要Continuous sign language recognition (CSLR) is an essential task for communication between hearing-impaired and people without limitations, which aims at aligning low-density video sequences with high-density text sequences. The current methods for CSLR were mainly based on convolutional neural networks. However, these methods perform poorly in balancing spatial and temporal features during visual feature extraction, making them difficult to improve the accuracy of recognition. To address this issue, we designed an end-to-end CSLR network: Spatial-Temporal Transformer Network (STTN). The model encodes and decodes the sign language video as a predicted sequence that is aligned with a given text sequence. First, since the image sequences are too long for the model to handle directly, we chunk the sign language video frames, i.e., "image to patch", which reduces the computational complexity. Second, global features of the sign language video are modeled at the beginning of the model, and the spatial action features of the current video frame and the semantic features of consecutive frames in the temporal dimension are extracted separately, giving rise to fully extracting visual features. Finally, the model uses a simple cross-entropy loss to align video and text. We extensively evaluated the proposed network on two publicly available datasets, CSL and RWTH-PHOENIX-Weather multi-signer 2014 (PHOENIX-2014), which demonstrated the superior performance of our work in CSLR task compared to the state-of-the-art methods.
关键词Spatial-temporal encoder Continuous sign language recognition Transformer Patched image
DOI10.1007/s40747-023-00977-w
收录类别SCI
语种英语
资助项目National Key Research and Development Program of China[2020YFC1523302] ; Research Initiation Project for High-Level Talents of Hebei University[521100221081] ; National Natural Science Foundation of China[62172392] ; Provincial Science and Technology Program of Hebei Province[22370301D]
WOS研究方向Computer Science
WOS类目Computer Science, Artificial Intelligence
WOS记录号WOS:000922561500001
出版者SPRINGER HEIDELBERG
引用统计
被引频次:7[WOS]   [WOS记录]     [WOS相关记录]
文献类型期刊论文
条目标识符http://119.78.100.204/handle/2XEOYT63/19959
专题中国科学院计算技术研究所期刊论文
通讯作者Li, Zhaoxin
作者单位1.Hebei Univ, Sch Cyber Secur & Comp, Baoding 071002, Hebei, Peoples R China
2.Hebei Univ, Hebei Machine Vis Engn Res Ctr, Baoding 071002, Hebei, Peoples R China
3.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
推荐引用方式
GB/T 7714
Cui, Zhenchao,Zhang, Wenbo,Li, Zhaoxin,et al. Spatial-temporal transformer for end-to-end sign language recognition[J]. COMPLEX & INTELLIGENT SYSTEMS,2023:12.
APA Cui, Zhenchao,Zhang, Wenbo,Li, Zhaoxin,&Wang, Zhaoqi.(2023).Spatial-temporal transformer for end-to-end sign language recognition.COMPLEX & INTELLIGENT SYSTEMS,12.
MLA Cui, Zhenchao,et al."Spatial-temporal transformer for end-to-end sign language recognition".COMPLEX & INTELLIGENT SYSTEMS (2023):12.
条目包含的文件
条目无相关文件。
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Cui, Zhenchao]的文章
[Zhang, Wenbo]的文章
[Li, Zhaoxin]的文章
百度学术
百度学术中相似的文章
[Cui, Zhenchao]的文章
[Zhang, Wenbo]的文章
[Li, Zhaoxin]的文章
必应学术
必应学术中相似的文章
[Cui, Zhenchao]的文章
[Zhang, Wenbo]的文章
[Li, Zhaoxin]的文章
相关权益政策
暂无数据
收藏/分享
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。