Motif-GCNs With Local and Non-Local Temporal Blocks for Skeleton-Based Action Recognition

doi:10.1109/TPAMI.2022.3170511

	Motif-GCNs With Local and Non-Local Temporal Blocks for Skeleton-Based Action Recognition
	Wen, Yu-Hui 1; Gao, Lin 2,3; Fu, Hongbo 4; Zhang, Fang-Lue 5; Xia, Shihong 2,3; Liu, Yong-Jin 1
	2023-02-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	45 期号:2 页码:2009-2023
摘要	Recent works have achieved remarkable performance for action recognition with human skeletal data by utilizing graph convolutional models. Existing models mainly focus on developing graph convolutional operations to encode structural properties of a skeletal graph, whose topology is manually predefined and fixed over all action samples. Some recent works further take sample-dependent relationships among joints into consideration. However, the complex relationships between arbitrary pairwise joints are difficult to learn and the temporal features between frames are not fully exploited by simply using traditional convolutions with small local kernels. In this paper, we propose a motif-based graph convolution method, which makes use of sample-dependent latent relations among non-physically connected joints to impose a high-order locality and assigns different semantic roles to physical neighbors of a joint to encode hierarchical structures. Furthermore, we propose a sparsity-promoting loss function to learn a sparse motif adjacency matrix for latent dependencies in non-physical connections. For extracting effective temporal information, we propose an efficient local temporal block. It adopts partial dense connections to reuse temporal features in local time windows, and enrich a variety of information flow by gradient combination. In addition, we introduce a non-local temporal block to capture global dependencies among frames. Our model can capture local and non-local relationships both spatially and temporally, by integrating the local and non-local temporal blocks into the sparse motif-based graph convolutional networks (SMotif-GCNs). Comprehensive experiments on four large-scale datasets show that our model outperforms the state-of-the-art methods. Our code is publicly available at https://github.com/wenyh1616/SAMotif-GCN.
关键词	Skeleton Feature extraction Joints Convolutional codes Topology Training Sparse matrices Action recognition graph convolutional neural networks spatio-temporal attention non-local block skeleton sequence
DOI	10.1109/TPAMI.2022.3170511
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Plan[2021YFF0307702] ; China Postdoctoral Sci-ence Foundation[2021M701891] ; Tsinghua University Initiative Scientific Research Program ; National Natural Science Foundation of China[61725204] ; National Natural Science Foundation of China[61872440] ; Beijing Munic-ipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; Youth Innovation Promotion Association CAS, Marsden Fund Council[MFP-20-VUW-180]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000912386000044
出版者	IEEE COMPUTER SOC
引用统计	被引频次：17[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/20042
专题	中国科学院计算技术研究所期刊论文
通讯作者	Gao, Lin; Liu, Yong-Jin
作者单位	1.Tsinghua Univ, Dept Comp Sci & Technol, BNRist, Beijing 100190, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Beijing 101408, Peoples R China 4.City Univ Hong Kong, Sch Creat Media, Hong Kong, Peoples R China 5.Victoria Univ Wellington, Sch Engn & Comp Sci, Wellington 6012, New Zealand
推荐引用方式 GB/T 7714	Wen, Yu-Hui,Gao, Lin,Fu, Hongbo,et al. Motif-GCNs With Local and Non-Local Temporal Blocks for Skeleton-Based Action Recognition[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(2):2009-2023.
APA	Wen, Yu-Hui,Gao, Lin,Fu, Hongbo,Zhang, Fang-Lue,Xia, Shihong,&Liu, Yong-Jin.(2023).Motif-GCNs With Local and Non-Local Temporal Blocks for Skeleton-Based Action Recognition.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(2),2009-2023.
MLA	Wen, Yu-Hui,et al."Motif-GCNs With Local and Non-Local Temporal Blocks for Skeleton-Based Action Recognition".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.2(2023):2009-2023.