Joint Learning in the Spatio-Temporal and Frequency Domains for Skeleton-Based Action Recognition | |
Guyue, Hu2,3,4; Bo, Cui2,3,4; Shan, Yu1,2,3,4 | |
刊名 | IEEE TRANSACTIONS ON MULTIMEDIA |
2020-09-01 | |
卷号 | 22期号:9页码:2207-2220 |
关键词 | Skeleton-based Action Recognition Frequency Attention Synchronous Local and Non-local Learning Soft-margin Focal Loss Pesudo Multi-task Learning |
ISSN号 | 1520-9210 |
DOI | 10.1109/TMM.2019.2953325 |
文献子类 | Article |
英文摘要 | Benefiting from its succinctness and robustness, skeleton-based action recognition has recently attracted much attention. Most existing methods utilize local networks (e.g. recurrent network, convolutional network, and graph convolutional network) to extract spatio-temporal dynamics hierarchically. As a consequence, the local and non-local dependencies, which contain more details and semantics respectively, are asynchronously captured in different level of layers. Moreover, existing methods are limited to the spatio-temporal domain and ignore information in the frequency domain. To better extract synchronous detailed and semantic information from multi-domains, we propose a residual frequency attention (rFA) block to focus on discriminative patterns in the frequency domain, and a synchronous local and non-local (SLnL) block to simultaneously capture the details and semantics in the spatio-temporal domain. In addition, to optimize the whole learning processes of the multi-branch network, we put it under a pseudo multi-task learning paradigm. During training, 1) a soft-margin focal loss (SMFL) is proposed to optimize the intra-branch separated learning process, which can automatically conduct data selection and encourage intrinsic margins in classifiers; 2) A mutual learning policy is also proposed to further facilitate the inter-branch collaborative learning process. Eventually, our approach achieves the state-of-the-art performance on several large-scale datasets for skeleton-based action recognition. |
资助项目 | National Key Research and Development Program of China[2017YFA0105203] ; Strategic Priority Research Program of the Chinese Academy of Sciences (CAS)[XDB32040200] ; Hundred-Talent Program of CAS |
WOS关键词 | ENSEMBLE |
WOS研究方向 | Computer Science ; Telecommunications |
语种 | 英语 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
WOS记录号 | WOS:000562310200002 |
资助机构 | National Key Research and Development Program of China ; Strategic Priority Research Program of the Chinese Academy of Sciences (CAS) ; Hundred-Talent Program of CAS |
内容类型 | 期刊论文 |
源URL | [http://ir.ia.ac.cn/handle/173211/40515] |
专题 | 自动化研究所_脑网络组研究中心 |
通讯作者 | Guyue, Hu |
作者单位 | 1.CAS Ctr Excellence Brain Sci & Intelligence Techn, Beijing 100190, Peoples R China 2.Chinese Acad Sci, Brainnetome Ctr, Inst Automat, Beijing 100190, Peoples R China 3.Chinese Acad Sci, Natl Labo Pattern Recognit, Inst Automat, Beijing 100190, Peoples R China 4.Univ Chinese Acad Sci, Beijing 100049, Peoples R China |
推荐引用方式 GB/T 7714 | Guyue, Hu,Bo, Cui,Shan, Yu. Joint Learning in the Spatio-Temporal and Frequency Domains for Skeleton-Based Action Recognition[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2020,22(9):2207-2220. |
APA | Guyue, Hu,Bo, Cui,&Shan, Yu.(2020).Joint Learning in the Spatio-Temporal and Frequency Domains for Skeleton-Based Action Recognition.IEEE TRANSACTIONS ON MULTIMEDIA,22(9),2207-2220. |
MLA | Guyue, Hu,et al."Joint Learning in the Spatio-Temporal and Frequency Domains for Skeleton-Based Action Recognition".IEEE TRANSACTIONS ON MULTIMEDIA 22.9(2020):2207-2220. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论