Dynamic Gesture Recognition Based on Space-time Feature Extraction of Neural Network

Abstract

Abstract: Aiming at the existing 3D convolution method of dynamic gesture recognition with large number of computational parameters, it is difficult to extract 2D convolution of long-time-series images in terms of time dimension. In this paper, a gesture recognition method based on the combination of 2D convolutional neural network and long and short term memory network is proposed. Firstly, spatial features are extracted based on 2D convolutional neural network, and then features in time dimension are extracted by interrelation of sequential images through long and short term memory network. In order to verify the validity of the algorithm in this paper, using the acquisition of 7 kinds of dynamic hand gestures and IsoGD public data sets to verify this proposed algorithm, the experimental results show that under using online enhancement algorithm the paper in the collection on a set of dynamic hand gestures recognition rate reached 87.14%, IsoGD public data sets on recognition rate of 57.89%, compared with the existing method, the recognition rate is improved.

Key words: convolutional neural network, long and short term memory network, dynamic gesture recognition, space-time feature extraction, online data enhancement

LIN Zhi-wei, ZHU Wen-zhang, CHEN Hao. Dynamic Gesture Recognition Based on Space-time Feature Extraction of Neural Network[J]. Computer and Modernization, 2021, 0(06): 41-47.

References ［24］

［1］	LIU H Y, WANG L H. Gesture recognition for human-robot collaboration: A review［J］. International Journal of Industrial Ergonomics, 2018,68:355-367.
［2］	MAITY S, BHATTACHARJEE D, CHAKRABARTI A. A novel approach for human action recognition from silhouette images［J］. IETE Journal of Research, 2017,63(2):160-171.
［3］	PARCHETA Z, MARTINEZ-HINAREJOS C D. Sign language gesture recognition using HMM［C］// Proceedings of Iberian Conference on Patten Recognition and Image Analysis. 2017:419-426.
［4］	MOHANTY A, SAHAY R R. Rasabodha: Understanding Indian classical dance by recognition emotions using deep learning［J］. Patten Recognition, 2018,79(7):97-113.
［5］	MOLCHANOV P, GUPTA S, KIN K, et al. Hand gesture recognition with 3D convolutional neural networks［C］// 2015 IEEE Conference on Computer Vision & Patten Recognition Workshops. 2015， DOI: 10.1109/CVPRW.2015.7301342.
［6］	WU D, PIGOU L, KINDERMANS P J, et al. Deep dynamic neural networks for multimodal gesture segmentation and recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016,38(8):1583-1597.
［7］	刘妍. 基于视频的多目标运动人体动作识别［D］．上海:东华大学, 2017.
［8］	ZHANG D M, ZHANG G F, DAI S l. Research on vision-based multi-user gesture recognition human-computer interaction［C］// 2008 Asia Simulation Conference-7th International Conference on System Simulation and Scientific Computing. IEEE, 2008:1455-1458.
［9］	VO M T， WAIBEL A. A multi-modal human-computer interface：Combination of gesture and speech recognition［C］// International Conference on Human-Computer Interaction(InterCHI 1993). 1993:69-70.
［10］	熊俊涛,刘梓健,孙宝霞,等. 基于视觉技术的手势跟踪与动作识别算法［J］. 计算机与现代化, 2014(7):75-79.
［11］	GUO D, ZHOU W G, LI H Q, et al. Online early-late fusion based on adaptive HMM for sign language recognition［J］. ACM Trans. on Multimedia Computing, Communications, and Applicatons, 2018,14(1): Article No.8.
［12］	TU Z G, XIE W, QIN Q Q, et al. Multi-stream CNN: Learning representation based on human related regions for action recognition［J］. Pattern Recognition, 2018,79(7):32-43.
［13］	HE K M， ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition［C］// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016:770-778.
［14］	LE CUN Y, BOSER B E, DENKER J S, et al. Handwritten digit recognition with a back-propagation network［C］// Advances in Neural Information Processing Systems. 1990:396-404.
［15］	PIGOU L, VAN DEN ORAD A, DIELEMAN S, et al. Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video［J］. International Journal of Computer Vision, 2018.126(2-4):430-439.
［16］	HUANG Z, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging［J］. Computation and Language, arXiv:1508.01991.
［17］	PICKERING C A. The search for a safer driver interface： A review of gesture recognition human machine interface［J］. Computing & Control Engineering Journal, 2005,16(1):34-40.
［18］	莫伟珑. 基于计算机视觉的手势识别方法研究［D］. 桂林：广西师范大学, 2019.
［19］	WAN J, LI S Z, ZHAO Y, et al. ChaLearn looking at people RGB-D isolated and continuous datasets for gesture recognition［C］// 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). IEEE, 2016:761-769.
［20］	GUYON I, ATHITSOS V, JANGYODSUK P, et al. The ChaLearn gesture dataset(CGD 2011)［J］. Machine Vision & Applications, 2014,25(8):1929-1951.
［21］	CHAI X J, LIU Z P, YIN F, et al. Two streams recurrent neural networks for large-scale continuous gesture recognition［C］// International Conference on Pattern Recognition(ICPR). IEEE, 2016, DOI: 10.1109/ICPR.2016.7899603.
［22］	ZHU G M, ZHANG L, MEI L, et al. Large-scale isolated gesture recognition using pyramidal 3D convolutional networks［C］// International Conference on Pattern Recognition(ICPR). IEEE, 2016, DOI: 10.1109/ICPR.2016.7899601.
［23］	WANG P C, LI W Q, LIU S, et al. Large-scale isolated gesture recognition using convolutional neural networks［C］// International Conference on Pattern Recognition(ICPR). IEEE, 2016, DOI: 10.1109/ICPR.2016.7899599.
［24］	LI Y N, MIAO Q G, TIANK, et al. Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model［C］// International Conference on Pattern Recognition(ICPR). IEEE, 2016, DOI: 10.1109/ICPR.2016.7899602.

[1]	ZHOU Cheng-cheng, ZENG Qing-jun, YANG Kang, HU Jia-ming, HAN Chun-wei. EEG Recognition of Motor Imagination Based on Efficiency Channel Attention Module [J]. Computer and Modernization, 2023, 0(12): 19-23.
[2]	LIU Fu-qi, ZHANG Da, SONG Jian-hua, WANG Hai-dong. Fault Diagnosis of Hydraulic Systems Based on CNN-BiLSTM [J]. Computer and Modernization, 2023, 0(09): 10-19.
[3]	JIANG Lei, TANG Jian, YANG Chao-yue, LYU Ting-ting. Bearing Fault Diagnosis Based on CWGAN-GP and CNN [J]. Computer and Modernization, 2023, 0(07): 1-6.
[4]	XU Ye-tong, GENG Xin-zhe, ZHAO Wei-qiang, ZHANG Yue, NING Hai-long, LEI Tao. A Remote Sensing Image Change Detection Model Based on CNN-Transformer Hybrid Structure [J]. Computer and Modernization, 2023, 0(07): 79-85.
[5]	ZHU Jian-bo, GE Ming-feng, DONG Wen-fei. Alzheimer’s Disease Image Classification Based on Improved EfficientNet [J]. Computer and Modernization, 2023, 0(06): 56-61.
[6]	LIU Jia-jia, HU Xu-xin, YU Ping. Monocular Depth Estimation Method by Aggregating Multi-dimensional Attention Features [J]. Computer and Modernization, 2023, 0(06): 76-81.
[7]	LIU Jing, CHEN Jin-guang. Image Caption Generation Method Based on Channel Attention and Transformer [J]. Computer and Modernization, 2023, 0(05): 8-12.
[8]	WANG Xin-yi, YIN Si-qing, HONG Jun. Asymmetric Deep Supervised Hashing with Attention Mechanism [J]. Computer and Modernization, 2023, 0(05): 26-31.
[9]	SU Jin-ku, GUI Zhi-ming. Prediction of Short-term Taxi Flow Based on Spatio-temporal Characteristics [J]. Computer and Modernization, 2023, 0(05): 32-38.
[10]	WANG Juan, LI Chuan-geng, ZHANG Qing-yuan, XIA Cheng-yi. Segmentation Method of Knee Meniscus Based on Multiscale-net [J]. Computer and Modernization, 2023, 0(05): 111-116.
[11]	WANG Lei, ZHANG Xiao-dong, DAI Huan. Fault Diagnosis of Pumping Unit Based on 1D-CNN-LSTM Attention Network [J]. Computer and Modernization, 2023, 0(04): 1-6.
[12]	XU Ya-xin, HE Ze-en, XU Xu-kan. Automatic Classification Method of CNC Machine Tool Fault Text Based on CNN-BiLSTM [J]. Computer and Modernization, 2023, 0(04): 7-14.
[13]	ZHU Yuan-ye, NI Jian-jun, TANG Guang-yi. An RGB-D Indoor Scene Classification Method Based on Improved Convolutional Neural Network [J]. Computer and Modernization, 2023, 0(04): 73-77.
[14]	CHEN Zhuo, QIAO Gui-fang, CHAI Xin-bo, DU Yi-jun, SHEN Chong-lin, WANG Yuan-hao. Multi-weather Vehicle Detection Algorithm Based on Modified Knowledge Distillation [J]. Computer and Modernization, 2023, 0(02): 50-57.
[15]	ZHANG Xiao, LYU Ji-yu, ZHAO Shuang, WU Yu-lun, WANG Chun-le. SAR Ship Classification Based on Multi-convolutional Neural Network Fusion [J]. Computer and Modernization, 2023, 0(01): 37-42.