基于深度学习框架的多模态动作识别

doi:10.3969/j.issn.1006-2475.2017.07.009

摘要/Abstract

摘要： 提出一种基于深度神经网络的多模态动作识别方法，根据不同模态信息的特性分别采用不同的深度神经网络，适应不同模态的视频信息，并将多种深度网络相结合，挖掘行为识别的多模态特征。主要考虑人体行为静态和动态2种模态信息，结合微软Kinect的多传感器摄像机获得传统视频信息的同时也能获取对应的深度骨骼点信息。对于静态信息采用卷积神经网络模型，对于动态信息采用递归循环神经网络模型。最后将2种模型提取的特征相融合进行动作识别和分类。在MSR 3D的行为数据库上实验结果表明，本文的方法对动作识别具有良好的分类效果。

关键词: 深度学习, 多模态, 动作识别

Abstract: This paper proposes an approach for multi-modal action recognition based on deep neural networks. In order to process different modal video information, different artificial networks are utilized and combined to exploit the multi-modal features. We mainly consider the static and dynamic modalities of human action. With the assistance of Microsoft Kinect sensor camera, the visual and depth skeleton data of video can be captured simultaneously. For the static RGB information, we implement Convolutional Neural Networks, while for the dynamic information we use Recurrent Neural Networks. Finally, we combine the extraction features through these two networks and train the action classifier. The experiment results on the MSR 3D datasets show the effectiveness of our method.

Key words: deep learning, multi-modality, action recognition

中图分类号:

TP391.41

韩敏捷. 基于深度学习框架的多模态动作识别[J]. 计算机与现代化, doi: 10.3969/j.issn.1006-2475.2017.07.009.

HAN Min-jie. Multi-modal Action Recognition Based on Deep Learning Framework[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2017.07.009.

参考文献

［1］Sipiran I, Bustos B. Harris 3D: A robust extension of the Harris operator for interest point detection on 3D meshes［J］. The Visual Computer， 2011,27(11):963-976.〖HJ0.9mm〗
［2］Piotr D, Rabaud V, Cottrell G, et al. Behavior recognition via sparse spatio-temporal features［C］//IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance. 2005:65-72.
［3］Geert W, Tuytelaars T, Luc V G. An efficient dense and scale-invariant spatio-temporal interest point detector［C］//European Conference on Computer Vision. 2008: 650-663.
［4］Christian T, Hlavác V. Pose primitive based human action recognition in videos or still images［C］//IEEE Conference on Computer Vision and Pattern Recognition. 2008:1-8.
［5］Wang Heng, Klser A. Action recognition by dense trajectories［C］//IEEE Conference on Computer Vision and Pattern Recognition. 2011: 3169-3176.
［6］Zhang Zhengyou. Microsoft Kinect sensor and its effect［J］. IEEE Multimedia, 2012,19(2): 4-10.
［7］LeCun Y, Boser B E, Denker J S, et al. Handwritten digit recognition with a back-propagation network［C］//Advances in Neural Information Processing Systems. 1989.
［8］LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition［J］. Proceedings of the IEEE， 1998, 86(11): 2278-2324.
［9］Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks［C］//Advances in Neural Information Processing Systems. 2012:1097-1105.
［10］Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3D convolutional networks［C］//IEEE International Conference on Computer Vision (ICCV). 2015:4489-4497.
［11］Donahue J, Hendricks L A, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015:2626-2634.
［12］Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos［C］//Advances in Neural Information Processing Systems. 2014:568-576.
［13］Laptev I, Lindeberg T. On space-time interest points［J］. International Journal of Computer Vision, 2005,64(2-3):107-123.
［14］Lowe D G. Distinctive image features from scale-invariant keypoints［J］. International Journal of Computer Vision, 2004,60(2):91-110.
［15］Dalal N, Triggs B. Histograms of oriented gradients for human detection［C］//IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). 2005:886-893.
［16］Scovanner P, Ali S, Shah M. A 3-dimensional sift descriptor and its application to action recognition［C］//Proceedings of the 15th ACM International Conference on Multimedia. 2007:357-360.
［17］Klaser A, Marszaｌek M, Schmid C. A spatio-temporal descriptor based on 3d-gradients［C］//BMVC 2008-19th British Machine Vision Conference. 2008:275.
［18］Karpathy A, Toderici G, Shetty S, et al. Large-scale video classification with convolutional neural networks［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014:1725-1732.
［19］Jordan M I. Serial order: A parallel distributed processing approach［J］. Advances in Psychology, 1997,121:471-495.
［20］Rabiner L R. A tutorial on hidden Markov models and selected applications in speech recognition［J］. Proceedings of the IEEE, 1989,77(2):257-286.
［21］Hochreiter S, Schmidhuber J. Long short-term memory［J］. Neural Computation, 1997,9(8):1735-1780.
［22］Chollet F. Keras［DB/OL］.https://github.com/fchollet/keras, 2015-01-01.
［23］Yu Gang, Liu Zicheng, Yuan Junsong. Discriminative orderlet mining for real-time recognition of human-object interaction［C］//Asian Conference on Computer Vision. 2014:50-65.
［24］Xia Lu, Aggarwal J K. Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013:2834-2841.
［25］Wang Jiang, Liu Zicheng, Wu Ying, et al. Mining actionlet ensemble for action recognition with depth cameras［C］//IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2012:1290-1297.
［26］Yang Xiaodong, Tian Yingli. Eigenjoints-based action recognition using naive-bayes-nearest-neighbor［C］//IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 2012:14-19.

[1]	胡崇佳, 刘金洲, 方立. 基于无监督域适应的室外点云语义分割[J]. 计算机与现代化, 2024, 0(01): 74-79.
[2]	林威. 基于自监督学习和数据回放的新闻推荐模型增量学习方法[J]. 计算机与现代化, 2023, 0(12): 1-6.
[3]	梁天恺, 黄康华, 刘凯航, 兰岚, 曾碧. 基于双向同态加密的深度联邦图片分类方法[J]. 计算机与现代化, 2023, 0(12): 36-40.
[4]	邱凯星, 冯广. 基于双重特征注意力的多标签图像分类模型[J]. 计算机与现代化, 2023, 0(12): 41-47.
[5]	张伯泉, 麦海鹏, 陈嘉敏, 逄锦聚. 基于高灰度值注意力机制的脑白质高信号分割[J]. 计算机与现代化, 2023, 0(12): 67-75.
[6]	马泽宇, 叶宁, 徐康, 王甦, 王汝传, . 基于FMCW雷达和ResNeSt-GRU的行为识别方法[J]. 计算机与现代化, 2023, 0(11): 101-107.
[7]	谷明轩, 范冰冰. 基于多模态特征融合的抑郁症识别[J]. 计算机与现代化, 2023, 0(10): 17-22.
[8]	李延满, 王必恒, 赵羚焱. 基于轻量化YOLOv5的安全帽检测[J]. 计算机与现代化, 2023, 0(10): 59-64.
[9]	黎世达, 项剑文. 一种提高图像识别模型鲁棒性的弱化强化方法[J]. 计算机与现代化, 2023, 0(10): 70-76.
[10]	沈加炜, 陆一鸣, 陈晓艺, 钱美玲, 陆卫忠, . 基于深度学习的人体行为检测方法研究综述[J]. 计算机与现代化, 2023, 0(09): 1-9.
[11]	刘禅奕, 黄丹, 薛林雁, 王涛, 朱桃, . 改进EfficientNet网络的COVID-19 X光分类[J]. 计算机与现代化, 2023, 0(09): 94-99.
[12]	马国祥, 杨凌菲, 严传波, 张志豪, 孙彬, 王晓荣. 基于深度DenseNet网络的肝包虫病超声影像诊断方法[J]. 计算机与现代化, 2023, 0(09): 100-104.
[13]	农皓程, 任德均, 任秋霖, 刘澎笠, 黄德成. 基于改进ConvNeXt的软塑包装表面异常检测算法[J]. 计算机与现代化, 2023, 0(08): 12-17.
[14]	欧阳飞, 吴旭, 向东升. 基于改进YOLOX的垃圾分类检测方法[J]. 计算机与现代化, 2023, 0(08): 68-73.
[15]	胡睿杰, 车逗. 红外小目标检测方法综述[J]. 计算机与现代化, 2023, 0(08): 79-86.