Abstract: This paper proposes an approach for multi-modal action recognition based on deep neural networks. In order to process different modal video information, different artificial networks are utilized and combined to exploit the multi-modal features. We mainly consider the static and dynamic modalities of human action. With the assistance of Microsoft Kinect sensor camera, the visual and depth skeleton data of video can be captured simultaneously. For the static RGB information, we implement Convolutional Neural Networks, while for the dynamic information we use Recurrent Neural Networks. Finally, we combine the extraction features through these two networks and train the action classifier. The experiment results on the MSR 3D datasets show the effectiveness of our method.

Key words: deep learning, multi-modality, action recognition