Deep Reinforcement Learning for Step Counting Approach

doi:10.3969/j.issn.1006-2475.2019.01.012

Abstract

Abstract: In order to deal with the problem that the user’s behavior is often uncertain in the use of step counting software, which is easy to produce various noises and the parameters in traditional algorithms cannot be continuously optimized, this paper proposes the deep reinforcement learning for step counting approach. Taking the step counting and noise discrimination as the action of the agent, the wave peak detection method is improved in the step counting and the mean crossing peak detection method is proposed. Using the recurrent neural network to save the internal state, the feedback of the user on the step effect of the pedometer is used as the reward signal to guide the parameter optimization. The experimental results show that the proposed method has high precision and strong anti-interference ability when there is noise and the mobile phone is placed in different positions, in which the noise recognition rate is 0.9151 and the step counting error rate is 0.0623.

Key words: pedometer, deep reinforcement learning, mean crosssing peak trough detection method

CLC Number:

TP391.41

PENG Chen, HAN Li-xin. Deep Reinforcement Learning for Step Counting Approach[J]. Computer and Modernization, 2019, 0(01): 63-.

References

［1］ UHM K E, JI S Y, CHUNG S H, et al. Effects of exercise intervention in breast cancer patients: Is mobile health (mHealth) with pedometer more effective than conventional program using brochure?［J］. Breast Cancer Research & Treatment, 2017,3(161):443-452.
［2］ ZHANG H M, YUAN W Z, SHEN Q, et al. A handheld inertial pedestrian navigation system with accurate step modes and device poses recognition［J］. IEEE Sensors Journal, 2014,15(3):1421-1429.
［3］黄政,韩立新,肖艳. 一种基于移动终端的新型计步方法［J］. 计算机学报, 2017,40(8):1856-1871.
［4］ LU Y T, VELIPASALAR S. Autonomous footstep counting and traveled distance calculation by mobile devices incorporating camera and accelerometer data［J］. IEEE Sensors Journal, 2017,17(21):7157-7166.
［5］ GENOVESE V, MANNINI A, SABATINI A M. A smartwatch step counter for slow and intermittent ambulation［J］. IEEE Access, 2017,5(99):13028-13037.
［6］ OSHIN T O, POSLAD S. ERSP: An energy-efficient real-time smartphone pedometer［C］// Proceedings of 2013 IEEE International Conference on Systems, Man, and Cybernetics. 2013:2067-2072.
［7］ TANG Z H, GUO Y, CHEN X Q. Self-adaptive step counting on smartphones under unrestricted stepping modes［C］// 2016 40th IEEE Annual Computer Software and Applications Conference (COMPSAC). 2016:788-797.
［8］ PAN M S, LIN H W. A step counting algorithm for smartphone users: Design and implementation［J］. IEEE Sensors Journal, 2015,15(4):2296-2305.
［9］〖KG-*5〗JAYALATH S, ABHAYASINGHE N. A gyroscopic data based pedometer algorithm［C］// 2013 8th International Conference on Computer Science & Education. 2013:551-555.
［10］GOWDA A K S N, BABU S R, SEKARAN D C. UMOISP: Usage mode and orientation invariant smartphone pedometer［J］. IEEE Sensors Journal, 2017,17(3):869-881.
［11］ANACLETO R, FIGUEIREDO L, ALMEIDA A, et al. Step count and classification using sensor information fusion［M］// Ambient Intelligence-Software and Applications. Springer International Publishing. 2015:85-93.
［12］ARULKUMARAN K, DEISENROTH M P, BRUNDAGE M, et al. Deep reinforcement learning: A brief survey［J］. IEEE Signal Processing Magazine, 2017,34(6):26-38.
［13］SCHMIDHUBER J. Deep learningin neural networks: An overview［J］. Neural Networks, 2014,61:85-117.
［14］MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning［J］. Nature, 2015,518(7540):529-533.
［15］SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search［J］. Nature, 2016,529(7587):484-489.
［16］ZHU Y K, MOTTAGHI R, KOLVE E, et al. Target-driven visual navigation in indoor scenes using deep reinforcement learning［C］// 2017 IEEE International Conference on Robotics and Automation(ICRA). 2017:3357-3364.
［17］LEVINE S, FINN C, DARRELL T, et al. End-to-end training of deep visuomotor policies［J］. Journal of Machine Learning Research, 2015,17(1):1334-1373.
［18］LEVINE S, PASTOR P, KRIZHEVSKY A, et al. Learning hand-eye coordination for robotic grasping with large-scale data collection［J］. International Journal of Robotics Research, 2016(10):173-184.
［19］LAKE B M, ULLMAN T D, TENENBAUM J B, et al. Building machines that learn and think like people［J］. The Behavioral & Brain Sciences, 2017.
［20］MNIH V, HEESS N, GRAVES A, et al. Recurrent models of visual attention［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014:2204-2212.
［21］WILLIAMS R J. Simple statistical gradient-following algorithms for connectionist reinforcement learning［J］. Machine Learning, 1992,8(3-4):229-256.
［22］WIERSTRA D, FOERSTER A, PETERS J, et al. Solving deep memory POMDPs with recurrent policy gradients［C］// Proceedings of the 17th International Conference on Artificial Neural Networks. 2007:697-706.
［23］BRAJDIC A, HARLE R. Walk detection and step counting on unconstrained smartphones［C］// Proceedings of 2013 ACM International Joint Conference on Pervasive and ubiquitous Computing. 2013:225-234.
［24］CHO K, VAN M B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation［C］// Proceedings of 2014 Empirical Methods in Natural Language Processing. 2014:1724-1734.
［25］ABADI M, AGARWAL A, BARHAM P, et al. TensorFlow: Large-scale machine learning on heterogeneous distributed systems［J］. ArXiv: 1603.04467, 2015.
［26］RUDER S. An overview of gradientdescent optimization algorithms［J］. ArXiv: 1609.04747, 2016.

[1]	WANG Jian-ming1, WANG Xin1, LI Yang-hui2, WANG Dian-long1. Path Planning of Parking Robot Based on Improved D3QN Algorithm [J]. Computer and Modernization, 2024, 0(03): 7-14.
[2]	LI Peng, XU Luo. An Autonomous Navigation Method for Intelligent Vehicles in Urban Battlefield [J]. Computer and Modernization, 2024, 0(01): 92-98.
[3]	ZHANG Guo-you, SONG Shi-feng. Traffic Light Control Optimization Based On D3QN [J]. Computer and Modernization, 2023, 0(07): 30-35.
[4]	LAI Jian-bin, FENG Gang. An Experience Replay Strategy Based on Mixed Samples [J]. Computer and Modernization, 2023, 0(06): 33-38.
[5]	DING Zhong-lin, LI Yang, CAO Wei, TAN Yu-hao, XU Bo. Deep Q-learning Based Task Offloading in Power IoT [J]. Computer and Modernization, 2022, 0(11): 75-80.
[6]	WU Shui-ming, JI Zhi-yuan, WANG Zhen-yu, JING Dong-sheng. Power Information Network Intrusion Detection Algorithm Based on Dueling-DDQN [J]. Computer and Modernization, 2021, 0(12): 43-47.
[7]	LIU Lu, SHEN Guo-wei, GUO Chun, CUI Yun-he, JIANG Chao-hui, WU Da-yong. A Spark Streaming Parameter Optimization Method Based on Deep Reinforcement Learning [J]. Computer and Modernization, 2021, 0(10): 49-56.
[8]	WANG Peng-yong, CHEN Gong-tao, ZHAO Jiang-shuo. Decision-making Method for Airport Taxi Drivers Based on Deep Reinforcement Learning [J]. Computer and Modernization, 2020, 0(08): 94-99.
[9]	YUAN Wen， LIU Hui-yi . Gait Optimization of Humanoid Robot Based on Deep Q Network [J]. Computer and Modernization, 2019, 0(04): 47-.
[10]	QI Yue1，2，3, HUANG Shuo-hua1. Portfolio Management Based on DDPG Algorithm of Deep Reinforcement Learning [J]. Computer and Modernization, 2018, 0(05): 93-.