基于深度强化学习的计步方法

doi:10.3969/j.issn.1006-2475.2019.01.012

摘要/Abstract

摘要： 针对计步软件使用中用户行为不定，容易产生各种噪声以及传统算法中参数不能持续优化的问题，本文提出基于深度强化学习的计步方法。将噪声判别及步数统计作为智能体的动作，在步数统计中改进波峰检测法，提出均值穿越波峰波谷检测法。利用循环神经网络保存内部状态，将用户对计步器计步好坏的反馈作为奖励信号，指导参数持续优化。实验结果表明，该方法在采集设备放置于不同位置并且有噪声时，噪声识别率为0.9151，计步误差率为0.0623，有较高的精度以及较强的抗干扰能力。

关键词: 计步器, 深度强化学习, 均值穿越波峰波谷检测法

Abstract: In order to deal with the problem that the user’s behavior is often uncertain in the use of step counting software, which is easy to produce various noises and the parameters in traditional algorithms cannot be continuously optimized, this paper proposes the deep reinforcement learning for step counting approach. Taking the step counting and noise discrimination as the action of the agent, the wave peak detection method is improved in the step counting and the mean crossing peak detection method is proposed. Using the recurrent neural network to save the internal state, the feedback of the user on the step effect of the pedometer is used as the reward signal to guide the parameter optimization. The experimental results show that the proposed method has high precision and strong anti-interference ability when there is noise and the mobile phone is placed in different positions, in which the noise recognition rate is 0.9151 and the step counting error rate is 0.0623.

Key words: pedometer, deep reinforcement learning, mean crosssing peak trough detection method

中图分类号:

TP391.41

彭琛,韩立新. 基于深度强化学习的计步方法[J]. 计算机与现代化, doi: 10.3969/j.issn.1006-2475.2019.01.012.

PENG Chen, HAN Li-xin. Deep Reinforcement Learning for Step Counting Approach[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2019.01.012.

参考文献

［1］ UHM K E, JI S Y, CHUNG S H, et al. Effects of exercise intervention in breast cancer patients: Is mobile health (mHealth) with pedometer more effective than conventional program using brochure?［J］. Breast Cancer Research & Treatment, 2017,3(161):443-452.
［2］ ZHANG H M, YUAN W Z, SHEN Q, et al. A handheld inertial pedestrian navigation system with accurate step modes and device poses recognition［J］. IEEE Sensors Journal, 2014,15(3):1421-1429.
［3］黄政,韩立新,肖艳. 一种基于移动终端的新型计步方法［J］. 计算机学报, 2017,40(8):1856-1871.
［4］ LU Y T, VELIPASALAR S. Autonomous footstep counting and traveled distance calculation by mobile devices incorporating camera and accelerometer data［J］. IEEE Sensors Journal, 2017,17(21):7157-7166.
［5］ GENOVESE V, MANNINI A, SABATINI A M. A smartwatch step counter for slow and intermittent ambulation［J］. IEEE Access, 2017,5(99):13028-13037.
［6］ OSHIN T O, POSLAD S. ERSP: An energy-efficient real-time smartphone pedometer［C］// Proceedings of 2013 IEEE International Conference on Systems, Man, and Cybernetics. 2013:2067-2072.
［7］ TANG Z H, GUO Y, CHEN X Q. Self-adaptive step counting on smartphones under unrestricted stepping modes［C］// 2016 40th IEEE Annual Computer Software and Applications Conference (COMPSAC). 2016:788-797.
［8］ PAN M S, LIN H W. A step counting algorithm for smartphone users: Design and implementation［J］. IEEE Sensors Journal, 2015,15(4):2296-2305.
［9］〖KG-*5〗JAYALATH S, ABHAYASINGHE N. A gyroscopic data based pedometer algorithm［C］// 2013 8th International Conference on Computer Science & Education. 2013:551-555.
［10］GOWDA A K S N, BABU S R, SEKARAN D C. UMOISP: Usage mode and orientation invariant smartphone pedometer［J］. IEEE Sensors Journal, 2017,17(3):869-881.
［11］ANACLETO R, FIGUEIREDO L, ALMEIDA A, et al. Step count and classification using sensor information fusion［M］// Ambient Intelligence-Software and Applications. Springer International Publishing. 2015:85-93.
［12］ARULKUMARAN K, DEISENROTH M P, BRUNDAGE M, et al. Deep reinforcement learning: A brief survey［J］. IEEE Signal Processing Magazine, 2017,34(6):26-38.
［13］SCHMIDHUBER J. Deep learningin neural networks: An overview［J］. Neural Networks, 2014,61:85-117.
［14］MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning［J］. Nature, 2015,518(7540):529-533.
［15］SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search［J］. Nature, 2016,529(7587):484-489.
［16］ZHU Y K, MOTTAGHI R, KOLVE E, et al. Target-driven visual navigation in indoor scenes using deep reinforcement learning［C］// 2017 IEEE International Conference on Robotics and Automation(ICRA). 2017:3357-3364.
［17］LEVINE S, FINN C, DARRELL T, et al. End-to-end training of deep visuomotor policies［J］. Journal of Machine Learning Research, 2015,17(1):1334-1373.
［18］LEVINE S, PASTOR P, KRIZHEVSKY A, et al. Learning hand-eye coordination for robotic grasping with large-scale data collection［J］. International Journal of Robotics Research, 2016(10):173-184.
［19］LAKE B M, ULLMAN T D, TENENBAUM J B, et al. Building machines that learn and think like people［J］. The Behavioral & Brain Sciences, 2017.
［20］MNIH V, HEESS N, GRAVES A, et al. Recurrent models of visual attention［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014:2204-2212.
［21］WILLIAMS R J. Simple statistical gradient-following algorithms for connectionist reinforcement learning［J］. Machine Learning, 1992,8(3-4):229-256.
［22］WIERSTRA D, FOERSTER A, PETERS J, et al. Solving deep memory POMDPs with recurrent policy gradients［C］// Proceedings of the 17th International Conference on Artificial Neural Networks. 2007:697-706.
［23］BRAJDIC A, HARLE R. Walk detection and step counting on unconstrained smartphones［C］// Proceedings of 2013 ACM International Joint Conference on Pervasive and ubiquitous Computing. 2013:225-234.
［24］CHO K, VAN M B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation［C］// Proceedings of 2014 Empirical Methods in Natural Language Processing. 2014:1724-1734.
［25］ABADI M, AGARWAL A, BARHAM P, et al. TensorFlow: Large-scale machine learning on heterogeneous distributed systems［J］. ArXiv: 1603.04467, 2015.
［26］RUDER S. An overview of gradientdescent optimization algorithms［J］. ArXiv: 1609.04747, 2016.

[1]	李鹏, 徐珞. 一种面向城市战场的智能车自主导航方法[J]. 计算机与现代化, 2024, 0(01): 92-98.
[2]	张国有, 宋世峰. 基于D3QN的交通灯控制优化[J]. 计算机与现代化, 2023, 0(07): 30-35.
[3]	赖建彬, 冯刚. 一种基于混合样本的经验回放策略[J]. 计算机与现代化, 2023, 0(06): 33-38.
[4]	丁忠林, 李洋, 曹委, 谈宇浩, 徐波. 基于深度Q学习的电力物联网任务卸载研究[J]. 计算机与现代化, 2022, 0(11): 75-80.
[5]	吴水明, 吉志远, 王震宇, 景栋盛. 基于Dueling-DDQN的电力信息网络入侵检测算法[J]. 计算机与现代化, 2021, 0(12): 43-47.
[6]	刘露, 申国伟, 郭春, 崔允贺, 蒋朝惠, 伍大勇. 一种基于深度强化学习的Spark Streaming参数优化方法[J]. 计算机与现代化, 2021, 0(10): 49-56.
[7]	王鹏勇, 陈龚涛, 赵江烁. 基于深度强化学习的机场出租车司机决策方法[J]. 计算机与现代化, 2020, 0(08): 94-99.
[8]	袁雯，刘惠义. 基于深度Q网络的仿人机器人步态优化[J]. 计算机与现代化, 2019, 0(04): 47-.
[9]	齐岳1，2，3，黄硕华1. 基于深度强化学习DDPG算法的投资组合管理[J]. 计算机与现代化, 2018, 0(05): 93-.