Gait Optimization of Humanoid Robot Based on Deep Q Network

doi:10.3969/j.issn.1006-2475.2019.04.009

Abstract

Abstract: In order to realize the fast and stable walking of humanoid robot, and under the condition that the effective parameter combination is satisfied, a walking parameter training algorithm based on deep reinforcement learning is proposed to optimize the gait of humanoid robot. First of all, we capture the robot gait model parameters from the environment as the input of DQN. And then, DQN is used to fit the robot state-action value function. At last, by action selection strategy, we choose the gait of a robot to perform current action, at the same time produce reward function to achieve the aim of updating DQN. By selecting NAO robot as the experimental object and conducting experiments on the RoboCup3D simulation platform, the results show that using this algorithm, NAO robot can achieve stable bipedal walking.

Key words: humanoid robot, deep reinforcement learning, DQN, gait optimization, RoboCup3D

CLC Number:

TP242.6

YUAN Wen， LIU Hui-yi . Gait Optimization of Humanoid Robot Based on Deep Q Network[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2019.04.009.

References ［24］

［1］	YUAN Q, XI Z, LU Q, et al. Method and experiment of the NAO humanoid robot walking on a slope based on CoM motion estimation and control［C］// International Conference on Intelligent Robotics and Applications. Springer, 2017:154-165.
［2］	KAJITA S, KANEHIRO F, KANEKO K, et al. Biped walking pattern generation by using preview control of zero-moment point［C］// Proceedings of IEEE International Conference on Robotics and Automation. 2003,2:1620-1626.
［3］	MAXIMO M R, COLOMBINI E L, RIBEIRO C H. Stable and fast model-free walk with arms movement for humanoid robots［J］. International Journal of Advanced Robotic System, 2017,14(3):1-11.
［4］	HUAN T T, ANH H P H. Novel stable walking for humanoid robot using particle swarm optimization algorithm［C］// Proceedings of the 2015 International Conference on Artificial Intelligence and Industrial Engineering. 2015.
［5］	RAJENDRA R, PRATIHAR D K. Analysis of double support phase of biped robot and multi-objective optimization using genetic algorithm and particle swarm optimization algorithm［J］. Sadhana, 2015,40(2):549-575.
［6］	SUTTON R S, BARTO A G. Reinforcement Learning: An Introduction［M］. MIT press, 1998.
［7］	JAIN A K, MAO J, MOHIUDDIN K M. Artificial neural networks: A tutorial［J］. Computer, 1996(3):31-44.
［8］	RIEDMILLER M. Neural fitted Q iteration-first experiences with a data efficient neural reinforcement learning method［C］// European Conference on Machine Learning. Springer, 2005:317-328.
［9］	ZHANG Z, LYONS M, SCHUSTER M, et al. Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron［C］// Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition. 1998:454-459.
［10］	LANGE S, RIEDMILLER M. Deep auto-encoder neural networks in reinforcement learning［C］// IEEE 2010 International Joint Conference on Neural Networks (IJCNN). 2010:1-8.
［11］	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing Atari with Deep Reinforcement Learning［EB/OL］. (2013-12-19). https://arxiv.org/abs/1312.5602.
［12］	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning［J］. Nature, 2015,518(7540):529-533.
［13］	WATKINS C J C H. Learning from delayed rewards［J］. Robotics & Autonomous Systems, 1989,15(4):233-235
［14］	PHANITEJA S, DEWANGAN P, GUHAN P, et al. A deep reinforcement learning approach for dynamically stable inverse kinematics of humanoid robots［C］// 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO). 2017:1818-1823.
［15］	DANEL M, SKRBEK M. Humanoid robot control by offline actor-critic learning［J］. CURE Working Proceedings, 2017, 1885.
［16］	GOUAILLIER D, HUGEL V, BLAZEVIC P, et al. Mechatronic design of NAO humanoid［C］// IEEE International Conference on Robotics and Automation(ICRA’09). 2009:769-774.
［17］	XIN J, ZHAO H, LIU D, et al. Application of deep reinforcement learning in mobile robot path planning［C］// Chinese Automation Congress (CAC). IEEE, 2017:7112-7116.
［18］	VUKOBRATOVI〖XCC.TIF,XQ〗 M, BOROVAC B. Zero-moment point--thirty five years of its life［J］. International Journal of Humanoid Robotics, 2004,1(1):157-173.
［19］	VUKOBRATOVI〖XCC.TIF,XQ〗 M, BOROVAC B, POTKONJAK V. ZMP: A review of some basic misunderstandings［J］. International Journal of Humanoid Robotics, 2006,3(2):153-175.
［20］	刘全,翟建伟,章宗长,等. 深度强化学习综述［J］. 计算机学报, 2018,41(1):1-27.
［21］	刘建伟,高峰,罗雄麟. 基于值函数和策略梯度的深度强化学习综述［J］. 计算机学报, 2018.
［22］	ABADI M, BARHAM P, CHEN J, et al. Tensorflow: A system for large-scale machine learning［C］// Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation(OSDI’16). 2016:265-283.
［23］	KONDA V R, TSITSIKLIS J N. Actor-critic algorithms［J］. SIAM Journal on Control and Optimization, 2003,42(4):1143-1166.
［24］	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous Control with Deep Reinforcement Learning［EB/OL］. (2015-09-09). https://arxiv.org/abs/1509.02971.

[1]	LI Peng, XU Luo. An Autonomous Navigation Method for Intelligent Vehicles in Urban Battlefield [J]. Computer and Modernization, 2024, 0(01): 92-98.
[2]	ZHANG Guo-you, SONG Shi-feng. Traffic Light Control Optimization Based On D3QN [J]. Computer and Modernization, 2023, 0(07): 30-35.
[3]	LAI Jian-bin, FENG Gang. An Experience Replay Strategy Based on Mixed Samples [J]. Computer and Modernization, 2023, 0(06): 33-38.
[4]	DING Zhong-lin, LI Yang, CAO Wei, TAN Yu-hao, XU Bo. Deep Q-learning Based Task Offloading in Power IoT [J]. Computer and Modernization, 2022, 0(11): 75-80.
[5]	WU Shui-ming, JI Zhi-yuan, WANG Zhen-yu, JING Dong-sheng. Power Information Network Intrusion Detection Algorithm Based on Dueling-DDQN [J]. Computer and Modernization, 2021, 0(12): 43-47.
[6]	LIU Lu, SHEN Guo-wei, GUO Chun, CUI Yun-he, JIANG Chao-hui, WU Da-yong. A Spark Streaming Parameter Optimization Method Based on Deep Reinforcement Learning [J]. Computer and Modernization, 2021, 0(10): 49-56.
[7]	WANG Peng-yong, CHEN Gong-tao, ZHAO Jiang-shuo. Decision-making Method for Airport Taxi Drivers Based on Deep Reinforcement Learning [J]. Computer and Modernization, 2020, 0(08): 94-99.
[8]	WANG Zhu-xiao, ZHANG Peng-peng, LI Wei, WU Ke-he, CUI Wen-chao, CHENG Rui. Electric Power Industrial Control Network Anomaly Detection #br# System Based on Deep Q Network [J]. Computer and Modernization, 2019, 0(12): 114-.
[9]	PENG Chen, HAN Li-xin. Deep Reinforcement Learning for Step Counting Approach [J]. Computer and Modernization, 2019, 0(01): 63-.
[10]	QI Yue1，2，3, HUANG Shuo-hua1. Portfolio Management Based on DDPG Algorithm of Deep Reinforcement Learning [J]. Computer and Modernization, 2018, 0(05): 93-.
[11]	HE Rong-yi1, LI Chun-guang2. Gait Optimization Research on RoboCup3D Simulation [J]. Computer and Modernization, 2018, 0(03): 20-.
[12]	YAO Qian-yan;YANG Yi-min. Method of Robot Self-localization in RoboCup3D Simulation System [J]. Computer and Modernization, 2011, 12(12): 141-143,.