Path Planning of Parking Robot Based on Improved D3QN Algorithm

doi:10.3969/j.issn.1006-2475.2024.03.002

Abstract

Abstract: Abstract： The parking robot emerges as a solution to the urban parking problem， and its path planning is an important research direction. Due to the limitations of the A* algorithm， the deep reinforcement learning idea is introduced in this article， and improves the D3QN algorithm. Through replacing the convolutional network with a residual network and introducing attention mechanisms， the SE-RD3QN algorithm is proposed to improve network degradation and convergence speed， and enhance model accuracy. During the algorithm training process， the reward and punishment mechanism is improved to achieve rapid convergence of the optimal solution. Through comparing the experimental results of the D3QN algorithm and the RD3QN algorithm with added residual layers， it shows that the SE-RD3QN algorithm achieves faster convergence during model training. Compared with the currently used A*+TEB algorithm， SE-RD3QN can obtain shorter path length and planning time in path planning. Finally， the effectiveness of the algorithm is further verified through physical experiments simulating a car， which provides a new solution for parking path planning.

Key words: Key words： deep reinforcement learning, parking robot, path planning, lidar sensors

CLC Number:

TP183
TP2

WANG Jian-ming1, WANG Xin1, LI Yang-hui2, WANG Dian-long1. Path Planning of Parking Robot Based on Improved D3QN Algorithm[J]. Computer and Modernization, 2024, 0(03): 7-14.

References

［1］叶学根. AGV智能机器人停车库的应用［J］. 智能建筑与智慧城市， 2021（11）：113-114.
［2］于振中，闫继宏，赵杰，等. 改进人工势场法的移动机器人路径规划［J］. 哈尔滨工业大学学报， 2011，43（1）：50-55
［3］张原，陈宇轩，魏璐璐. 基于改进A~*算法的AGV智能泊车算法［J］. 计算机系统应用， 2019，28（1）：216-221.
［4］金云，周苗，黄仁欢，等. 一种基于Dijkstra算法的动态进路规划方法［J］. 铁路通信信号工程技术， 2022，19（1）：20-23.
［5］庞峰. 模拟退火算法的原理及算法在优化问题上的应用［D］. 长春：吉林大学， 2006.
［6］侯翔. 基于人工势场和量子遗传算法的移动机器人路径规划方法［J］. 计算机应用与软件， 2018，35（6）：263-266.
［7］张荣霞，武长旭，孙同超，等. 深度强化学习及在路径规划中的研究进展［J］. 计算机工程与应用， 2021，57（19）：44-56.
［8］毛嘉琪. 改进蚁群算法的机器人路径规划研究［J］. 计算机应用与软件， 2021，38（5）：300-306.
［9］ SASAKI H.， HORIUCHI T， KATO S. A study on vision-based mobile robot learning by deep Q-network［C］// 2017 56th Annual Conference of the Society of Instrument and Control Engineers of Japan （SICE）. 2017：799-804.
［10］ GAO J L， YE W J， GUO J， et al. Deep reinforcement learning for indoor mobile robot path planning［J］. Sensors， 2020，20（19）：1-15.
［11］牛鹏飞，王晓峰，芦磊，等. 强化学习在车辆路径问题中的研究综述［J］. 计算机工程与应用， 2022，58（1）：41-55.
［12］ MNIH V， KAVUKCUOGLU K， SILVER D， et al. Playing Atari with deep reinforcement learning［J］. arXiv preprint arXiv：1312.5602， 2013.
［13］ HASSELT H V， GUEZ A， SILVER D. Deep reinforcement learning with double Q-learning［J］. arXiv preprint arXiv：1509.06461， 2015.
［14］ WANG Z Y，SCHAUL T，HESSEL M，et al. Dueling network architectures for deep reinforcement learning［J］. arXiv preprint arXiv：1511.06581，2015.
［15］ TAI L， LIU M. Towards cognitive exploration through deep reinforcement learning for mobile robots［J］. arXiv preprint arXiv：1610.01733， 2016.
［16］董瑶，葛莹莹，郭鸿湧，等. 基于深度强化学习的移动机器人路径规划［J］. 计算机工程与应用， 2019，55（13）：15-19.
［17］董永峰，杨琛，董瑶，等. 基于改进的DQN机器人路径规划［J］. 计算机工程与设计， 2021，42（2）：552-558.
［18］成怡，郝密密. 改进深度强化学习的室内移动机器人路径规划［J］. 计算机工程与应用， 2021，57（21）：256-262.
［19］邓悟. 基于深度强化学习的智能体避障与路径规划研宄与应用［D］. 成都：电子科技大学， 2019．
［20］ LIMA J， OLIVEIRA R， COSTA C. Trajectory simulation approach for autonomous vehicles path planning using deep reinforcement learning［J］. International Journal for Innovation Education and Research， 2020，8（12）：436-454.
［21］ BARTOLOMEI L， TEIXEIRA L， CHLI M. Semantic-aware active perception for UAVs using deep reinforcement learning［C］// 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems （IROS）. 2021：3101-3108.
［22］ HUANG Y， WEI G L， WANG Y X. V-D D3QN： The variant of double deep Q-learning network with dueling architecture［C］// 第37界中国控制会议论文. 2018：558-563.
［23］ HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2016：770-778.
［24］ HU J， SHEN L， ALBANIE S， et al. Squeeze-and-excitation networks［C］// Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2018：7132-7144.
［25］ KOENIG N， HOWARD A. Design and use paradigms for Gazebo， an open-source multi-robot simulator［C］// Proceedings of 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems （IROS 2004）. 2004：2149-2154.
［26］梁宏斌. 基于openAI Gym和DRL的移动机器人路径规划算法研究［D］. 重庆：重庆理工大学， 2021.
［27］ SCHULZ D， BEHNKE S. Real-time motion planning with 3D environment representations using a dynamic window approach［J］. Journal of Field Robotics， 2014，31（1）：92-115.

[1]	JIAO Jian, JI Yuanfa, SUN Xiyan, WU Jianhui, LIANG Weibin. Multiple Unmanned Aerial Vehicles Three-dimensional Cooperative Route Planning Based on Improved GWO Algorithm [J]. Computer and Modernization, 2024, 0(10): 1-6.
[2]	YANG Yufeng1, 2, XIA Xiaoyun2, CHEN Zefeng3, LIAO Weizhi2, LI Jiwu2. Dung Beetle Optimization Algorithm Integrating Multiple Strategies for Take-out Order Distribution Route Optimization [J]. Computer and Modernization, 2024, 0(09): 25-32.
[3]	WANG Qian, HUANG Miao, Tao Lili. PCB Board Welding Path Optimization Based on Improved Cuckoo Algorithm [J]. Computer and Modernization, 2024, 0(07): 82-86.
[4]	LI Peng, XU Luo. An Autonomous Navigation Method for Intelligent Vehicles in Urban Battlefield [J]. Computer and Modernization, 2024, 0(01): 92-98.
[5]	HAN Xue. Multi Path Planning Based on Constrained Clustering and Particle Swarm Optimization [J]. Computer and Modernization, 2023, 0(08): 7-11.
[6]	DENG Yun-zheng, HUANG Yi-hu. Improved DWA Obstacle Avoidance Algorithm in Dense Obstacle Environment [J]. Computer and Modernization, 2023, 0(07): 48-53.
[7]	DU Chuan-sheng, GAO Huan-bing, HOU Yu-xiang, WANG Zi-jian. Path Planning of Fire Robot Based on Improved Bidirectional A* Algorithm [J]. Computer and Modernization, 2023, 0(04): 15-19.
[8]	WANG Zi-wei. Multi-robot Path Planning Based on Double Fuzzy Inference and Improved DWA Algorithm [J]. Computer and Modernization, 2023, 0(04): 20-25.
[9]	TIAN Xiao-zhuang, LI Song, FU Guo-ping, TAN Qi-yun, SHAN De-shuai, WANG Wei-guang, WANG Zhu. Time-optimal Obstacle Avoidance Path Planning for UAV Inspection [J]. Computer and Modernization, 2023, 0(03): 43-47.
[10]	SHEN Hao-yang, CHEN Xiao-lei, YUAN Jun-ling, HAN Lu. Fast Path Planning Algorithm in 3D Space for UAV [J]. Computer and Modernization, 2023, 0(02): 6-11.
[11]	WANG Kuo, HAO Fu-zhen. Optimization of Plant Growth Simulation Algorithm for Vehicle Routing Problem with Time Windows [J]. Computer and Modernization, 2022, 0(12): 26-32.
[12]	ZHANG Xiao-fang, FENG Hui-fang. Dynamic Optimal Path Planning Based on Trajectory Big Data [J]. Computer and Modernization, 2021, 0(11): 82-88.
[13]	LIU Yu-fei, ZHANG Xu-mei, LIANG Xiao-lei. Research Hotspot Analysis of AGV Path Planning Based on CiteSpace [J]. Computer and Modernization, 2021, 0(05): 112-119.
[14]	LI Shu-xia, YANG Jun-cheng, . An Improved Complete Path Planning Algorithm [J]. Computer and Modernization, 2021, 0(02): 100-103.
[15]	WANG Zhen-ting, CHEN Yong-fu, LIU Tian. Multi-robot Scheduling Method in Intelligent Warehouse [J]. Computer and Modernization, 2020, 0(07): 65-70.