一种基于混合样本的经验回放策略

doi:10.3969/j.issn.1006-2475.2023.06.006

计算机与现代化 ›› 2023, Vol. 0 ›› Issue (06): 33-38.doi: 10.3969/j.issn.1006-2475.2023.06.006

一种基于混合样本的经验回放策略

赖建彬, 冯刚

华南师范大学计算机学院,广东广州 510635

收稿日期:2022-07-11 修回日期:2022-08-24 出版日期:2023-06-28 发布日期:2023-06-28
通讯作者: 冯刚（1967—）,男,湖南衡阳人,副教授,博士,研究方向：深度强化学习,E-mail： fenggang@scnu.edu.com。
作者简介:赖建彬（1997—）,男,广东广州人,硕士研究生,研究方向：深度强化学习,E-mail： laijianbin67@163.com。

An Experience Replay Strategy Based on Mixed Samples

LAI Jian-bin, FENG Gang

School of Computer Science, South China Normal University, Guangzhou 510635, China

Received:2022-07-11 Revised:2022-08-24 Online:2023-06-28 Published:2023-06-28

摘要/Abstract

摘要： 经验回放策略已经成为深度强化学习算法的一个重要组成部分,它不仅可以加速深度强化学习算法收敛,而且还能加强智能体的表现。主流的经验回放策略使用均匀抽样、优先经验回放、专家经验回放等方法加速学习。为了进一步提高深度强化学习中经验样本的利用率,本文提出一种基于混合样本的经验回放策略（ER-MS）。该策略主要使用立即学习最新经验和复习成功经验2种方法,对智能体与环境交互产生的最新样本进行立即学习,同时使用额外的经验缓存池保存成功回合样本进行经验回放。实验表明,基于混合样本的经验回放策略结合DDPG算法能够在OpenAI mujoco任务中取得更优成绩。

关键词: 经验回放, 深度强化学习, 专家经验

Abstract: Experience replay strategy has become an important part of deep reinforcement learning algorithm. It can not only accelerate the convergence of deep reinforcement learning algorithm, but also enhance the performance of agents. Mainstream experience replay strategies use uniform sampling, priority experience replay, expert experience replay and other methods to accelerate learning. In order to further improve the utilization of experience samples in deep reinforcement learning, this paper proposes an experience replay strategy based on mixed samples （ER-MS）. This strategy mainly uses two methods： immediate learning of the latest experience and review of successful experience. It immediately learns the latest samples generated by the interaction between the agent and the environment, and uses an additional experience buffer pool to save the samples of successful rounds for experience replay. Experiments show that the experience replay strategy based on mixed samples combined with DDPG algorithm can achieve better results in Open AI mujoco task.

Key words: experience replay, deep reinforcement learning, expert experience

中图分类号:

TP301.6

赖建彬, 冯刚. 一种基于混合样本的经验回放策略[J]. 计算机与现代化, 2023, 0(06): 33-38.

LAI Jian-bin, FENG Gang. An Experience Replay Strategy Based on Mixed Samples[J]. Computer and Modernization, 2023, 0(06): 33-38.

参考文献 27

[1]	LECUN Y, BENGIO Y, HINTON G.Deep learning[J]. Nature, 2015,521(7553):436-444.
[2]	SUTTON R S, BARTO A G.Reinforcement Learning: An Introduction[M]. MIT Press, 2018.
[3]	刘朝阳,穆朝絮,孙长银. 深度强化学习算法与应用研究现状综述[J]. 智能科学与技术学报, 2020,2(4):314-326.
[4]	杨思明,单征,丁煜,等. 深度强化学习研究综述[J]. 计算机工程, 2021,47(12):19-29.
[5]	MNIH V, KAVUKCUOGLU K, SILVER D, et al.Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540):529-533.
[6]	SILVER D, HUANG A, MADDISON C J, et al.Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016,529(7587):484-489.
[7]	SILVER D, SCHRITTWIESER J, SIMONYAN K, et al.Mastering the game of Go without human knowledge[J]. Nature, 2017,550(7676):354-359.
[8]	刘威,张东霞,王新迎,等. 基于深度强化学习的电网紧急控制策略研究[J]. 中国电机工程学报, 2018,38(1):109-119.
[9]	李航,李国杰,汪可友. 基于深度强化学习的电动汽车实时调度策略[J]. 电力系统自动化, 2020,44(22):161-167.
[10]	孔松涛,刘池池,史勇,等. 深度强化学习在智能制造中的应用展望综述[J]. 计算机工程与应用, 2021,57(2):49-59.
[11]	齐义文,张弛,陈禹西. 基于强化学习方法的变循环航空发动机推力控制[J]. 沈阳航空航天大学学报, 2022,39(3):40-49.
[12]	董豪,丁子涵,仉尚航. 深度强化学习:基础、研究与应用[M]. 北京:电子工业出版社, 2020.
[13]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning[J]. arXiv preprint arXiv:1312.5602, 2013.
[14]	PAN Y, ZAHEER M, WHITE A, et al. Organizing experience: A deeper look at replay mechanisms for sample-based planning in continuous state domains[J]. arXiv preprint arXiv:1806.04624, 2018.
[15]	ANDRYCHOWICZ M, WOLSKI F, RAY A, et al.Hindsight experience replay[C]// Advances in Neural Information Processing Systems 30 (NIPS 2017). 2017.
[16]	LIN L J.Self-improving reactive agents based on reinforcement learning, planning and teaching[J]. Machine Learning, 1992,8(3):293-321.
[17]	SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[J]. arXiv preprint arXiv:1511.05952, 2015.
[18]	HESTER T, VECERIK M, PIETQUIN O, et al.Deep q-learning from demonstrations[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018,32(1). DOI:10.1609/aaai.v32i1.11757.
[19]	OH J, GUO Y, SINGH S, et al.Self-imitation learning[C]// Proceedings of the 35th International Conference on Machine Learning. 2018:3878-3887.
[20]	LUO J L, LI H.Dynamic experience replay[C]// Proceedings of the Conference on Robot Learning. 2020:1191-1200.
[21]	LIU X H, XUE Z, PANG J, et al.Regret minimization experience replay in off-policy reinforcement learning[C]// Advances in Neural Information Processing Systems 34 (NeurIPS 2021). 2021.
[22]	ZHANG S, SUTTON R S. A deeper look at experience replay[J]. arXiv preprint arXiv:1712.01275, 2017.
[23]	SUN P, ZHOU W, LI H.Attentive experience replay[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020,34(4):5900-5907.
[24]	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. arXiv preprint arXiv:1509.02971, 2015.
[25]	VAN HASSELT H, GUEZ A, SILVER D.Deep reinforcement learning with double q-learning[C]// Proceedings of the 30th AAAI conference on artificial intelligence. 2016,30(1).
[26]	WENG J, CHEN H, YAN D, et al.Tianshou: A highly modularized deep reinforcement learning library[J]. arXiv preprint arXiv:2107.14171, 2021.
[27]	BROCKMAN G, CHEUNG V, PETTERSSON L, et al. OpenAI gym[J]. arXiv preprint arXiv:1606.01540, 2016.

一种基于混合样本的经验回放策略

An Experience Replay Strategy Based on Mixed Samples

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献 27

相关文章 9

编辑推荐

Metrics

本文评价

[1]	李鹏, 徐珞. 一种面向城市战场的智能车自主导航方法[J]. 计算机与现代化, 2024, 0(01): 92-98.
[2]	张国有, 宋世峰. 基于D3QN的交通灯控制优化[J]. 计算机与现代化, 2023, 0(07): 30-35.
[3]	丁忠林, 李洋, 曹委, 谈宇浩, 徐波. 基于深度Q学习的电力物联网任务卸载研究[J]. 计算机与现代化, 2022, 0(11): 75-80.
[4]	吴水明, 吉志远, 王震宇, 景栋盛. 基于Dueling-DDQN的电力信息网络入侵检测算法[J]. 计算机与现代化, 2021, 0(12): 43-47.
[5]	刘露, 申国伟, 郭春, 崔允贺, 蒋朝惠, 伍大勇. 一种基于深度强化学习的Spark Streaming参数优化方法[J]. 计算机与现代化, 2021, 0(10): 49-56.
[6]	王鹏勇, 陈龚涛, 赵江烁. 基于深度强化学习的机场出租车司机决策方法[J]. 计算机与现代化, 2020, 0(08): 94-99.
[7]	袁雯，刘惠义. 基于深度Q网络的仿人机器人步态优化[J]. 计算机与现代化, 2019, 0(04): 47-.
[8]	彭琛,韩立新. 基于深度强化学习的计步方法[J]. 计算机与现代化, 2019, 0(01): 63-.
[9]	齐岳1，2，3，黄硕华1. 基于深度强化学习DDPG算法的投资组合管理[J]. 计算机与现代化, 2018, 0(05): 93-.