基于深度强化学习DDPG算法的投资组合管理

doi:10.3969/j.issn.1006-2475.2018.05.019

计算机与现代化

基于深度强化学习DDPG算法的投资组合管理

（1.南开大学商学院,天津300071； 2.南开大学中国公司治理研究院,天津300071； 
3.中国特色社会主义经济建设协同创新中心,天津300071）

收稿日期:2018-02-08 出版日期:2018-06-13 发布日期:2018-06-13
作者简介:齐岳（1970-），男，天津人，南开大学商学院、南开大学中国公司治理研究院、中国特色社会主义经济建设协同创新中心教授，博士，研究方向：投资组合管理，基金管理；黄硕华（1996-），女，北京人，本科生，研究方向：计算机技术应用，投资组合管理。
基金资助:
国家自然科学基金重点资助项目（71533002）；教育部人文社会科学重点研究基地重大项目(16JJD630003)

Portfolio Management Based on DDPG Algorithm of Deep Reinforcement Learning

(1. School of Business, Nankai University, Tianjing 300071, China； 
2. China Institute of Corporate Governance, Nankai University, Tianjing 300071, China; 
3. Socialist Economic Construction with Chinese Characteristics Collaborative Innovation Center, Tianjing 300071, China)

Received:2018-02-08 Online:2018-06-13 Published:2018-06-13

摘要/Abstract

摘要： 将深度强化学习技术应用于投资组合管理，采用深度强化学习中的深度确定性策略梯度DDPG（Deep Deterministic Policy Gradient）算法，通过限制单只股票的投资权重，分散风险，并采用丢弃算法（Dropout），即在训练模型时随机丢弃节点，解决过拟合问题。以中国股市为例，选取16只中证100指数成分股作为风险资产进行实验。结果表明，本文基于深度强化学习方法构建的投资组合，在实验期间的价值增幅显著高于对照组（等权重组合），2年达到65%，约为对照组的2.5倍，表明了本文方法的有效性。而且通过进一步实验，表明了当用于训练的数据离测试数据时间越近，则本文构建的投资组合表现越好。

关键词: 深度强化学习, 深度确定性策略梯度, 投资组合管理

Abstract: This paper applies DRL(Deep Reinforcement Learning) technology to portfolio management and adopts DDPG (Deep Deterministic Policy Gradient) algorithm. By limiting the weight of individual stock, risk diversification is achieved and by using Dropout, that is, randomly dropping some nodes when training models, over-fitting problems are solved. Taking Chinese stock market as an example, this paper selects 16 of China securities 100 index constituent stocks as risky assets. The experimental results show that the 2-year accumulative return rate of the portfolio constructed in this paper reaches 65%， which is about 2.5 times of that of the control group（a portfolio with weights evenly distributed among the same 16 stocks). This strongly indicates the effectiveness of the method. Moreover, through further experiments, this paper indicates that the closer the data used for training is from the test data, the better the performance of the portfolio constructed in this paper.

Key words: deep reinforcement learning(DRL), deep deterministic policy gradient(DDPG), portfolio management

中图分类号:

TP399

齐岳1，2，3，黄硕华1. 基于深度强化学习DDPG算法的投资组合管理[J]. 计算机与现代化, doi: 10.3969/j.issn.1006-2475.2018.05.019.

QI Yue1，2，3, HUANG Shuo-hua1. Portfolio Management Based on DDPG Algorithm of Deep Reinforcement Learning[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2018.05.019.

参考文献

［1］ Markowitz H M. Portfolio Selection: Efficient Diversification of Investments［M］. 2nd Ed. New Jersey: Wiley, 1991.
［2］ Qi Yue, Wang Zhi-hao, Zhang Su. On analyzing and detecting multiple optima of portfolio optimization ［J］. Journal of Industrial and Management Optimization, 2018,14(1):309-323.
［3］ Qi Yue. On the criterion vectors of lines of portfolio selection with multiple quadratic and multiple linear objectives［J］. Central European Journal of Operations Research, 2017,25(1):145-158.
［4］赵冬斌,邵坤,朱圆恒,等. 深度强化学习综述: 兼论计算机围棋的发展［J］. 控制理论与应用, 2016,33(6):701-717.
［5］李玉鑑,张婷. 深度学习导论及案例分析［M］. 北京:机械工业出版社, 2016.
［6］王雪松，朱美强，程玉虎. 强化学习原理及其应用［M］. 北京:科学出版社, 2014.
［7］ Selvin S, Vinayakumar R, Gopalakrishnan E A, et al. Stock price prediction using LSTM, RNN and CNN-sliding window model［C］// 2017 International Conference on Advances in Computing, Communications and Informatics. 2017:1643-1647.
［8］ Khare K, Darekar O, Gupta P, et al. Short term stock price prediction using deep learning［C］//2017 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology. 2017:482-486.
［9］曾志平，萧海东，张新鹏. 基于DBN的金融时序数据建模与决策［J］. 计算机技术与发展, 2017,27(4):1-5.
［10］李文鹏，高宇菲，钱佳佳，等. 深度学习在量化投资中的应用［J］. 统计与管理, 2017(8):104-106.
［11］胡文伟，胡建强，李湛，等. 基于强化学习算法的自适应配对交易模型［J］. 管理科学, 2017,30(2):148-160.
［12］Li Wei, Liao Jian. A comparative study on trend forecasting approach for stock price time series［C］//2017 11th IEEE International Conference on Anti-counterfeiting, Security, and Identification. 2017:74-78.
［13］孙瑞奇. 基于LSTM神经网络的美股股指价格趋势预测模型的研究［D］. 北京:首都经济贸易大学, 2015.
［14］Mnih V, Kavukcuoglu K, Silver D, et al. Playing Atari with deep reinforcement learning［C］// Advances in Neural Information Processing Systems. 2013:1-9.
［15］Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning［J］. Nature, 2015,518(7540):529-533.
［16］Deng Yue, Bao Feng, Kong Youyong, et al. Deep direct reinforcement learning for financial signal representation and trading［J］. IEEE Transactions on Neural Networks and Learning Systems, 2017,28(3):1-12.
［17］Jiang Zhengyao, Xu Dixing, Liang Jinjun. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem ［DB/OL］. https://arxiv.org/abs/1706.10059, 2017-11-30.
［18］Zhang Chi, Zhang Limian, Chen Corey. Deep Reinforcement Learning for Portfolio Management［DB/OL］. http://www-scf.usc.edu/~zhan527/post/cs599, 2017-11-18.
［19］Srivastava N， Hinton G， Krizhevsky A. Dropout: A simple way to prevent neural networks from overfitting［J］. Journal of Machine Learning Research, 2014,15(1):1929-1958.
［20］Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks［J］. Science, 2006,313(5786):504-507.
［21］陈学松,杨宜民. 强化学习研究综述［J］. 计算机应用研究, 2010,27(8):2835-2844.
［22］李晨溪,曹雷,张永亮,等. 基于知识的深度强化学习研究综述［J］. 系统工程与电子技术, 2017,39(11):2603-2613.
［23］Silver D, Huang Aja, Maddison C J, et al. Mastering the game of Go with deep neural networks and tree search［J］. Nature, 2016,529(7587):484-489.
［24］Silver D, Schrittwieser J, Simonyan K, et al. Mastering the game of Go without human knowledge［J］. Nature, 2017,550(7676):354-359.
［25］Timothy P L, Jonathan J H, Alexander P. Continuous control with deep reinforcement learning［C］// International Conference on Learning Representations. 2016:1-14.

[1]	李鹏, 徐珞. 一种面向城市战场的智能车自主导航方法[J]. 计算机与现代化, 2024, 0(01): 92-98.
[2]	张国有, 宋世峰. 基于D3QN的交通灯控制优化[J]. 计算机与现代化, 2023, 0(07): 30-35.
[3]	赖建彬, 冯刚. 一种基于混合样本的经验回放策略[J]. 计算机与现代化, 2023, 0(06): 33-38.
[4]	丁忠林, 李洋, 曹委, 谈宇浩, 徐波. 基于深度Q学习的电力物联网任务卸载研究[J]. 计算机与现代化, 2022, 0(11): 75-80.
[5]	吴水明, 吉志远, 王震宇, 景栋盛. 基于Dueling-DDQN的电力信息网络入侵检测算法[J]. 计算机与现代化, 2021, 0(12): 43-47.
[6]	刘露, 申国伟, 郭春, 崔允贺, 蒋朝惠, 伍大勇. 一种基于深度强化学习的Spark Streaming参数优化方法[J]. 计算机与现代化, 2021, 0(10): 49-56.
[7]	王鹏勇, 陈龚涛, 赵江烁. 基于深度强化学习的机场出租车司机决策方法[J]. 计算机与现代化, 2020, 0(08): 94-99.
[8]	袁雯，刘惠义. 基于深度Q网络的仿人机器人步态优化[J]. 计算机与现代化, 2019, 0(04): 47-.
[9]	彭琛,韩立新. 基于深度强化学习的计步方法[J]. 计算机与现代化, 2019, 0(01): 63-.

基于深度强化学习DDPG算法的投资组合管理

Portfolio Management Based on DDPG Algorithm of Deep Reinforcement Learning

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 9

编辑推荐

Metrics

本文评价