基于图神经网络的多智能体强化学习对抗策略检测算法

doi:10.3969/j.issn.1006-2475.2025.04.007

摘要/Abstract

摘要： 在多智能体环境中，强化学习模型在应对对抗攻击方面存在安全漏洞，容易遭受对抗攻击。其中基于对抗策略的对抗攻击由于不直接修改受害者的观测，对其进行防御的难度更大。为解决这一问题，本文提出一种基于图神经网络的对抗策略检测算法，旨在有效识别智能体间的恶意行为。通过在智能体协作过程中采用替代对抗策略训练图神经网络作为对抗策略检测器，根据智能体局部观测计算其他智能体的信任分数来检测对抗策略。本文的检测方法提供2种粒度的检测：对局级别的对抗检测以非常高的精度检测对抗策略；时间步级别的对抗检测可以在对局初期进行对抗检测，及时发现对抗攻击。在星际争霸平台上进行一系列实验，实验结果表明，本文所提出的检测方法在检测最先进的基于对抗策略的对抗攻击时最高可以达到1.0的AUC值，优于最先进的检测方法。本文检测方法比现有的方法能够更快地检测出对抗策略，最快可以在第5个时间步检测出对抗攻击。将本文检测方法应用于对抗防御，使受攻击对局提升最高61个百分点的胜率。此外实验结果显示了本文的算法具有很强的泛化性，本文的检测方法无需再次训练，可以直接用来检测基于观测的对抗攻击。因此，本文提出的方法为多智能体环境中的强化学习模型提供了一种有效的对抗攻击检测机制。

关键词: 强化学习, 多智能体系统, 对抗攻击, 对抗检测, 图神经网络

Abstract: In a multi-agent environment， the reinforcement learning model has security vulnerabilities in coping with adversarial attacks and is susceptible to adversarial attacks， of which adversarial policy-based adversarial attacks are more difficult to defend against because they do not directly modify the victim’s observations. To solve this problem， this paper proposes a graph neural network-based adversarial policy detection algorithm， which aims to effectively identify malicious behaviors among agents. This paper detects adversarial policy by training the graph neural network as an adversarial policy detector by employing alternative adversarial policies during the collaboration process of the agents， and calculates the trust scores of the other agent based on the local observations of the agents. The detection method in this paper provides two levels of granularity； adversarial detection at the game level detects adversarial policies with very high accuracy， and time-step level adversarial detection allows for adversarial detection at the early stage of the game and timely detection of adversarial attacks. This paper conducts a series of experiments on the StarCraft platform， whose results show that the detection method proposed in this paper can achieve an AUC value of up to 1.0 in detecting the most advanced adversarial policy-based adversarial attacks， which is better than the state-of-the-art detection methods. The detection method in this paper can detect adversarial policy faster than existing methods， and can detect the adversarial attack at the 5th time step at the earliest. Applying this paper’s detection method to adversarial defense improves the win rate of the attacked game by up to 61 percentage points. In addition experimental results show that the algorithm in this paper is highly generalizable and the detection method in this paper does not need to be trained again and can be directly used to detect observation-based adversarial attacks. Therefore， the method proposed in this paper provides an effective adversarial attack detection mechanism for reinforcement learning models in a multi-agent environment.

Key words: , reinforcement learning, multi-agent system, adversarial attack, adversarial detection, graph neural network

中图分类号:

TP391

孙启宁1, 桂智明1, 刘艳芳2, 范鑫鑫3, 路云峰4. 基于图神经网络的多智能体强化学习对抗策略检测算法[J]. 计算机与现代化, 2025, 0(04): 42-49.

SUN Qining1, GUI Zhiming1, LIU Yanfang2, FAN Xinxin3, LU Yunfeng4. Graph Neural Network-based Multi-agent Reinforcement Learning for Adversarial Policy Detection Algorithm[J]. Computer and Modernization, 2025, 0(04): 42-49.

参考文献

［1］ GUPTA J K， EGOROV M， KOCHENDERFER M. Cooperative multi-agent control using deep reinforcement learning［C］// Autonomous Agents and Multiagent Systems： AAMAS 2017 Workshops. Springer， 2017：66-83.
［2］ LI T X， ZHU K， LUONG N C， et al. Applications of multi-agent reinforcement learning in future Internet： A comprehensive survey［J］. IEEE Communications Surveys & Tutorials， 2022，24（2）：1240-1279.
［3］ CANESE L， CARDARILLI G C， DI NUNZIO L D， et al. Multi-agent reinforcement learning： A review of challenges and applications［J］. Applied Sciences， 2021，11（11）. DOI: 10.3390/app11114948.
［4］ LI K. Novel multi-agent reinforcement learning for maximizing throughput in UAV-Enabled 5G networks［J］. Wireless Networks， 2023，30：7029-7040.
［5］ GLEAVE A， DENNIS M， WILD C， et al. Adversarial policies： Attacking deep reinforcement learning［J］. arXiv preprint arXiv：1905.10615， 2019.
［6］ SAMVELYAN M， RASHID T， DE WITT C S， et al. The starcraft multi-agent challenge［J］. arXiv preprint arXiv：1902.04043， 2019.
［7］ GUO W B， WU X， HUANG S， et al. Adversarial policy learning in two-player competitive games［C］// International Conference on Machine Learning. PMLR， 2021： 3910-3919.
［8］ ZHANG H M， SUN K， XU B， et al. A simple unified framework for anomaly detection in deep reinforcement learning［J］. arXiv preprint arXiv：2109.09889， 2021.
［9］ SEDLMEIER A， MULLER R， ILLIUM S， et al. Policy entropy for out-of-distribution classification［C］// Artificial Neural Networks and Machine Learning-ICANN. Springer， 2020：420-431.
［10］ LI S M， GUO J， XIU J Q， et al. Attacking cooperative multi-agent reinforcement learning by adversarial minority influence［J］. arXiv preprint arXiv：2302.03322， 2023.
［11］ SILVER D， HUANG A， MADDISION C J， et al. Mastering the game of Go with deep neural networks and tree search［J］. Nature， 2016，529（7587）：484-489.
［12］ GARCIA F， RACHELSON E. Markov decision processes［M］// Markov Decision Processes in Artificial Intelligence. WILEY， 2013：1-38.
［13］ SHEN G C， YANG W. Review on Dec-POMDP model for ARL algorithms［C］// Proceedings of 4th International Conference on Wireless Communications and Applications（ICWCA 2020）. Springer ， 2022：29-35.
［14］ TAMPUU A， MATIISEN T， KODELJA D， et al. Multiagent cooperation and competition with deep reinforcement learning［J］. PlOS One， 2017，12（4）. DOI： 10.1371/journal.pone.0172395.
［15］ SUNEHAG P， LEVER G， GRUSLYS A， et al. Value-decomposition networks for cooperative multi-agent learning［J］. arXiv preprint arXiv：1706.05296， 2017.
［16］ RASHID T， FARQUHAR G， PENG B， et al. Weighted QMIX： Expanding monotonic value function factorisation for deep multi-agent reinforcement learning［C］// Proceedings of the 34th International Conference on Neural Information Processing Systems. ACM， 2020：10199-10210.
［17］ FOERSTER J， FARQUHAR G， AFOURAS T， et al. Counterfactual multi-agent policy gradients［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI， 2018，32（1）：2974-2982.
［18］ SON K， KIM D， KANG W J， et al. Qtran： Learning to factorize with transformation for cooperative multi-agent reinforcement learning［C］// 36th International Conference on Machine Learning. PMLR， 2019：5887-5896.
［19］ WANG J H， REN Z Z， LIU T， et al. QPLEX： Duplex dueling multi-agent q-learning［J］. arXiv preprint arXiv：2008.01062， 2020.
［20］ PENG B， RASHID T， DEWITT C A S， et al. FACMAC： Factored multi-agent centralised policy gradients［C］// Proceedings of the 35th International Conference on Neural Information Processing Systems. ACM， 2021：12208-12221.
［21］ LOWE R， WU Y， TAMAR A， et al. Multi-agent actor-critic for mixed cooperative-competitive environments［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems， 2017：6382-6393.
［22］ DE WITT C S， GUPTA T， MAKOVIICHUK D， et al. Is independent learning all you need in the starcraft multi-agent challenge?［J］. arXiv preprint arXiv：2011.09533， 2020.
［23］ YU C， VELU A， VINITSKY E， et al. The surprising effectiveness of PPO in cooperative multi-agent games［C］// Proceedings of the 36th International Conference on Neural Information Processing Systems. ACM， 2022：24611-24624.
［24］ GOODFELLOW I J， SHLENS J， SZEGEDY C. Explaining and harnessing adversarial examples［J］. arXiv preprint arXiv：1412.6572， 2014.
［25］ HUANG S， PAPERNOT N， GOODFELLOW I， et al. Adversarial attacks on neural network policies［J］. arXiv preprint arXiv：1702.02284， 2017.
［26］ PATTANAIK A， TAMG Z Y， LIU S J， et al. Robust deep reinforcement learning with adversarial attacks［J］. arXiv preprint arXiv：1712.03632， 2017.
［27］ WU X， GUO W B， WEI H， et al. Adversarial policy training against deep reinforcement learning［C］// 30th USENIX Security Symposium（USENIX Security 21）. USENIX， 2021：1883-1900.
［28］ SCHULNMAN J， WOLSKI F， DHARIWAL P， et al. Proximal policy optimization algorithms［J］. arXiv preprint arXiv：1707.06347， 2017.
［29］ YI J， CHEN Y， LI J， et al. Predictive model performance： Offline and online evaluations［C］// Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM， 2013：1294-1302.
［30］ LIN J Y， DZEPAROSKA K， ZHANG S Q， et al. On the robustness of cooperative multi-agent reinforcement learning［C］// 2020 IEEE Security and Privacy Workshops （SPW）. IEEE， 2020：62-68.

[1]	王鑫, 肖韬睿. 基于生成对抗网络的人脸识别对抗攻击[J]. 计算机与现代化, 2023, 0(10): 115-120.
[2]	李世宝, 王杰伟, 崔学荣, 刘建航, 黄庭培. 基于图像着色的无限制攻击[J]. 计算机与现代化, 2022, 0(11): 52-59.
[3]	衡红军;高燕;习开磊. 基于MAS的区域管制系统模型[J]. 计算机与现代化, 2009, 1(10): 52-4.