UBS/
Articles/
2508001129
  • Open Access
  • Review

A Conceptual Framework for Performance Evaluation and Classification of Multi-Agent Hybrid Game Problems Based on Reinforcement Learning: Impact Factors and Evaluation Metrics

  • Lihua Zhang 1, *,   
  • Xiaolei Yuan 2, *,   
  • Wenqi Jia 3,   
  • Angelica Elhokayem 4

Received: 13 Jul 2025 | Revised: 04 Aug 2025 | Accepted: 18 Aug 2025 | Published: 25 Aug 2025

Abstract

Multi-agent reinforcement learning (MARL) method for hybrid game is characterized by the coexistence of cooperativeness and competitiveness. This method has become more applicable for their complexity and wide applications in fields of architectural optimization, urban resource scheduling, and intelligent environmental control. However, the absence of a systematic framework for MARL algorithms’ assessment limits practical deployment in such fields. To address this concern, this paper intends to conduct a critical literature review to delve into the decisive factors of the scenarios and further develop conceptual framework for the subdivision classification, algorithm design and evaluation of hybrid games. The core research goal is to expound the differentiated classification system of hybrid game and the key determinants that significantly affect the performance of multi-agent reinforcement learning algorithms. Therefore, this paper proposes an evaluation conceptual framework for classification of hybrid games based on objectives and application scenarios, providing a reference for developers to select and design appropriate MARL algorithms and evaluate their performance according to different types of cooperative games.

References 

  • 1.
    Hu, K.; Li, M.; Song, Z.; et al. A review of research on reinforcement learning algorithms for multi-agents. Neurocomputing 2024, 599, 128068.
  • 2.
    Nguyen, T.T.; Nguyen, N.D.; Nahavandi, S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Trans. Cybern. 2020, 50, 3826–3839.
  • 3.
    Du, W.; Ding, S. A survey on multi-agent deep reinforcement learning: From the perspective of challenges and applications. Artif. Intell. Rev. 2021, 54, 3215–3238.
  • 4.
    Zhang, K.; Yang, Z.; Başar, T. Multi-agent reinforcement learning: A selective overview of theories and algorithms. In Handbook of Reinforcement Learning and Control; Springer: Berlin/Heidelberg, Germany, 2021; pp. 321–384.
  • 5.
    Li, H.; Huang, W.; Duan, Z.; et al. A survey on algorithms for Nash equilibria in finite normal-form games. Comput. Sci. Rev. 2024, 51, 100613.
  • 6.
    Li, Z.; Wu, W.R.; Guo, Y.L.; et al. Embodied Multi-Agent Systems: A Review. IEEE-CAA J. Autom. 2025, 12, 1095–1116. https://doi.org/10.1109/Jas.2025.125552.
  • 7.
    Dou, Y.H.; Xing, G.S.; Ma, A.H.; et al. A review of event-triggered consensus control in multi-agent systems. J. Control Decis. 2025, 12, 1–23. https://doi.org/10.1080/23307706.2024.2388551.
  • 8.
    Wang, Y.M.; Lin, H.; Lam, J.; et al. Differentially private consensus and distributed optimization in multi-agent systems: A review. Neurocomputing 2024, 597, 127986. https://doi.org/10.1016/j.neucom.2024.127986.
  • 9.
    Amirkhani, A.; Barshooi, A.H. Consensus in multi-agent systems: A review. Artif. Intell. Rev. 2022, 55, 3897–3935. https://doi.org/10.1007/s10462-021-10097-x.
  • 10.
    Shakya, A.K.; Pillai, G.; Chakrabarty, S. Reinforcement learning algorithms: A brief survey. Expert Syst. Appl. 2023, 231, 120495.
  • 11.
    Luo, F.M.; Xu, T.; Lai, H.; et al. A survey on model-based reinforcement learning. Sci. China Inf. Sci. 2024, 67, 121101. https://doi.org/10.1007/s11432-022-3696-5.
  • 12.
    Vázquez-Canteli, J.R.; Nagy, Z. Reinforcement learning for demand response: A review of algorithms and modeling techniques. Appl. Energy 2019, 235, 1072–1089. https://doi.org/10.1016/j.apenergy.2018.11.002.
  • 13.
    Noaeen, M.; Naik, A.; Goodman, L.; et al. Reinforcement learning in urban network traffic signal control: A systematic literature review. Expert Syst. Appl. 2022, 199, 116830. https://doi.org/10.1016/j.eswa.2022.116830.
  • 14.
    Fu, G.T.; Jin, Y.W.; Sun, S.A.; et al. The role of deep learning in urban water management: A critical review. Water Res. 2022, 223, 118973. https://doi.org/10.1016/j.watres.2022.118973.
  • 15.
    Dey, S.; Xu, H.; Fadali, M.S. Adaptive distributed formation control for multi-group large-scale multi-agent systems: A hybrid game approach. IFAC-Pap. 2023, 56, 5482–5487.
  • 16.
    Khoury, J.; Nassar, M. A hybrid game theory and reinforcement learning approach for cyber-physical systems security. In Proceedings of the NOMS 2020 IEEE/IFIP Network Operations and Management Symposium, Budapest, Hungary, 20–24 April 2020.
  • 17.
    Gan, W.; Qiao, L. Many-Versus-Many UUV Attack-Defense Game in 3D Scenarios Using Hierarchical Multi-Agent Reinforcement Learning. IEEE Internet Things J. 2025, 12, 23479–23494.
  • 18.
    Fitch, N.; Clancy, D. Genetic Programming+ Multi-Agent Reinforcement Learning: Hybrid Approaches for Decision Processes. In Proceedings of the 2022 IEEE Aerospace Conference (AERO), Big Sky, MT, USA, 5–12 March 2022.
  • 19.
    Qiu, S.; Li, Z.; Pang, Z.; et al. Multi-agent optimal control for central chiller plants using reinforcement learning and game theory. Systems 2023, 11, 136.
  • 20.
    Yehoshua, R.; Amato, C. Hybrid Independent Learning in Cooperative Markov Games. In Proceedings of the Distributed Artificial Intelligence: Second International Conference, DAI 2020, Nanjing, China, 24–27 October 2020.
  • 21.
    Jiang, T.Y.; Ju, P.; Lin, Z.J.; et al. Competitive Incentive Mechanism for Multi-Agents in Demand Response via a Hierarchical Game Considering Joint Uncertainties. IEEE Trans. Smart Grid 2025, 16, 2913–2925. https://doi.org/10.1109/Tsg.2025.3555590.
  • 22.
    Azimi, Z.; Afshar, A. Hybrid game-theoretic security assessment of cyber-physical power systems using partial-information multi-agent reinforcement learning. Sustain. Energy Grids 2025, 43, 101727. https://doi.org/10.1016/j.segan.2025.101727.
  • 23.
    Marc, F.; Degirmenciyan-Cartault, I.; El Fallah-Seghrouchni, A. An integral cycle for building feasible multi-agent plans. In Proceedings of the IEEE/WIC/ACM International Conference on Intelligent Agent Technology, Beijing, China, 24 September 2004.
  • 24.
    Zhou, L.Q.; Zheng, Y.S.; Zhao, Q.; et al. Game-based coordination control of multi-agent systems. Syst. Control Lett. 2022, 169, 105376. https://doi.org/10.1016/j.sysconle.2022.105376.
  • 25.
    Zhang, J.; Wang, G.; Yue, S.H.; et al. Multi-agent system application in accordance with game theory in bi-directional coordination network model. J. Syst. Eng. Electron. 2020, 31, 279–289. https://doi.org/10.23919/Jsee.2020.000006.
  • 26.
    Fley, B.; Florian, M. Trust and the economy of symbolic goods: A contribution to the scalability of open multi-agent systems. Lect. Notes Artif. Int. 2005, 3413, 176–198.
  • 27.
    Sajid, A.H.; Iftikhar, M.Z.; Kazmi, S.A.A.; et al. Multi-agent system for optimized energy management in multi-smart buildings via deregulated market system. Energy Rep. 2025, 14, 455–472. https://doi.org/10.1016/j.egyr.2025.06.031.
  • 28.
    Mareddy, S.K.R.; Maity, D. Learning Deceptive Strategies in Adversarial Settings: A Two-Player Game with Asymmetric Information. Appl. Sci. 2025, 15, 7805.
  • 29.
    Zhang, H.P.; Wang, Z.W.; Yu, J.L.; et al. Learning Simultaneous and Sequential Decisions in Multi-Agent Systems With Application to Traffic Signal Control. IEEE Trans. Intell. Transp. Syst. 2025, 26, 8257–8267. https://doi.org/10.1109/Tits.2025.3560712.
  • 30.
    Guan, Q.Q.; Zhang, H.X.; Yang, J.; et al. Design of energy management strategy for electro-hydraulic hybrid vehicles based on multi-agent systems. Energy Sources Part A 2025, 47, 2512996. https://doi.org/10.1080/15567036.2025.2512996.
  • 31.
    Liu, S.; Feng, Y.; Ren, N. Distributed Urban Road Network Signal Coordination Control Method Based on Game Theory and Multi-Agent Reinforcement Learning. In Proceedings of the 2024 12th International Conference on Information Systems and Computing Technology (ISCTech), Xi'an, China, 8–11 November 2024.
  • 32.
    Laudan, J.; Heinrich, P.; Nagel, K. High-Performance Simulations for Urban Planning: Implementing Parallel Distributed Multi-Agent Systems in MATSim. In Proceedings of the 2024 23rd International Symposium on Parallel and Distributed Computing (ISPDC), Chur, Switzerland, 8–10 July 2024. https://doi.org/10.1109/Ispdc62236.2024.10705395.
  • 33.
    Caprioli, C. The integration of multi-agent system and multicriteria analysis for developing participatory planning alternatives in urban contexts. Environ. Impact Assess. Rev. 2025, 113, 107855. https://doi.org/10.1016/j.eiar.2025.107855.
  • 34.
    do Nascimento, L.V.; de Oliveira, J.P.M. A multi-agent architecture for context sources integration in smart cities. Future Gener. Comput.Syst. 2025, 172, 107862. https://doi.org/10.1016/j.future.2025.107862.
  • 35.
    Kalyuzhnaya, A.; Mityagin, S.; Lutsenko, E.; et al. LLM Agents for Smart City Management: Enhancing Decision Support Through Multi-Agent AI Systems. Smart Cities 2025, 8, 19. https://doi.org/10.3390/smartcities8010019.
  • 36.
    Suanpang, P.; Jamjuntr, P. Optimizing Electric Vehicle Charging Recommendation in Smart Cities: A Multi-Agent Reinforcement Learning Approach. World Electr. Veh. J. 2024, 15, 67. https://doi.org/10.3390/wevj15020067.
  • 37.
    Lin, Z.; Xiuying, X.; Guiqing, Z.; et al. Building energy saving design based on multi-agent system. In Proceedings of the 2010 5th IEEE conference on industrial electronics and applications, Taichung, Taiwan, 15–17 June 2010.
  • 38.
    Qiu, D.; Wang, J.; Dong, Z.; et al. Mean-field multi-agent reinforcement learning for peer-to-peer multi-energy trading. IEEE Trans. Power Syst. 2022, 38, 4853–4866.
  • 39.
    Mancy, H.; Ghannam, N.E.; Abozeid, A.; et al. Decentralized multi-agent federated and reinforcement learning for smart water management and disaster response. Alex Eng. J. 2025, 126, 8–29. https://doi.org/10.1016/j.aej.2025.04.033.
  • 40.
    Zhang, K.; Bian, F.L. A design and application of a multi-agent system and GIS for simulation of expansion in urban planning. In Proceedings of the 2nd IEEE International Conference on Advanced Computer Control (Icacc 2010), Shenyang, China, 27–29 March 2010.
  • 41.
    Kong, G.H.; Chen, F.C.; Yang, X.H.; et al. Optimal Deception Asset Deployment in Cybersecurity: A Nash Q-Learning Approach in Multi-Agent Stochastic Games. Appl. Sci. 2024, 14, 357. https://doi.org/10.3390/app14010357.
  • 42.
    Haase, H.; Glake, D.; Clemen, T. Multi-Agent Imitation Learning for Agent Typification: A Proof-of-Concept for Markov Games with Chess. In Proceedings of the 2024 Annual Modeling and Simulation Conference, Annsim, Washington, DC, USA, 20–23 May 2024. https://doi.org/10.23919/Annsim61499.2024.10732442.
  • 43.
    Hernes, M.; Korczak, J.; Krol, D.; et al. Multi-agent platform to support trading decisions in the FOREX market. Appl. Intell. 2024, 54, 11690–11708. https://doi.org/10.1007/s10489-024-05770-x.
  • 44.
    Thiel, D.; Hovelaque, V.; Pham, D.N. A multi-agent model for optimizing supermarkets location in emerging countries. In Proceedings of the 2012 IEEE 13th International Symposium on Computational Intelligence and Informatics (CINTI), Budapest, Hungary, 20–22 November 2012.
  • 45.
    Çevikarslan, S. Research Joint Ventures in an R&D Driven Market with Evolving Consumer Preferences: An Evolutionary Multi-agent Based Modeling Approach. In Conference of the European Social Simulation Association; Springer Nature: Cham, Switzerland, 2024. https://doi.org/10.1007/978-3-031-57785-7_16.
  • 46.
    Agah, A. Robot teams, human workgroups and animal sociobiology: A review of research on natural and artificial multi-agent autonomous systems. Adv. Robot. 1996, 10, 523–545.
  • 47.
    Yeung, C.L.; Bunker, R.; Fujii, K. Unveiling Multi-Agent Strategies: A Data-Driven Approach for Extracting and Evaluating Team Tactics from Football Event and Freeze-Frame Data. J. Robot. Mechatron. 2024, 36, 603–617. https://doi.org/10.20965/jrm.2024.p0603.
  • 48.
    Song, Y.; Jiang, H.; Tian, Z.; et al. An Empirical Study on Google Research Football Multi-agent Scenarios. Mach. Intell. Res. 2024, 21, 549–570. https://doi.org/10.1007/s11633-023-1426-8.
  • 49.
    Ribeiro, A.F.A.; Lopes, A.C.C.; Ribeiro, T.A.; et al. Probability-Based Strategy for a Football Multi-Agent Autonomous Robot System. Robotics 2024, 13, 5. https://doi.org/10.3390/robotics13010005.
  • 50.
    Smit, A.; Engelbrecht, H.A.; Brink, W.; et al. Scaling multi-agent reinforcement learning to full 11 versus 11 simulated robotic football. Auton. Agents Multi-Agent Syst. 2023, 37, 20. https://doi.org/10.1007/s10458-023-09603-y.
  • 51.
    Gu, C.Y.; De Silva, V.; Artaud, C.; et al. Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football. In Proceedings of the 2023 Ieee 13th International Conference on Pattern Recognition Systems, Guayaquil, Ecuador, 4–7 July 2023. https://doi.org/10.1109/Icprs58416.2023.10179030.
  • 52.
    Yu, J.C.; Hu, B.M. Multi-agent Simulation of Knowledge Innovation Activities of University Research Team. In Proceedings of the 2009 International Conference on Industrial and Information, Haikou, China, 24–25 April 2009. https://doi.org/10.1109/Iis.2009.72.
  • 53.
    Ozsoyeller, D. TAP: Distributed team assignment in heterogeneous multi-agent systems. Future Gener. Comput. Syst. 2026, 174, 107925. https://doi.org/10.1016/j.future.2025.107925.
  • 54.
    Qiao, B.; Liu, K.; Guy, C. Multi-agent building control in shared environment. In Proceedings of the International Conference on Enterprise Information Systems, Funchal, Portugal, 12–16 June 2007.
  • 55.
    Kofinas, P.; Dounis, A.; Korkidis, P. Fuzzy Reinforcement Learning Multi-agent System for Comfort and Energy Management in Buildings. In Proceedings of the Sixth International Congress on Information and Communication Technology: ICICT 2021, London, UK, 25–26 February 2021.
  • 56.
    Li, X.; Zhang, P.; Gu, Q.; et al. Multi-agent Hybrid Architecture Design for Naval Warfare Game. In Proceedings of the International Conference on Autonomous Unmanned Systems, Shenyang, China, 19–21 September 2023.
  • 57.
    Panisson, A.R.; Farias, G.P. A Multi-level Semantics Formalism for Multi-Agent Microservices. In Proceedings of the Brazilian Conference on Intelligent Systems, Belém do Pará, Brazil, 17–21 November 2024.
  • 58.
    Rodriguez-Soto, M.; Lopez-Sanchez, M.; Rodriguez-Aguilar, J.A. Multi-objective reinforcement learning for designing ethical multi-agent environments. Neural Comput. Appl. 2023. https://doi.org/10.1007/s00521-023-08898-y.
  • 59.
    Yang, Y.; Gao, Y.; Ding, Z.; et al. Advancements in Q‐learning meta‐heuristic optimization algorithms: A survey. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2024, 14, e1548.
  • 60.
    Ferdous, J.; Murshed, M.; Meneguette, R.I.; et al. SARSA RL for Edge Connectivity Management in Vehicular Edge Networks. In Proceedings of the 2024 IEEE 13th International Conference on Cloud Networking (CloudNet), Rio de Janeiro, Brazil, 27–29 November 2024.
  • 61.
    Cai, W.; Chen, H.; Chen, H.; et al. Optimal Defense Strategy for Multi-agents Using Value Decomposition Networks. In Proceedings of the International Conference on Intelligent Computing, Tianjin, China, 5–8 August 2024.
  • 62.
    Zhang, M.; Tong, W.; Zhu, G.; et al. SQIX: QMIX algorithm activated by general softmax operator for cooperative multiagent reinforcement learning. IEEE Trans. Syst. Man Cybern. Syst. 2024, 54, 6550–6560.
  • 63.
    Jiang, C.; Lin, Z.; Liu, C.; et al. MADDPG-Based Active Distribution Network Dynamic Reconfiguration with Renewable Energy. Prot. Control Mod. Power Syst. 2024, 9, 143–155.
  • 64.
    Xing, X.; Zhou, Z.; Li, Y.; et al. Multi-UAV adaptive cooperative formation trajectory planning based on an improved MATD3 algorithm of deep reinforcement learning. IEEE Trans. Veh. Technol. 2024, 73, 12484–12499.
  • 65.
    Ivison, H.; Wang, Y.; Liu, J.; et al. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Adv. Neural Inf. Process. Syst. 2024, 37, 36602–36633.
  • 66.
    Tang, C.; Peng, T.; Xie, X.; et al. 3D path planning of unmanned ground vehicles based on improved DDQN. J. Supercomput. 2025, 81, 1–31.
  • 67.
    Cao, J.; Dong, L.; Yuan, X.; et al. Hierarchical multi-agent reinforcement learning for cooperative tasks with sparse rewards in continuous domain. Neural Comput. Appl. 2024, 36, 273–287.
  • 68.
    Zhao, R.; Liu, Y.; Wang, X. Graph limit and exponential consensus for the large-scale multi-agent system with delay. Commun. Nonlinear Sci. Numer. Simul. 2025, 149, 108919.
  • 69.
    Liu, R.; Piplani, R.; Toro, C. A deep multi-agent reinforcement learning approach to solve dynamic job shop scheduling problem. Comput. Oper. Res. 2023, 159, 106294.
  • 70.
    Velasquez, A.; Bissey, B.; Barak, L.; et al. Multi-agent tree search with dynamic reward shaping. In Proceedings of the International Conference on Automated Planning and Scheduling, Virtual, 13–24 June 2022; pp. 652–661.
  • 71.
    Liu, Z.; Yu, C.; Yang, Y.; et al. A unified diversity measure for multiagent reinforcement learning. Adv. Neural Inf. Process. Syst. 2022, 35, 10339–10352.
  • 72.
    Zhu, X.; Liu, J.; Zhang, T.; et al. CPPU: Policy Space Diversity for Informative Path Planning and GAI-enabled Updating CSI in ISAC. IEEE Trans. Cogn. Commun. Netw. 2024, 11, 777–790.
  • 73.
    He, W.; Tan, J.; Guo, Y.; et al. A deep reinforcement learning-based deception asset selection algorithm in differential games. IEEE Trans. Inf. Forensics Secur. 2024, 19, 8353–8368.
  • 74.
    Abouelazm, A.; Michel, J.; Zöllner, J.M. A review of reward functions for reinforcement learning in the context of autonomous driving. In Proceedings of the 2024 IEEE Intelligent Vehicles Symposium (IV), Jeju Island, Republic of Korea, 2–5 June 2024; pp. 156–163.
  • 75.
    Feng, Z.; Hu, G.Q.; Dong, X.W.; et al. Adaptively Distributed Nash Equilibrium Seeking of Noncooperative Games for Uncertain Heterogeneous Linear Multi-Agent Systems. IEEE Trans. Netw. Sci. Eng. 2023, 10, 3871–3882. https://doi.org/10.1109/Tnse.2023.3275326.
  • 76.
    Sunehag, P.; Lever, G.; Gruslys, A.; et al. Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. arXiv 2018, arXiv:1706.05296.
  • 77.
    Rashid, T.; Samvelyan, M.; de Witt, C.S.; et al. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv 2018, arXiv:1803.11485.
  • 78.
    Yang, R.Q.; Yu, L.; Li, Z.; et al. Rethinking Offline Reinforcement Learning for Sequential Recommendation from A Pair-Wise Q-Learning Perspective. In Proceedings of the 2024 International Joint Conference on Neural Networks (IJCNN), Yokohama, Japan, 30 June 2024–5 July 2024. https://doi.org/10.1109/Ijcnn60899.2024.10650400.
  • 79.
    Suh, J.; Tanaka, T. SARSA(0) Reinforcement Learning over Fully Homomorphic Encryption. In Proceedings of the 2021 SICE International Symposium on Control Systems (SICE ISCS), Tokyo, Japan, 2–4 March 2021.
  • 80.
    Michailidis, P.; Michailidis, I.; Kosmatopoulos, E. Review and Evaluation of Multi-Agent Control Applications for Energy Management in Buildings. Energies 2024, 17, 4835. https://doi.org/10.3390/en17194835.
  • 81.
    Olorunfemi, B.O.; Nwulu, N. Multi-agent system implementation in demand response: A literature review and bibliometric evaluation. Aims Energy 2023, 11, 1179–1210. https://doi.org/10.3934/energy.2023054.
  • 82.
    Saldaña, D.; Ovalle, D.; Montoya, A. A Multi-Agent Model to Control Robotic Sensor Networks. In Proceedings of the 2012 7th Colombian Computing Congress (CCC), Medellin, Colombia, 1–5 October 2012.
Share this article:
Download PDF
DOI
10.53941/ubs.2025.100004
Issue
Volume 1, Issue 1
How to Cite
Zhang, L.; Yuan, X.; Jia, W.; Elhokayem, A. A Conceptual Framework for Performance Evaluation and Classification of Multi-Agent Hybrid Game Problems Based on Reinforcement Learning: Impact Factors and Evaluation Metrics. Urban and Building Science 2025, 1 (1), 4. https://doi.org/10.53941/ubs.2025.100004.
RIS
BibTex
Copyright & License
article copyright Image
Copyright (c) 2025 by the authors.