Scilight Press

Author Information

Received: 29 Aug 2023 | Accepted: 11 Dec 2023 | Published: 19 Dec 2023

Abstract

Keywords

reinforcement learning | energy management system | multi-agent reinforcement learning | hybrid electric vehicles | connected hybrid electric vehicles

References 

• 1.
  Chen, G.; He, L.; Zhang, B.; et al. Dynamics integrated control for four-wheel independent control electric vehicle. International Journal of Heavy Vehicle Systems 2019, 26(3–4), 515–534.
• 2.
  Liu, W.; Xia, X.; Xiong, L.; et al. Automated vehicle sideslip angle estimation considering signal measurement characteristic. IEEE Sensors Journal 2021, 21(19), 21675–21687.
• 3.
  Zhao, S.; Chen, G.; Hua, M.; et al. An identification algorithm of driver steering characteristics based on backpropagation neural network. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 2019, 233(9), 2333–2342.
• 4.
  Zhou, Q.; Li, Y.; Zhao, D.; et al. Transferable representation modelling for real-time energy management of the plug-in hybrid vehicle based on k-fold fuzzy learning and Gaussian process regression. Applied Energy 2022, 305, 117853.
• 5.
  Chen, G.; Yao, J.; Gao, Z.; et al. Emergency Obstacle Avoidance Trajectory Planning Method of Intelligent Vehicles Based on Improved Hybrid A. SAE International Journal of Vehicle Dynamics, Stability, and NVH 2023, 8(10–08–01–0001).
• 6.
  Hua, M.; Chen, G.; Zong, C.; et al. Research on synchronous control strategy of steer-by-wire system with dual steering actuator motors. International Journal of Vehicle Autonomous Systems 2020, 15(1), 50–76.
• 7.
  Zhang, C.; Zhou, Q.; Shuai, B.; et al. Dedicated Adaptive Particle Swarm Optimization Algorithm for Digital Twin Based Control Optimization of the Plug-in Hybrid Vehicle. IEEE Transactions on Transportation Electrification 2022. DOI: 10.1109/TTE.2022.3219290.
• 8.
  Liu, W.; Hua, M.; Deng, Z.; et al. A systematic survey of control techniques and applications in connected and automated vehicles. IEEE Internet of Things Journal 2023. DOI: 10.1109/JIOT.2023.3307002.
• 9.
  Chen, G.; Hua, M.; Liu, W.; et al. Planning and tracking control of full drive-by-wire electric vehicles in unstructured scenario. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 2023, 09544070231195233.
• 10.
  Hua, M.; Zhou, Q.; Wang, C.; et al. Surrogate modelling for battery state-of-charge estimation in electric vehicles based on pseudo-2-dimensional model and gradient boosting machines. Energy Proceedings 2021. Available online: https://www.energy-proceedings.org/wp-content/uploads/icae2021/1643020588.pdf (Accessed on 29 August 2023).
• 11.
  Hua, M.; Shuai, B.; Zhou, Q.; et al. Recent Progress in Energy Management of Connected Hybrid Electric Vehicles Using Reinforcement Learning. arXiv Preprint 2023, arXiv: 2308.14602.
• 12.
  Chen, D.; Hajidavalloo, M.R.; Li, Z.; et al. Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic. IEEE Transactions on Intelligent Transportation Systems 2021, 24: 11623–11638.
• 13.
  Dai, C.; Zong, C.; Zhang, D.; et al. A Bargaining Game-Based Human–Machine Shared Driving Control Authority Allocation Strategy. IEEE Transactions on Intelligent Transportation Systems 2023. DOI: 10.1109/TITS.2023.3277015.
• 14.
  Hua, M.; Chen, G.; Zhang, B.; et al. A hierarchical energy efficiency optimization control strategy for distributed drive electric vehicles. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 2019, 233(3), 605–621.
• 15.
  Chen, G.; Hua, M.; Zong, C.; et al. Comprehensive chassis control strategy of FWIC‐EV based on sliding mode control. IET Intelligent Transport Systems 2019, 13(4), 703–713.
• 16.
  Mo, L.; Hua, M.; Sun, H.; et al. Study on the Impacts of Hazardous Behaviors on Autonomous Vehicle Collision Rates Based on Humanoid Scenario Generation in CARLA. arXiv preprint 2023, arXiv: 2307.10229.
• 17.
  Yan, F.; Wang, J.; Du, C.; et al. Multi-objective energy management strategy for hybrid electric vehicles based on TD3 with non-parametric reward function. Energies 2023, 16(1): 74.
• 18.
  Chen, D.; Chen, K.; Li, Z.; et al. Powernet: Multi-agent deep reinforcement learning for scalable powergrid control. IEEE Transactions on Power Systems 2021, 37(2): 1007–1017.
• 19.
  Hua, M.; Zhang, C, Zhang, F.; et al. Energy Management of Multi-mode Plug-in Hybrid Electric Vehicle using Multi-agent Deep Reinforcement Learning. arXiv preprint 2023, arXiv: 2303.09658.
• 20.
  Zhang, C.; Zhou, Q.; Hua, M.; et al. Cuboid equivalent consumption minimization strategy for energy management of multi-mode plug-in hybrid vehicles considering diverse time scale objectives. Applied Energy 2023, 351, 121901.
• 21.
  Hu, B.; Li, J. A deployment-efficient energy management strategy for connected hybrid electric vehicle based on offline reinforcement learning. IEEE Transactions on Industrial Electronics 2021, 69(9), 9644–9654.
• 22.
  Wang, J.; Du, C.; Yan, F.; et al. Hierarchical Rewarding Deep Deterministic Policy Gradient Strategy for Energy Management of Hybrid Electric Vehicles. IEEE Transactions on Transportation Electrification 2023. DOI: 10.1109/TTE.2023.3263927.
• 23.
  Ganesh, A.H.; Xu, B. A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution. Renewable and Sustainable Energy Reviews 2022, 154, 111833.
• 24.
  Zhang, K.; Yang, Z.; Başar, T. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control 2021, 321–384.
• 25.
  Pateria, S.; Subagdja, B.; Tan, A.H.; et al. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR) 2021, 54(5), 1–35.
• 26.
  Gronauer, S.; Diepold, K. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review 2022, 1–49.
• 27.
  Oroojlooy, A.; Hajinezhad, D. A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence 2023, 53(11), 13677–13722.
• 28.
  Nguyen, T.T.; Nguyen, N.D.; Nahavandi, S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics 2020, 50(9), 3826–3839.
• 29.
  Li, S.; Hu, W.; Cao, D.; et al. A Multiagent Deep Reinforcement Learning Based Approach for the Optimization of Transformer Life Using Coordinated Electric Vehicles. IEEE Transactions on Industrial Informatics 2022, 18(11), 7639–7652.
• 30.
  Zhou, Q.; Guo, S.; Xu, L.; et al. Global optimization of the hydraulic-electromagnetic energy-harvesting shock absorber for road vehicles with human-knowledge-integrated particle swarm optimization scheme. IEEE/ASME Transactions on Mechatronics 2021, 26(3), 1225–1235.
• 31.
  Zhou, Q.; Wang, C.; Sun, Z.; et al. Human-knowledge-augmented Gaussian process regression for state-of-health prediction of lithium-ion batteries with charging curves. Journal of Electrochemical Energy Conversion and Storage 2021, 18(3), 030907.
• 32.
  Shuai, B.; Li, Y, Zhou, Q.; et al. Supervisory control of the hybrid off-highway vehicle for fuel economy improvement using predictive double Q-learning with backup models. Journal of Central South University 2022, 29(7), 2266–2278.
• 33.
  Zhou, Q.; Zhao, D.; Shuai, B.; et al. Knowledge implementation and transfer with an adaptive learning network for real-time power management of the plug-in hybrid vehicle. IEEE Transactions on Neural Networks and Learning Systems 2021, 32(12), 5298–5308.
• 34.
  Shuai, B.; Zhou, Q.; Li, J.; et al. Heuristic action execution for energy efficient charge-sustaining control of connected hybrid vehicles with model-free double Q-learning. Applied Energy 2020, 267, 114900.
• 35.
  Zhou, Q.; Zhang, Y.; Li, Z.; et al. Cyber-physical energy-saving control for hybrid aircraft-towing tractor based on online swarm intelligent programming. IEEE Transactions on Industrial Informatics 2017, 14(9), 4149–4158.
• 36.
  Zhou, Q.; Li, J.; Shuai, B.; et al. Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle. Applied Energy 2019, 255, 113755.
• 37.
  Shuai, B.; Hua, M.; Li, Y.; et al. Optimal Energy Management of Plug-in Hybrid Vehicles Through Exploration-to-Exploitation Ratio Control in Ensemble Reinforcement Learning. arXiv Preprint 2023, arXiv: 2303.08981.
• 38.
  Xu, B.; Hou, J.; Shi, J.; et al. Learning time reduction using warm-start methods for a reinforcement learning-based supervisory control in hybrid electric vehicle applications. IEEE Transactions on Transportation Electrification 2020, 7(2), 626–635.
• 39.
  Liessner, R.; Schroer, C.; Dietermann, A.M.; et al. Deep reinforcement learning for advanced energy management of hybrid electric vehicles. In ICAART (2). 2018, pp. 61–72. Available online: https://www.scitepress.org/PublishedPapers/2018/65730/65730.pdf (Accessed on 29 August 2023).
• 40.
  Liu, T.; Hu, X.; Hu, W.; et al. A heuristic planning reinforcement learning-based energy management for power-split plug-in hybrid electric vehicles. IEEE Transactions on Industrial Informatics 2019, 15(12), 6436–6445.
• 41.
  Hu, Y.; Li, W.; Xu, K.; et al. Energy management strategy for a hybrid electric vehicle based on deep reinforcement learning. Applied Sciences 2018, 8(2), 187.
• 42.
  Zhou, W.; Chen, D, Yan J, et al. Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic. Autonomous Intelligent Systems 2022, 2(1), 5.
• 43.
  Song, C.; Lee, H.; Kim, K.; et al. A power management strategy for parallel PHEV using deep Q-networks. 2018 IEEE Vehicle Power and Propulsion Conference (VPPC). DOI: 10.1109/VPPC.2018.8604984.
• 44.
  Sun, H.; Fu, Z.; Tao, F.; et al. Data-driven reinforcement-learning-based hierarchical energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles. Journal of Power Sources 2020, 455, 227964.
• 45.
  Reddy, N.P.; Pasdeloup, D.; Zadeh, M.K.; et al. An intelligent power and energy management system for fuel cell/battery hybrid electric vehicle using reinforcement learning. IEEE Transportation Electrification Conference and Expo (ITEC) 2019. DOi: 10.1109/ITEC.2019.8790451.
• 46.
  Wu, J.; He, H.; Peng, J.; et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Applied energy 2018, 222, 799–811.
• 47.
  Han, X.; He, H.; Wu, J.; et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle. Applied Energy 2019, 254, 113708.
• 48.
  Liu, W.; Xiong, L.; Xia, X.; et al. Vision‐aided intelligent vehicle sideslip angle estimation based on a dynamic model. IET Intelligent Transport Systems 2020, 14(10), 1183–1189.
• 49.
  Meng, Z.; Xia, X.; Xu, R.; et al. HYDRO-3D: Hybrid Object Detection and Tracking for Cooperative Perception Using 3D LiDAR. IEEE Transactions on Intelligent Vehicles 2023. DOI: 10.1109/TIV.2023.3282567.
• 50.
  Wang, Z.; Wu, G.; Barth, M.J. Cooperative eco-driving at signalized intersections in a partially connected and automated vehicle environment. IEEE Transactions on Intelligent Transportation Systems 2019, 21(5), 2029–2038.
• 51.
  Guo, Q.; Angah, O.; Liu, Z.; et al. Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors. Transportation Research Part C: Emerging Technologies 2021, 124, 102980.
• 52.
  Liu, B.; Han, W.; Wang, E.; et al. An efficient message dissemination scheme for cooperative drivings via multi-agent hierarchical attention reinforcement learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS). IEEE: Piscataway, NJ, USA, 2021, pp. 326–336.
• 53.
  Zhao, W.; Ngoduy, D.; Shepherd, S.; et al. A platoon based cooperative eco-driving model for mixed automated and human-driven vehicles at a signalised intersection. Transportation Research Part C: Emerging Technologies 2018, 95, 802–821.
• 54.
  Tang, X.; Duan, Z.; Hu, X.; et al. Improving ride comfort and fuel economy of connected hybrid electric vehicles based on traffic signals and real road information. IEEE Transactions on Vehicular Technology 2021, 70(4), 3101–3112.
• 55.
  Wang, S.; Lin, X. Eco-driving control of connected and automated hybrid vehicles in mixed driving scenarios. Applied Energy 2020, 271, 115233.
• 56.
  Qiu, D.; Wang. Y.; Zhang, T.; et al. Hybrid multiagent reinforcement learning for electric vehicle resilience control towards a low-carbon transition. IEEE Transactions on Industrial Informatics 2022, 18(11), 8258–8269.
• 57.
  Foerster, J.; Assael, I.A.; De, Freitas.N.; et al. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems 2016, 29.
• 58.
  Li, S.; Wu, Y.; Cui. X.; et al. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. Proceedings of the AAAI conference on artificial intelligence 2019, 33(01), 4213–4220.
• 59.
  Lowe, R.; Wu. Y.I.; Tamar. A.; et al. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 2017, 30, 6379–6390.
• 60.
  Foerster, J.; Farquhar, G.; Afouras, T.; et al. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence 2018, 32(1).
• 61.
  Gupta, J.K.; Egorov, M.; Kochenderfer, M. Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16. Springer International Publishing, 2017, pp. 66–83.
• 62.
  Jiang, J.; Lu, Z. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems 2018, 31.