Distributed optimization plays an important role in multi-node collaborative computing and has developed rapidly in recent years, both in theory and applications. However, most classical distributed optimization algorithms are generally developed under idealized communication settings, which, to some extent, limit their applicability to practical systems with complex constraints. This survey aims to bridge the gap between theory and practice in distributed optimization by examining its theoretical foundations, recent progress, and major open challenges. Initially, we outline some theoretical breakthroughs regarding graph topologies, objective functions, and acceleration strategies. The discussion then turns to approaches for addressing practical constraints like stochastic noise, limited bandwidth, and asynchronous delays. Finally, we present several directions that are worth exploring in the future.
- Open Access
- Review
Comprehensive Survey of Distributed Optimization: Challenges, Strategies, and Applications
- Bingxue Luo
Author Information
Received: 18 June 2026 | Revised: 20 September 2026 | Accepted: 30 September 2026 | Published: 20 October 2026
Abstract
Keywords
multi-node system | distributed algorithms | stochastic optimization | acceleration methods | communication strategies | asynchronous update
References
- 1.
Liu, J.; Ma, E.; Zha, L.; et al. Distributed Energy Management for Microgrids: When Privacy Preservation Meets Multiple Mechanisms. IEEE Trans. Consum. Electron. 2026, 72, 1514–1523.
- 2.
Wang, J.; Hong, Y.; Wang, J.; et al. Cooperative and Competitive Multi-Agent Systems: From Optimization to Games. IEEE/CAA J. Autom. Sin. 2022, 9, 763–783.
- 3.
Gao, C.; Zhang, Q.; Varshney, P.K.; et al. A Distributed Multi-Objective Detection Method for Multi-Sensor Systems with Unknown Local SNR. IEEE Trans. Signal Process. 2025, 73, 649–663.
- 4.
Strinati, E.C.; Alexandropoulos, G.C.; Amani, N.; et al. Toward Distributed and Intelligent Integrated Sensing and Communications for 6G Networks. IEEE Wireless Commun. 2025, 32, 60–67.
- 5.
Lu, Q.; Li, H.; Wu, C.; et al. Decentralized Constrained Optimization Over Time-Varying Directed Networks via Subgradient Rescaling. IEEE Trans. Cybern. 2026, 56, 3725–3738.
- 6.
Nedic, A.; Ozdaglar, A. Distributed Subgradient Methods for Multi-Agent Optimization. IEEE Trans. Autom. Control 2009, 54, 48–61.
- 7.
Shi, W.; Ling, Q.; Wu, G.; et al. Extra: An Exact First-Order Algorithm for Decentralized Consensus Optimization. SIAM J. Optim. 2015, 25, 944–966.
- 8.
Ghaderyan, D.; Aybat, N.S.; Aguiar, A.P. A Fast Row-Stochastic Decentralized Method for Distributed Optimization Over Directed Graphs. IEEE Trans. Autom. Control 2024, 69, 275–289.
- 9.
Zheng, Y.; Cenedese, C.; Cucuzzella, M.; et al. A Specified-Time Distributed Optimization Algorithm for Multi-Agent Systems With Time-Varying Constraints. IEEE Trans. Autom. Control 2025, 71, 1999–2006.
- 10.
Ou, M.; Zhang, H.; Yan, Z.; et al. A Novel Distributed Gradient Algorithm for Composite Constrained Optimization Over Directed Network. Sci. Rep. 2026, 16, 5185.
- 11.
Li, J.; Li, C.; Fan, J.; et al. Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization with Compressed Communication. IEEE Trans. Autom. Control 2023, 69, 936–951.
- 12.
Rahili, S.; Ren, W. Distributed Continuous-Time Convex Optimization with Time-Varying Cost Functions. IEEE Trans. Autom. Control 2016, 62, 1590–1605.
- 13.
Liu, T.; Bin, M.; Notarnicola, I.; et al. Robust Distributed Optimization for Multi-Agent Systems with General Linear Models. IEEE Trans. Autom. Control 2026, 71, 6685–6697.
- 14.
Zhou, X.; Ma, Z.; Zou, S.; et al. Distributed Momentum-Based Multiagent Optimization with Different Constraint Sets. IEEE Trans. Autom. Control 2024, 70, 963–978.
- 15.
Liao, F.Y.; Madden, T.; Zheng, Y. An Accelerated Proximal Bundle Method for Convex Optimization. arXiv 2025, arXiv:2512.04523.
- 16.
Gao, R.; Kleywegt, A. Distributionally Robust Stochastic Optimization with Wasserstein Distance. Math. Oper. Res. 2023, 48, 603–655.
- 17.
Horvath, S.; Kovalev, D.; Mishchenko, K.; et al. Stochastic Distributed Learning with Gradient Quantization and Double-Variance Reduction. Optim. Methods Softw. 2023, 38, 91–106.
- 18.
Wu, X.; Liu, C.; Magnusson, S.; et al. Asynchronous Distributed Optimization with Delay-Free Parameters. IEEE Trans. Autom. Control 2025, 71, 259–274.
- 19.
Zheng, Y.; Liu, Q. A Review of Distributed Optimization: Problems, Models and Algorithms. Neurocomputing 2022, 483, 446–459.
- 20.
Yang, T.; Yi, X.; Wu, J.; et al. A Survey of Distributed Optimization. Annu. Rev. Control 2019, 47, 278–305.
- 21.
Li, X.; Xie, L.; Li, N. A Survey on Distributed Online Optimization and Online Games. Annu. Rev. Control 2023, 56, 100904.
- 22.
AbdelRaouf, H.; Hasnaine, Q.R.; Fouda, M.M.; et al. Paradigm Shift toward Distributed Learning in IoT Intelligence: A Comprehensive Survey of Opportunities and Challenges. IEEE Internet Things J. 2026, 13, 20128–20154.
- 23.
Vandenberghe, L.; Boyd, S. Convex Optimization, 1st ed.; Cambridge University Press: Cambridge, UK, 2004.
- 24.
Sawamura, R.; Hayashi, N.; Inuiguchi, M. A Distributed Primal–Dual Push-Sum Algorithm on Open Multiagent Networks. IEEE Trans. Autom. Control 2024, 70, 1192–1199.
- 25.
Xi, C.; Khan, U.A. DEXTRA: A Fast Algorithm for Optimization Over Directed Graphs. IEEE Trans. Autom. Control 2017, 62, 4980–4993.
- 26.
Xi, C.; Xin, R.; Khan, U.A. ADD-OPT: Accelerated Distributed Directed Optimization. IEEE Trans. Autom. Control 2017, 63, 1329–1339.
- 27.
Nedic, A.; Olshevsky, A.; Shi, W. Achieving Geometric Convergence for Distributed Optimization Over Time-Varying Graphs. SIAM J. Optim. 2017, 27, 2597–2633.
- 28.
Xin, R.; Xi, C.; Khan, U.A. FROST—Fast Row-Stochastic Optimization with Uncoordinated Step-Sizes. EURASIP J. Adv. Signal Process. 2019, 2019, 1–14.
- 29.
Pu, S.; Shi, W.; Xu, J.; et al. Push–Pull Gradient Methods for Distributed Optimization in Networks. IEEE Trans. Autom. Control 2020, 66, 1–16.
- 30.
Xin, R.; Khan, U.A. A Linear Algorithm for Optimization Over Directed Graphs with Geometric Convergence. IEEE Control Syst. Lett. 2018, 2, 315–320.
- 31.
Saadatniaki, F.; Xin, R.; Khan, U.A. Decentralized Optimization Over Time-Varying Directed Graphs with Row and Column-Stochastic Matrices. IEEE Trans. Autom. Control 2020, 65, 4769–4780.
- 32.
Nedic, A.; Nguyen, D.T.A.; Nguyen, D.T. AB/Push-Pull Method for Distributed Optimization in Time-Varying Directed Networks. Optim. Methods Softw. 2025, 40, 1044–1071.
- 33.
Duchi, J.C.; Agarwal, A.; Wainwright, M.J. Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling. IEEE Trans. Autom. Control 2011, 57, 592–606.
- 34.
Shi, W.; Ling, Q.; Yuan, K.; et al. On the Linear Convergence of the ADMM in Decentralized Consensus Optimization. IEEE Trans. Signal Process. 2014, 62, 1750–1761.
- 35.
Aybat, N.S.; Wang, Z.; Lin, T.; et al. Distributed Linearized Alternating Direction Method of Multipliers for Composite Convex Consensus Optimization. IEEE Trans. Autom. Control 2017, 63, 5–20.
- 36.
Chang, T.H.; Hong, M.; Wang, X. Multi-Agent Distributed Optimization via Inexact Consensus ADMM. IEEE Trans. Signal Process. 2014, 63, 482–497.
- 37.
Shi, W.; Ling, Q.; Wu, G.; et al. A Proximal Gradient Algorithm for Decentralized Composite Optimization. IEEE Trans. Signal Process. 2015, 63, 6013–6023.
- 38.
Di Lorenzo, P.; Scutari, G. NEXT: In-Network Nonconvex Optimization. IEEE Trans. Signal Inf. Process. Netw. 2016, 2, 120–136.
- 39.
Zeng, J.; Yin, W. On Nonconvex Decentralized Gradient Descent. IEEE Trans. Signal Process. 2018, 66, 2834–2848.
- 40.
Zeng, J.; He, T.; Wang, M. A Fast Proximal Gradient Algorithm for Decentralized Composite Optimization Over Directed Networks. Syst. Control Lett. 2017, 107, 36–43.
- 41.
Sun, Y.; Scutari, G.; Palomar, D. Distributed Nonconvex Multiagent Optimization Over Time-Varying Networks. In Proceedings of the 50th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, 6–9 November 2016.
- 42.
Xin, R.; Das, S., Khan, U. A., et al. A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency. arXiv 2021, arXiv:2110.01594.
- 43.
Xing, M.; Ma, D.; Ma, K.; et al. Implicit Enhanced Distributed Heavy-Ball Energy Management Strategy for Microgrids with Time-Varying Social Welfare and Delay. IEEE Trans. Smart Grid 2025, 16, 2492–2503.
- 44.
Xue, S.; Tan, J.; Cao, H.; et al. Nesterov Accelerated Gradient-Based Fixed-Time Convergent Actor-Critic Control for Nonlinear Systems. IEEE Trans. Autom. Sci. Eng. 2026, 23, 5075–5083.
- 45.
Qu, G.; Li, N. Accelerated Distributed Nesterov Gradient Descent. IEEE Trans. Autom. Control 2019, 65, 2566–2581.
- 46.
Yang, Z.; He, W.; Aguiar, A.P. An Accelerated Distributed Proportional-Integral Method for Distributed Constrained Optimization. IEEE Control Syst. Lett. 2025, 9, 2241–2246.
- 47.
Xin, R.; Khan, U.A. Distributed Heavy-Ball: A Generalization and Acceleration of First-Order Methods with Gradient Tracking. IEEE Trans. Autom. Control 2019, 65, 2627–2633.
- 48.
Xin, R.; Jakovetic, D.; Khan, U.A. Distributed Nesterov Gradient Methods Over Arbitrary Graphs. IEEE Signal Process. Lett. 2019, 26, 1247–1251.
- 49.
Li, H.; Cheng, H.; Wang, Z.; et al. Distributed Nesterov Gradient and Heavy-Ball Double Accelerated Asynchronous Optimization. IEEE Trans. Neural Netw. Learn. Syst. 2020, 32, 5723–5737.
- 50.
Luo, B.; Lu, Q.; Cheng, H.; et al. A Distributed Stochastic Unified Accelerated Algorithm for Convex Optimization Over Time-Varying Directed Networks. IEEE Trans. Syst. Man Cybern. Syst. 2026, 56, 4467–4480.
- 51.
Pu, S.; Nedic, A. Distributed Stochastic Gradient Tracking Methods. Math. Program. 2021, 187, 409–457.
- 52.
Gao, J.; Liu, X.W.; Dai, Y.H.; et al. Distributed Stochastic Gradient Tracking Methods with Momentum Acceleration for Non-Convex Optimization. Comput. Optim. Appl. 2023, 84, 531–572.
- 53.
Nguyen, D.T.A.; Nguyen, D.T.; Nedic, A. Accelerated AB/Push–Pull Methods for Distributed Optimization Over Time-Varying Directed Networks. IEEE Trans. Control Netw. Syst. 2023, 11, 1395–1407.
- 54.
Mohammadi, H.; Razaviyayn, M.; Jovanovic, M.R. Tradeoffs Between Convergence Rate and Noise Amplification for Momentum-Based Accelerated Optimization Algorithms. IEEE Trans. Autom. Control 2024, 70, 889–904.
- 55.
Nazari, P.; Tarzanagh, D.A.; Michailidis, G. Dadam: A Consensus-Based Distributed Adaptive Gradient Method for Online Optimization. IEEE Trans. Signal Process. 2022, 70, 6065–6079.
- 56.
Carnevale, G.; Farina, F.; Notarnicola, I.; et al. GTAdam: Gradient Tracking with Adaptive Momentum for Distributed Online Optimization. IEEE Trans. Control Netw. Syst. 2022, 10, 1436–1448.
- 57.
Wang, Z.; Ling, Q.; Yin, W. Decentralized Bundle Method for Nonsmooth Consensus Optimization. In Proceedings of the 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Montreal, QC, Canada, 14–16 November 2017.
- 58.
Zhu, Z.; Tian, Y.P.; Wu, X. Historical Information Accelerates Decentralized Optimization: A Proximal Bundle Method. arXiv 2025, arXiv:2512.15189.
- 59.
Liu, L.; Huang, Y.; Deng, C. Accelerated Distributed Cooperative Energy Management for Integrated Energy Systems. IEEE/CAA J. Autom. Sin. 2025, 12, 2589–2601.
- 60.
Wang, X.; Wang, G.; Li, S. A Distributed Fixed-Time Optimization Algorithm for Multi-Agent Systems. Automatica 2020, 122, 109289.
- 61.
Firouzbahrami, M.; Nobakhti, A. Cooperative Fixed-Time/Finite-Time Distributed Robust Optimization of Multi-Agent Systems. Automatica 2022, 142, 110358.
- 62.
Chen, S.; Jiang, H.; Yu, Z. A Distributed Prescribed-Time Optimization Analysis for Multi-Agent Systems. Inf. Sci. 2022, 607, 346–360.
- 63.
Li, H.; Zhang, M.; Yin, Z.; et al. Prescribed-Time Distributed Optimization Problem with Constraints. ISA Transactions 2024, 148, 255–263.
- 64.
Pu, S.; Olshevsky, A.; Paschalidis, I.C. A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent. IEEE Trans. Autom. Control 2021, 67, 5900–5915.
- 65.
Xin, R.; Khan, U.A.; Kar, S. An Improved Convergence Analysis for Decentralized Online Stochastic Non-Convex Optimization. IEEE Trans. Signal Process. 2021, 69, 1842–1858.
- 66.
Chen, Y.; Hashemi, A.; Vikalo, H. Accelerated Distributed Stochastic Nonconvex Optimization Over Time-Varying Directed Networks. IEEE Trans. Autom. Control 2024, 70, 2196–2211.
- 67.
Defazio, A.; Bach, F.; Lacoste-Julien, S. SAGA: A Fast Incremental Gradient Method with Support for Non-Strongly Convex Composite Objectives. Adv. Neural Inf. Process. Syst. 2014, 27, 1646–1654.
- 68.
Johnson, R.; Zhang, T. Accelerating Stochastic Gradient Descent Using Predictive Variance Reduction. Adv. Neural Inf. Process. Syst. 2013, 26, 315–323.
- 69.
Xin, R.; Khan, U.A.; Kar, S. Variance-Reduced Decentralized Stochastic Optimization with Accelerated Convergence. IEEE Trans. Signal Process. 2020, 68, 6255–6271.
- 70.
Xin, R.; Khan, U.A.; Kar, S. A Fast Randomized Incremental Gradient Method for Decentralized Nonconvex Optimization. IEEE Trans. Autom. Control 2021, 67, 5150–5165.
- 71.
Niu, Y.; Li, H.; Wang, Z.; et al. A Distributed Stochastic Proximal-Gradient Algorithm for Composite Optimization. IEEE Trans. Control Netw. Syst. 2021, 8, 1383–1393.
- 72.
Craciunas, S.S.; Oliver, R.S. Combined Task-and Network-Level Scheduling for Distributed Time-Triggered Systems. Real-Time Syst. 2016, 52, 161–200.
- 73.
Zhang, X.; Zhao, X. Architecture Design of Distributed Redundant Flight Control Computer Based on Time-Triggered Buses for UAVs. IEEE Sensors J. 2020, 21, 3944–3954.
- 74.
Hayashi, N.; Sugiura, T.; Kajiyama, Y.; et al. Event-Triggered Consensus-Based Optimization Algorithm for Smooth and Strongly Convex Cost Functions. In Proceedings of the 2018 IEEE Conference on Decision and Control (CDC), Miami Beach, FL, USA, 17–19 December 2018.
- 75.
Hu, C.; Zhang, X.; Wu, Q. Collaborative Active and Reactive Power Control of DERs for Voltage Regulation and Frequency Support by Distributed Event-Triggered Heavy Ball Method. IEEE Trans. Smart Grid 2023, 14, 3804–3815.
- 76.
Xu, L.; Yi, X.; Shi, Y.; et al. Distributed Nonconvex Optimization with Event-Triggered Communication. IEEE Trans. Autom. Control 2023, 69, 2745–2752.
- 77.
Liu, L.; Yang, G.H. Distributed Dynamic Event-Triggered Optimal Economic Dispatch for Multienergy Systems with Prescribed-Time Convergence. IEEE Trans. Ind. Inform. 2025, 21, 4533–4542.
- 78.
Zhang, J.; You, K.; Basar, T. Distributed Discrete-Time Optimization in Multiagent Networks Using Only Sign of Relative State. IEEE Trans. Autom. Control 2018, 64, 2352–2367.
- 79.
Aji, A.F.; Heafield, K. Sparse Communication for Distributed Gradient Descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark, 7–11 September 2017.
- 80.
Alistarh, D.; Grubic, D.; Li, J.; et al. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA, 4–9 December 2017.
- 81.
Koloskova, A.; Stich, S.; Jaggi, M. Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication. In Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, CA, USA, 9–15 June 2019.
- 82.
Lu, Q.; Guan, M.; He, C.; et al. A Quantised Push-Sum Distributed Adaptive Momentum Algorithm for Optimisation Over Directed Networks. CAAI Trans. Intell. Technol. 2026, 11, 1284–1301.
- 83.
Yi, X.; Zhang, S.; Yang, T.; et al. Communication Compression for Distributed Nonconvex Optimization. IEEE Trans. Autom. Control 2022, 68, 5477–5492.
- 84.
Liu, W.; Panda, A.; Pandey, U.; et al. Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization. arXiv 2025, arXiv:2508.04950.
- 85.
Li, H.; Lu, Q.; Chen, G.; et al. Distributed Constrained Optimization Over Unbalanced Directed Networks Using Asynchronous Broadcast-Based Algorithm. IEEE Trans. Autom. Control 2020, 66, 1102–1115.
- 86.
Iutzeler, F.; Bianchi, P.; Ciblat, P.; et al. Asynchronous Distributed Optimization Using a Randomized Alternating Direction Method of Multipliers. In Proceedings of the 52nd IEEE Conference on Decision and Control (CDC), Firenze, Italy, 10–13 December 2013.
- 87.
Wu, X.; Liu, C.; Magnusson, S.; et al. Delay-Agnostic Asynchronous Distributed Optimization. In Proceedings of the 62nd IEEE Conference on Decision and Control (CDC), Singapore, 13–15 December 2023.
- 88.
Tian, Y.; Sun, Y.; Scutari, G. Achieving Linear Convergence in Distributed Asynchronous Multiagent Optimization. IEEE Trans. Autom. Control 2020, 65, 5264–5279.
- 89.
Nie, R.; Du, W.; Wang, T.; et al. Distributed Asynchronous Optimization of Multiagent Systems: Convergence Analysis and Its Application. IEEE Trans. Ind. Inform. 2024, 20, 8983–8992.
- 90.
Su, E.; Li, H.; Zheng, Z.; et al. Distributed Constrained Optimization with Asynchrony and Delays. IEEE Trans. Circuits Syst. II Express Briefs 2021, 69, 2772–2776.
- 91.
Ran, L.; Li, H.; Zheng, L.; et al. Asynchronous Distributed Nonsmooth Composite Optimization via Computation-Efficient Primal-Dual Proximal Algorithms. IEEE Trans. Emerg. Top. Comput. Intell. 2024, 9, 1595–1609.
- 92.
Kungurtsev, V.; Morafah, M.; Javidi, T.; et al. Decentralized Asynchronous Nonconvex Stochastic Optimization on Directed Graphs. IEEE Trans. Control Netw. Syst. 2023, 10, 1796–1804.
This work is licensed under a Creative Commons Attribution 4.0 International License.
