JMLIS/
Articles/
2607004561
  • Open Access
  • Review

A Survey of Second-Order Neurodynamic Systems from Optimization Perspective

  • Jiali Wei 1,*,   
  • Guo-Cheng Wu 2,   
  • Jingxin Liu 3

Received: 06 May 2026 | Revised: 03 Jul 2026 | Accepted: 07 Jul 2026 | Published: 22 Jul 2026

Abstract

Over the past two decades, second-order neurodynamic systems have attracted increasing attention due to their strong capability in modeling and solving complex optimization problems. Unlike classical first-order approaches, these continuous neurodynamics exhibit inertial effects, which essentially provide a mechanism for accelerated convergence. We provide an overview of the development of secondorder neurodynamic systems from an optimization perspective. We first introduced the basic theoretical framework of second-order neurodynamic systems, including the continuous-time formulations and their connection to optimization models. The review is divided into two main threads: unconstrained and constrained optimization. Based on this categorization, the representative models and solution strategies are analyzed in detail, covering smooth and nonsmooth, as well as convex and nonconvex optimization problems. Finally, we highlight the limitations of the current study and outline some directions for future research.

References 

  • 1.

    Tseng, P. Approximation Accuracy, Gradient Methods, and Error Bound for Structured Convex Optimization. Math. Program. 2010, 125, 263–295.

  • 2.

    Castera, C.; Bolte, J.; F´evotte, C.; et al. An Inertial Newton Algorithm for Deep Learning. J. Mach. Learn. Res. 2021, 22, 1–31. 

  • 3.

    Parikh, N.; Boyd, S. Proximal Algorithms. Found. Trends Optim. 2014, 1, 127–239.

  • 4.

    Li, H.; Liu, S.; Soh, Y.C.; et al. Event-Triggered Communication and Data Rate Constraint for Distributed Optimization of Multiagent Systems. IEEE Trans. Syst. Man Cybern. Syst. 2018, 48, 1908–1919.

  • 5.

    Wei, J.; Liao, X.; Yang, J. Hessian-Riemannian Second-Order Neurodynamic Approach for Constrained Optimization: Theory and Applications. Neural Netw. 2026, 195, 108251.

  • 6.

    Liao, X.R.; Wei, X.K.; Zhou, M.L. Minimax Concave Penalty Regression for Superresolution Image Reconstruction. IEEE Trans. Consum. Electron. 2024, 70, 2999–3007.

  • 7.

    Wei, J.; Liao, X. Dynamical Threshold-Based Fractional Anisotropic Diffusion for Speckle Noise Removal. IEEE Trans. Image Process. 2025, 34, 2826–2839.

  • 8.

    Pourmohammad Azizi, S.; Neisy, A.; Ahmad Waloo, S. A Dynamical Systems Approach to Machine Learning. Int. J. Comput. Methods 2023, 20, 2350007.

  • 9.

    Levine, M.; Stuart, A. A Framework for Machine Learning of Model Error in Dynamical Systems. Commun. Amer. Math. Soc. 2022, 2, 283–344.

  • 10.

    Li, H.; Lu, Q.; Chen, G.; et al. Distributed Constrained Optimization Over Unbalanced Directed Networks Using Asynchronous Broadcast-Based Algorithm. IEEE Trans. Autom. Control 2021, 66, 1102–1115.

  • 11.

    Yang, S.; Zhang, L.; Wang, J.; et al. Fixed-Time Consensus of Nonlinear Multi-Agent Systems with Uncertain Disturbances via Saturation Constraint Impulsive Control. J. Mach. Learn. Inf. Sec. 2026, 2, 7.

  • 12.

    Zhu, Y.; Li, C.; Zeng, R. A Survey of Learning in Optimal Control and Differential Game. J. Mach. Learn. Inf. Sec. 2026, 2, 4.

  • 13.

    Hopfield, J.J. Neural Networks and Physical Systems with Emergent Collective Computational Abilities. Proc. Natl. Acad. Sci. USA 1982, 79, 2554–2558.

  • 14.

    Polyak, B.T. Some Methods of Speeding Up the Convergence of Iteration Methods. USSR Comput. Math. Math. Phys. 1964, 4, 1–17.

  • 15.

    Nesterov, Y. A Method for Solving a Convex Programming Problem with Convergence Rate O(1/k2). Proc. USSR Acad. Sci. 1983, 269, 543–547.

  • 16.

    Su, W.; Boyd, S.; Cand`es, E.J. A Differential Equation for Modeling Nesterov’s Accelerated Gradient Method: Theory and Insights. J. Mach. Learn. Res. 2016, 17, 1–43.

  • 17.

    Boyd, S.; Vandenberghe, L. Convex Optimization; Cambridge University Press: Cambridge, UK, 2004.

  • 18.

    Bertsekas, D.P. Nonlinear Programming; Athena Scientific: Nashua, NH, USA, 1999.

  • 19.

    Beck, A.; Teboulle, M. A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems. SIAM J. Imaging Sci. 2009, 2, 183–202.

  • 20.

    Cai, Z.; De Queiroz, M.S.; Dawson, D.M. A Sufficiently Smooth Projection Operator. IEEE Trans. Autom. Control 2006, 51, 135–139.

  • 21.

    Feijer, D.; Paganini, F. Stability of Primal-Dual Gradient Dynamics and Applications to Network Optimization. Automatica 2010, 46, 1974–1981.

  • 22.

    Attouch, H.; Chbani, Z.; Riahi, H. Rate of Convergence of the Nesterov Accelerated Gradient Method in the Subcritical Case α ≤ 3. ESAIM Control. Optim. Calc. Var. 2019, 25, 2.

  • 23.

    Attouch, H.; Maing´e, P.E. Asymptotic Behavior of Second-Order Dissipative Evolution Equations Combining Potential with Non-Potential Effects. ESAIM Control Optim. Calc. Var. 2011, 17, 836–857.

  • 24.

    Alvarez, F. On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces. SIAM J. Control Optim. 2000, 38, 1102–1119.

  • 25.

    Begout, P.; Bolte, J.; Jendoubi, M.A. On Damped Second-Order Gradient Systems. J. Differ. Equ. 2015, 259, 3115–3143.

  • 26.

    Attouch, H.; Cabot, A. Asymptotic Stabilization of Inertial Gradient Dynamics with Time-Dependent Viscosity. J. Differ. Equ. 2017, 263, 5412–5458.

  • 27.

    Attouch, H.; Cabot, A.; Chbani, Z.; et al. Rate of Convergence of Inertial Gradient Dynamics with Time-Dependent Viscous Damping Coefficient. Evol. Equ. Control Theory 2018, 7, 353–371.

  • 28.

    Wibisono, A.; Wilson, A.C.; Jordan, M.I. A Variational Perspective on Accelerated Methods in Optimization. Proc. Natl. Acad. Sci. USA 2016, 113, E7351–E7358.

  • 29.

    Wilson, A.C.; Recht, B.; Jordan, M.I. A Lyapunov Analysis of Accelerated Methods in Optimization. J. Mach. Learn. Res. 2021, 22, 1–34.

  • 30.

    Attouch, H.; Chbani, Z.; Riahi, H. Fast Proximal Methods via Time Scaling of Damped Inertial Dynamics. SIAM J. Optim. 2019, 29, 2227–2256.

  • 31.

    Attouch, H.; Chbani, Z.; Riahi, H. Fast Convex Optimization via Time Scaling of Damped Inertial Gradient Dynamics. Pure Appl. Funct. Anal. 2021, 6, 1081–1117.

  • 32.

    Balhag, A.; Chbani, Z.; Riahi, H. Linear Convergence of Inertial Gradient Dynamics with Constant Viscous Damping Coefficient and Time-Dependent Rescaling Parameter. 2020. Available online: https://hal.science/hal-02610699v1/document (accessed on 30 April 2026).

  • 33.

    Aujol, J.F.; Dossal, C.; Rondepierre, A. Optimal Convergence Rates for Nesterov Acceleration. SIAM J. Optim. 2019, 29, 3131–3153.

  • 34.

    Shi, B.; Du, S.S.; Jordan, M.I.; et al. Understanding the Acceleration Phenomenon via High-Resolution Differential Equations. Math. Program. 2022, 195, 79–148.

  • 35.

    Luo, H.; Chen, L. From Differential Equation Solvers to Accelerated First-Order Methods for Convex Optimization. Math. Program. 2022, 195, 735–781.

  • 36.

    Alvarez, F.; Attouch, H.; Bolte, J.; et al. A Second-Order Gradient-Like Dissipative Dynamical System with Hessian-Driven Damping: Application to Optimization and Mechanics. J. Math. Pures Appl. 2002, 81, 747–779.

  • 37.

    Attouch, H.; Maing´e, P.E.; Redont, P. A Second-Order Differential System with Hessian-Driven Damping; Application to Non-Elastic Shock Laws. Differ. Equ. Appl. 2012, 4, 27–65.

  • 38.

    Attouch, H.; Peypouquet, J.; Redont, P. Fast Convex Optimization via Inertial Dynamics with Hessian Driven Damping. J. Differ. Equ. 2016, 261, 5734–5783.

  • 39.

    Adly, S.; Attouch, H. Finite Time Stabilization of Inertial Dynamics Combining Dry Friction with Hessian-Driven Damping. J. Convex Anal. 2021, 28, 281–310.

  • 40.

    Adly, S.; Attouch, H. Finite Convergence of Proximal-Gradient Inertial Algorithms Combining Dry Friction with Hessian-Driven Damping. SIAM J. Optim. 2020, 30, 2134–2162.

  • 41.

    Attouch, H.; Chbani, Z.; Fadili, J.; et al. First-Order Optimization Algorithms via Inertial Systems with Hessian Driven Damping. Math. Program. 2022, 193, 113–155.

  • 42.

    Attouch, H.; Balhag, A.; Chbani, Z.; et al. Fast Convex Optimization via Inertial Dynamics Combining Viscous and Hessian-Driven Damping with Time Rescaling. Evol. Equ. Control Theory 2022, 11, 487–514.

  • 43.

    Aujol, J.F.; Dossal, C.; Ho`ang, V.H.; et al. Fast Convergence of Inertial Dynamics with Hessian-Driven Damping Under Geometry Assumptions. Appl. Math. Optim. 2023, 88, 81.

  • 44.

    Attouch, H.; Fadili, J.M.; Kungurtsev, V. On the Effect of Perturbations in First-Order Optimization Methods with Inertia and Hessian Driven Damping. Evol. Equ. Control Theory 2023, 12, 71–117.

  • 45.

    Maulen-Soto, R.; Fadili, J.; Ochs, P. Inertial Methods with Viscous and Hessian Driven Damping for Non-Convex Optimization. arXiv 2024, arXiv:2407.12518.

  • 46.

    Nguyen, D.K. On a Class of Second-Order Dynamical Systems Involving Viscous and Hessian-Driven Damping. Evol. Equ. Control Theory 2025, 17, 231–258.

  • 47.

    Apidopoulos, V.; Mavrogeorgou, V.; Tsironis, T.G. Heavy-Ball Dynamics with Hessian-Driven Damping for Non-Convex Optimization Under the Łojasiewicz Condition. arXiv 2025, arXiv:2506.11705.

  • 48.

    Alecsa, C.D.; Laszlo, S.C.; Pint, a, T. An Extension of the Second Order Dynamical System that Models Nesterov’s Convex Gradient Method. Appl. Math. Optim. 2021, 84, 1687–1716.

  • 49.

    Adly, S.; Mitchozounnou, D.K.; Prot, O. From Inertial Systems to Proximal Flows: The Large Damping Limit of Implicit Hessian-Driven Dynamics. Evol. Equ. Control Theory 2025, 17, 201–230.

  • 50.

    Gao, Z.; Sun, X.; He, L. Inertial Dynamical Systems and Accelerated Algorithms with Implicit Hessian-Driven Damping for Nonconvex Optimization. J. Optim. Theory Appl. 2026, 210, 21.

  • 51.

    Cominetti, R.; Peypouquet, J.; Sorin, S. Strong Asymptotic Convergence of Evolution Equations Governed by Maximal Monotone Operators with Tikhonov Regularization. J. Differ. Equ. 2008, 245, 3753–3763.

  • 52.

    Attouch, H.; Chbani, Z.; Riahi, H. Combining Fast Inertial Dynamics for Convex Optimization with Tikhonov Regularization. J. Math. Anal. Appl. 2018, 457, 1065–1094.

  • 53.

    Xu, B.; Wen, B. On the Convergence of a Class of Inertial Dynamical Systems with Tikhonov Regularization. Optim. Lett. 2021, 15, 2025–2052.

  • 54.

    Bot, R.I.; Csetnek, E.R.; Laszlo, S.C. Tikhonov Regularization of a Second Order Dynamical System with Hessian Driven Damping. Math. Program. 2021, 189, 151–186.

  • 55.

    Alecsa, C.D.; Laszlo, S.C. Tikhonov Regularization of a Perturbed Heavy Ball System with Vanishing Damping. SIAM J. Optim. 2021, 31, 2921–2954.

  • 56.

    Attouch, H.; Balhag, A.; Chbani, Z.; et al. Damped Inertial Dynamics with Vanishing Tikhonov Regularization: Strong Asymptotic Convergence Towards the Minimum Norm Solution. J. Differ. Equ. 2022, 311, 29–58.

  • 57.

    Attouch, H.; Balhag, A.; Chbani, Z.; et al. Accelerated Gradient Methods Combining Tikhonov Regularization with Geometric Damping Driven by the Hessian. Appl. Math. Optim. 2023, 88, 29.

  • 58.

    Laszlo, S.C. On the Strong Convergence of the Trajectories of a Tikhonov Regularized Second Order Dynamical System with Asymptotically Vanishing Damping. J. Differ. Equ. 2023, 362, 355–381.

  • 59.

    Attouch, H.; L´aszl´o, S.C. Convex Optimization via Inertial Algorithms with Vanishing Tikhonov Regularization: Fast Convergence to the Minimum Norm Solution. Math. Methods Oper. Res. 2024, 99, 307–347.

  • 60.

    Bot, R.I.; Csetnek, E.R.; Laszlo, S.C. On the Strong Convergence of Continuous Newton-Like Inertial Dynamics with Tikhonov Regularization for Monotone Inclusions. J. Math. Anal. Appl. 2024, 530, 127689.

  • 61.

    Laszlo, S.C. Solving Convex Optimization Problems via a Second Order Dynamical System with Implicit Hessian Damping and Tikhonov Regularization. Comput. Optim. Appl. 2025, 90, 113–149.

  • 62.

    Laszlo, S.C. On the Convergence of an Inertial Proximal Algorithm with a Tikhonov Regularization Term. Commun. Nonlinear Sci. Numer. Simul. 2025, 149, 108924.

  • 63.

    Sun, X.; Tian, G.; Zhang, H. Tikhonov Regularized Second-Order Dynamics with Hessian-Driven Damping for Solving Convex Optimization Problems. arXiv 2025, arXiv:2506.15968.

  • 64.

    Attouch, H.; Chbani, Z.; Fadili, J.; et al. Fast Convergence of Dynamical ADMM via Time Scaling of Damped Inertial Dynamics. J. Optim. Theory Appl. 2022, 193, 704–736.

  • 65.

    He, X.; Hu, R.; Fang, Y.P. Convergence Rates of Inertial Primal-Dual Dynamical Methods for Separable Convex Optimization Problems. SIAM J. Control Optim. 2021, 59, 3278–3301.

  • 66.

    Bot, R.I.; Nguyen, D.K. Improved Convergence Rates and Trajectory Convergence for Primal-Dual Dynamical Systems with Vanishing Damping. J. Differ. Equ. 2021, 303, 369–406.

  • 67.

    He, X.; Hu, R.; Fang, Y.P. Inertial Accelerated Primal-Dual Methods for Linear Equality Constrained Convex Optimization Problems. Numer. Algorithms 2022, 90, 1669–1690.

  • 68.

    He, X.; Hu, R.; Fang, Y.P. Inertial Primal-Dual Dynamics with Damping and Scaling for Linearly Constrained Convex Optimization Problems. Appl. Anal. 2023, 102, 4114–4139.

  • 69.

    He, X.; Hu, R.; Fang, Y.P. Second-Order Primal + First-Order Dual Dynamical Systems with Time Scaling for Linear Equality Constrained Convex Optimization Problems. IEEE Trans. Autom. Control 2022, 67, 4377–4383.

  • 70.

    Zeng, X.; Lei, J.; Chen, J. Dynamical Primal-Dual Nesterov Accelerated Method and Its Application to Network Optimization. IEEE Trans. Autom. Control 2023, 68, 1760–1767.

  • 71.

    Zhao, Y.; Liao, X.; He, X.; et al. Accelerated Primal-Dual Mirror Dynamics for Centralized and Distributed Constrained Convex Optimization Problems. J. Mach. Learn. Res. 2023, 24, 1–59.

  • 72.

    Sun, X.; He, L.; Long, X.J. Inertial Primal-Dual Dynamics with Hessian-Driven Damping and Tikhonov Regularization for Convex-Concave Bilinear Saddle Point Problems. Optimization 2025, 1–29. https://doi.org/10.1080/02331934.
    2025.2578403.

  • 73.

    Sun, X.; Zheng, L.; Teo, K.L. Tikhonov Regularization of Second-Order Plus First-Order Primal-Dual Dynamical Systems for Separable Convex Optimization. J. Optim. Theory Appl. 2025, 207, 12.

  • 74.

    Zhang, H.; Sun, X.; Li, S.; et al. Inertial Accelerated Primal-Dual Algorithms for Non-Smooth Convex Optimization Problems with Linear Equality Constraints. Appl. Math. Optim. 2026, 93, 81.

  • 75.

    Li, H.L.; Hu, R.; He, X.; et al. A General Mixed-Order Primal-Dual Dynamical System with Tikhonov Regularization. J. Optim. Theory Appl. 2025, 207, 34.

  • 76.

    He, X.; Tian, F.; Li, A.Q.; et al. Convergence Rates of Mixed Primal-Dual Dynamical Systems with Hessian Driven Damping. Optimization 2025, 74, 365–390.

  • 77.

    Zhu, T.T.; Hu, R.; Fang, Y.P. Fast Convergence Rates and Trajectory Convergence of a Tikhonov Regularized Inertial Primal-Dual Dynamical System with Time Scaling and Vanishing Damping. J. Comput. Appl. Math. 2025, 460, 116394.

  • 78.

    Sun, X.; Guo, F.; He, L.; et al. Primal-Dual Dynamics Featuring Hessian-Driven Damping and Variable Mass for Convex Optimization Problems. Commun. Nonlinear Sci. Numer. Simul. 2026, 163, 110553.

  • 79.

    Li, H.L.; He, X.; Xiao, Y.B. Fast Convergence of Primal-Dual Dynamical Systems with Implicit Hessian Damping and Tikhonov Regularization. Commun. Nonlinear Sci. Numer. Simul. 2026, 161, 110051.

  • 80.

    Zhang, H.; Sun, X.; Li, S.; et al. Primal-Dual Dynamical Systems with Closed-Loop Control for Convex Optimization in Continuous and Discrete Time. arXiv 2026, arXiv:2602.16402.

  • 81.

    Attouch, H.; Alvarez, F. The Heavy Ball with Friction Dynamical System for Convex Constrained Minimization Problems. In Lecture Notes in Economics and Mathematical Systems, Optimization: Proceedings of the 9th Belgian-French-German Conference on Optimization, Namur, Belgium, 7–11 September 1998; Springer: Berlin/Heidelberg, Germany, 2000; pp. 25–35.

  • 82.

    He, X.; Huang, T.; Yu, J.; et al. An Inertial Projection Neural Network for Solving Variational Inequalities. IEEE Trans. Cybern. 2016, 47, 809–814.

  • 83.

    Qin, S.; Xue, X. A Two-Layer Recurrent Neural Network for Nonsmooth Convex Optimization Problems. IEEE Trans. Neural Netw. Learn. Syst. 2015, 26, 1149–1160.

  • 84.

    Zhao, Y.; He, X.; Huang, T.; et al. Smoothing Inertial Projection Neural Network for Minimization Lp−q in Sparse Signal Reconstruction. Neural Netw. 2018, 99, 31–41.

  • 85.

    Zhao, Y.; Liao, X.; He, X.; et al. Smoothing Inertial Neurodynamic Approach for Sparse Signal Reconstruction via Lp-Norm Minimization. Neural Netw. 2021, 140, 100–112.

  • 86.

    Zhao, Y.; Liao, X.; He, X. Novel Projection Neurodynamic Approaches for Constrained Convex Optimization. Neural Netw. 2022, 150, 336–349.

  • 87.

    Han, C.; Xu, J.; Zheng, B. A Second-Order Projection Neurodynamic Approach with Exponential Convergence for Sparse Signal Reconstruction. Appl. Soft Comput. 2024, 165, 112044.

  • 88.

    Zhang, H.; Zhang, J.; Han, X. A Novel Accelerated Projection Neurodynamic Model for Convex Optimization Problem Constrained by Set and Linear-Equality. In Proceedings of the 2024 6th International Conference on Electronic Engineering and Informatics (EEI), Chongqing, China, 28–30 June 2024; pp. 329–334.

  • 89.

    Zhao, Y.; He, X.; Che, H.; et al. Accelerated Projection Neurodynamic Approach for Convex Optimization with Affine Equality and Inequality Constraints. In Proceedings of the 2024 International Conference on Neuromorphic Computing (ICNC), Chongqing, China, 13–15 December 2024; pp. 1–4.

  • 90.

    Zhao, Y.; He, X.; Zhou, M.; et al. Accelerated Primal-Dual Projection Neurodynamic Approach with Time Scaling for Linear and Set Constrained Convex Optimization Problems. IEEE/CAA J. Autom. Sin. 2024, 11, 1485–1498.

  • 91.

    Zhao, Y.; Allen-Zhao, Z.; Wang, L.; et al. Inertial Primal-Dual Projection Neurodynamic Approaches for Constrained Convex Optimization Problems and Application to Sparse Recovery. Neural Netw. 2025, 186, 107274.

  • 92.

    Liu, K.; Che, H.; Chen, H. A Second-Order Projection Neurodynamic Optimization Approach for αl1 − βl2 Sparsity Regularization Problem. J. Appl. Math. Comput. 2025, 71, 1611–1638.

Share this article:
Download PDF
DOI
10.53941/jmlis.2026.100015
Issue
Volume 2, Issue 3
How to Cite
Wei, J.; Wu, G.-C.; Liu, J. A Survey of Second-Order Neurodynamic Systems from Optimization Perspective. Journal of Machine Learning and Information Security 2026, 2 (3), 15. https://doi.org/10.53941/jmlis.2026.100015.
RIS
BibTex
Copyright & License
article copyright Image
Copyright (c) 2026 by the authors.