Iranian Journal of Numerical Analysis and Optimization

Iranian Journal of Numerical Analysis and Optimization

A conjugate gradient algorithm based on a combined descent scheme for two-layer neural network training and unconstrained optimization

Document Type : Research Article

Authors
1 Laboratory of Fundamental and Numerical Mathematics (LMFN), Department of Mathematics, University Setif - 1 - Ferhat Abbas, Setif, Algeria.
2 Laboratory of Partial Differential Equations, Department of Mathematics, Faculty of Mathematics and Computer Science, University of Batna 2, Batna, Algeria.
3 University M’Hamed Bougara of Boumerdes, Boumerdes 35000, Algeria..
Abstract
Conjugate gradient methods are considered among the most efficient methods for solving large-scale optimization problems. Among them, the CD (Conjugate-Descent) variant has attracted the attention of many researchers thanks to its straightforward iterative process and low memory requirements. In the present work, we propose an enhancement of the CD method using an efficient combined descent scheme to address large-scale unconstrained optimization problems, with a specific application to training a two-layer neural network model. The sufficient descent condition is analyzed under the strong Wolfe line search conditions, and global convergence is established under mild assumptions. The performance of the suggested algorithm is evaluated on various two-layer neural network models, with the number of weights ranging from 32 to 4080. Furthermore, to validate its effectiveness, the approach is tested on a broad set of unconstrained benchmark functions. Comparisons with recent conjugate gradient methods confirm the robustness and efficiency of the proposed method.
Keywords
Subjects

[1] Andrei, N., An unconstrained optimization test functions collection, Adv. Model. Optim. 10
(2008), 147–161.
[2] Belciug, S., Logistic regression paradigm for training a single-hidden layer feedforward neural
network. Application to gene expression datasets for cancer research, J. Biomed. Inform. 102
(2020), 103373.
[3] Dai, Y.H., Han, J.Y., Liu, G.H., Sun, D.F., Yin, X., and Yuan, Y., Convergence properties
of nonlinear conjugate gradient methods, SIAM J. Optim. 10 (1999), 348–358.
[4] Dai, Y.H., and Yuan, Y., A nonlinear conjugate gradient method with strong global conver-
gence properties, SIAM J. Optim. 10 (1999), 177–182.
[5] Dolan, E.D., and Moré, J.J., Benchmarking optimization software with performance profiles,
Math. Program. 91 (2002), 201–213.
[6] Fletcher, R., and Reeves, C.M., Function minimization by conjugate gradients, Comput. J.
7 (1964), 149–154.
[7] Fletcher, R., Practical Methods of Optimization, 2nd ed., John Wiley & Sons, New York,
1987.
[8] Gilbert, J.C., and Nocedal, J., Global convergence properties of conjugate gradient methods
for optimization, SIAM J. Optim. 2 (1992), 21–42.
[9] Guefassa, I., and Chaib, Y., A modified conjugate gradient method for unconstrained opti-
mization with application in regression function, RAIRO-Oper. Res. 59 (2025), 311–324.
[10] Hestenes, M.R., and Stiefel, E.L., Methods of conjugate gradients for solving linear systems,
J. Res. Natl. Bur. Stand. 49 (1952), 409–436.
[11] Huang, G.B., Zhu, Q.Y., and Siew, C.K., Extreme learning machine: theory and applica-
tions, Neurocomputing, 70 (2006), 489–501.
[12] Jiang, X.Z., and Jian, J.B., Improved Fletcher-Reeves and Dai-Yuan conjugate gradient
methods with the strong Wolfe line search, J. Comput. Appl. Math. 348 (2019), 525–534.
[13] Jiang, X., Pan, L., Liu, M., and Jian, J., A family of spectral conjugate gradient methods
with strong convergence and its applications in image restoration and machine learning, J.
Franklin Inst. 361 (2024), 107033.
[14] Lagaris, I.E., Likas, A., and Fotiadis, D.I., Artificial neural networks for solving ordinary
and partial differential equations, IEEE Trans. Neural Netw. 9 (1998), 987–1000.
[15] Liu, J.K., and Li, S.J., New hybrid conjugate gradient method for unconstrained optimization,
Appl. Math. Comput. 245 (2014), 36–43.
[16] Liu, Y., and Storey, C., Efficient generalized conjugate gradient algorithms, part 1: theory,
J. Comput. Theory Appl. 69 (1991), 129–137.
[17] Maiza, A., Ziadi, R., Saleh, M.A., and Almaymuni, A.Z., An improved conjugate gradient
algorithm by adapting a new line search technique, Electron. Res. Arch. 33 (2025), 2148–
2171.
[18] McDonnell, M.D., Tissera, M.D., Vladusich, T., van Schaik, A., and Tapson, J., Fast, simple
and accurate handwritten digit classification by training shallow neural network classifiers
with the extreme learning machine algorithm, PLoS ONE 10 (2015), e0134254.
[19] Polak, E., and Ribière, G., Note sur la convergence de méthodes de directions conjuguées,
Rev. Française Informat. Recherche Opérationnelle, 3(R1) (1969), 35–43.
[20] Polyak, B.T., The conjugate gradient method in extreme problems, USSR Comput. Math.
Math. Phys. 9 (1969), 94–112.
[21] Rivaie, M., Mamat, M., June, L.W., and Mohd, I., A new class of nonlinear conjugate
gradient coefficients with global convergence properties, Appl. Math. Comput. 218 (2012),
11323–11332.
[22] Souli, C., Ziadi, R., Bencherif-Madani, A., and Khudhur, H.M., A hybrid CG algorithm for
nonlinear unconstrained optimization with application in image restoration, J. Math. Model.
12 (2024), 301–317.
[23] Souli, C., Ziadi, R., Lakhdari, I., and Leulmi, A., An efficient hybrid conjugate gradient
method for unconstrained optimization and image restoration problems, Iran. J. Numer.
Anal. Optim. 15 (2025), 99–123.
[24] Touati-Ahmed, D., and Storey, C., Efficient hybrid conjugate gradient technique, J. Optim.
Theory Appl. 64 (1990), 379–397.
[25] Wei, Z., Yao, S., and Liu, L., The convergence properties of some new conjugate gradient
methods, Appl. Math. Comput. 183 (2006), 1341–1350.
[26] Yousif, O.O.O., Ziadi, R., Almaymuni, A.Z., and Saleh, M.A., An improved version of Polak-
Ribière-Polyak conjugate gradient method with its applications in neural networks training,
Electron. Res. Arch. 33 (2025), 4799–4815.
[27] Zhang, L., An improved Wei-Yao-Liu nonlinear conjugate gradient method for optimization
computation, Appl. Math. Comput. 215 (2009), 2269–2274.
[28] Zhu, Z., Zhang, D., and Wang, S., Two modified DY conjugate gradient methods for uncon-
strained optimization problems, Appl. Math. Comput. 373 (2020), 125004.
[29] Ziadi, R., Maiza, A., Mehamdia, A.E., and Saleh, M.A., An enhanced PRP conjugate
gradient method by adapting a new line search for unconstrained optimization and image
restoration problems, Appl. Set-Valued Anal. Optim. 8 (2026), 357–372.
Send comment about this article
Enter Name.
Enter a valid email address.
Enter a vaid affiliation.
Enter comments (At leaset 10 words)
CAPTCHA Image
Enter Security Code Correctly.