Large-scale Unconstrained Optimization Research Articles

A New Hybrid Descent Algorithm for Large-Scale Nonconvex Optimization and Application to Some Image Restoration Problems

Conjugate gradient methods are widely used and attractive for large-scale unconstrained smooth optimization problems, with simple computation, low memory requirements, and interesting theoretical information on the features of curvature. Based on the strongly convergent property of the Dai–Yuan method and attractive numerical performance of the Hestenes–Stiefel method, a new hybrid descent conjugate gradient method is proposed in this paper. The proposed method satisfies the sufficient descent property independent of the accuracy of the line search strategies. Under the standard conditions, the trust region property and the global convergence are established, respectively. Numerical results of 61 problems with 9 large-scale dimensions and 46 ill-conditioned matrix problems reveal that the proposed method is more effective, robust, and reliable than the other methods. Additionally, the hybrid method also demonstrates reliable results for some image restoration problems.

A Convex combination of improved Fletcher-Reeves and Rivaie-Mustafa-Ismail-Leong conjugate gradient methods for unconstrained optimization problems and applications

Conjugate gradient methods are probably the most used methods in solving large scale unconstrained optimization problems. They have become popular because of their simplicity and low memory requirements. In this paper, we propose a hybrid conjugate gradient method based on the improved Fletcher-Reeves (IFR) and Rivaie-Mustafa-Ismail-Leong+ (RMIL+) methods and establish its global convergence under the strong Wolfe line search conditions. The new conjugate gradient direction satisfies the sufficient descent condition. The method is then compared to other methods in the literature and numerical experiments show that it is competent when solving large scale unconstrained optimization problems. Furthermore, the method is applied to solve a problem in portfolio selection.

A family of spectral conjugate gradient methods with strong convergence and its applications in image restoration and machine learning

In this paper, we propose a family of spectral conjugate gradient methods for solving unconstrained optimization problems. Specifically, we provide two classes of bounded spectral parameters to be chosen, design a new truncation scheme of the non-negative conjugate parameter and set a restart procedure in our search direction. Independently of the specific spectral parameter, conjugate parameter and line search criterion, we prove that our proposed family satisfies the sufficient descent condition. We also prove its strong convergence under mild assumptions and the weak Wolfe line search. Numerical comparisons with other methods demonstrate the outstanding performances of our algorithm for solving medium–large-scale unconstrained optimization, image restoration and machine learning problems.

Stabilized BB projection algorithm for large-scale convex constrained nonlinear monotone equations to signal and image processing problems

The Barzilai–Borwein (BB) method is a popular and efficient gradient method for solving large-scale unconstrained optimization problems. In general, it converges much faster than the steepest descent (Cauchy) method. However, it may not converge, even when the objective function is strongly convex. To overcome this, by virtue of bounding the distance between sequential iterates, [Burdakov et al. (2019)] proposed a stabilized BB method. In this paper, by combining the stabilized BB method with a projection approach, we develop a stabilized BB projection (SBBP) method to solve nonlinear monotone equations with convex constraints. SBBP does not involve computing matrices proving the usefulness for solving large-scale problems. Its global convergence and Q-linear convergence rate are also established. We report numerical experiments on recovering sparse signals and restoring blurred images arising from compressive sensing, demonstrating the usefulness of SBBP.

A New Dai-Liao Conjugate Gradient Method based on Approximately Optimal Stepsize for Unconstrained Optimization

Conjugate gradient methods are a class of very effective iterative methods for large-scale unconstrained optimization. In this paper, a new Dai-Liao conjugate gradient method for solving large-scale unconstrained optimization problem is proposed. Based on the approximately optimal stepsize for the gradient method, we derive three new choices for the important parameters tk in Dai-Liao conjugate gradient method. The search direction satisfies the sufficient descent condition, and the global convergences of the proposed method for uniformly convex and general functions are proved under some mild conditions. Numerical experiments on a set of test problems from the CUTEst library show that the proposed method is superior to some well-known conjugate gradient methods.

Solving Large-Scale Unconstrained Optimization Problems with an Efficient Conjugate Gradient Class

The main goal of this paper is to introduce an appropriate conjugate gradient class to solve unconstrained optimization problems. The presented class enjoys the benefits of having three free parameters, its directions are descent, and it can fulfill the Dai–Liao conjugacy condition. Global convergence property of the new class is proved under the weak-Wolfe–Powell line search technique. Numerical efficiency of the proposed class is confirmed in three sets of experiments including 210 test problems and 11 disparate conjugate gradient methods.

An efficient augmented memoryless quasi-Newton method for solving large-scale unconstrained optimization problems

<p>In this paper, an augmented memoryless BFGS quasi-Newton method was proposed for solving unconstrained optimization problems. Based on a new modified secant equation, an augmented memoryless BFGS update formula and an efficient optimization algorithm were established. To improve the stability of the numerical experiment, we obtained the scaling parameter by minimizing the upper bound of the condition number. The global convergence of the algorithm was proved, and numerical experiments showed that the algorithm was efficient.</p>

A new conjugate gradient for unconstrained optimization problems and its applications in neural networks

We introduce a novel efficient and effective conjugate gradient approach for large-scale unconstrained optimization problems. The primary goal is to improve the conjugate gradient method's search direction in order to propose a new, more active method based on the modified vector , which is dependent on the step size of Barzilai and Borwein. The suggested algorithm features the following traits: (i) The ability to achieve global convergence; (ii) numerical results for large-scale functions show that the proposed algorithm is superior to other comparable optimization methods according to the number of iterations (NI) and the number of functions evaluated (NF); and (iii) training neural networks is done to improve their performance.

An adaptive simple model trust region algorithm based on new weak secant equations

<abstract><p>In this work, we proposed a new trust region method for solving large-scale unconstrained optimization problems. The trust region subproblem with a simple form was constructed based on new weak secant equations, which utilized both gradient and function values and available information from the three most recent points. A modified Metropolis criterion was used to determine whether to accept the trial step, and an adaptive strategy was used to update the trust region radius. The global convergence and locally superlinearly convergence of the new algorithm were established under appropriate conditions. Numerical experiments showed that the proposed algorithm was effective.</p></abstract>

A Sufficient Descent 3-Term Conjugate Gradient Method for Unconstrained Optimization Algorithm

In recent years, 3-term conjugate gradient algorithms (TT-CG) have sparked interest for large scale unconstrained optimization algorithms due to appealing practical factors, such as simple computation, low memory requirement, better sufficient descent property, and strong global convergence property. In this study, minor changes were made to the BRB-CG method used for addressing the optimization algorithms discussed. Then, a new 3-term BRB-CG (MTTBRB) was presented. This new method solved large-scale unconstrained optimization problems. Despite the fact that the BRB algorithm achieved global convergence by employing a modified strong Wolfe line search, in this new MTTBRB-CG method the researchers employed the classical strong Wolfe-Powell condition (SWPC). This study also attempted to quantify how much better 3-term efficiency is than 2-term efficiency. As a result, in the numerical analysis, the new modification was compared to an effective 2-term CG- method. The numerical analysis demonstrated the effectiveness of the proposed method in solving optimization problems.

Some improved Dai–Yuan conjugate gradient methods for large-scale unconstrained optimization problems

Some improved Dai–Yuan conjugate gradient methods for large-scale unconstrained optimization problems

A Three-term Conjugate Gradient Method with a Random Parameter for Large-scale Unconstrained Optimization and its Application in Regression Model

A Three-term Conjugate Gradient Method with a Random Parameter for Large-scale Unconstrained Optimization and its Application in Regression Model

An accelerated descent CG algorithm with clustering the eigenvalues for large-scale nonconvex unconstrained optimization and its application in image restoration problems

Conjugate gradient methods are widely used for solving large-scale unconstrained optimization problems. Since they have the attractive practical factors of simple computation and low memory requirement, interesting theoretical features of curvature information and strong global convergence property. Based on the analysis of the minimization of the condition number and the positiveness of the corresponding matrix, we propose a choice for the parameter in Dai-Liao method and design a descent conjugate gradient algorithm which owns the sufficient descent property independent of the choices for line search techniques. Under some common conditions, the global convergence property for uniformly convex function and the general nonlinear function are established. In the numerical experiments, we firstly focus on 46 ill-conditioned matrix problems and present the corresponding primal results. Then 450 large-scale unconstrained problems are referred. Finally, we give an accelerated strategy for the proposed algorithm and apply it to some image restoration problems. Numerical results indicate that the algorithm is reliable and much more efficient and effective than the other methods for the test problems.

A family of hybrid conjugate gradient method with restart procedure for unconstrained optimizations and image restorations

Conjugate gradient method is one of the most effective methods for solving large-scale optimization problems. Based on the CD conjugate parameter and an improved PRP conjugate parameter, a hybrid conjugate parameter with a single-parameter is designed by using two hybrid techniques, and then a restart procedure is set in its search direction to improve its descent property and computational efficiency. Accordingly, a family of hybrid conjugate gradient method with restart procedure is established, which is sufficient descent at each iteration without depending on any selection of line search criterions. Under usual assumptions and using the weak Wolfe line search criterion to generate the steplengths, the global convergence of the proposed family is proved. Finally, choosing a specific algorithm from this family to solve large-scale unconstrained optimization problems and image restorations, all the numerical results show that the proposed algorithm is effective.

Fuzzy Adaptive Parameter in the Dai–Liao Optimization Method Based on Neutrosophy

The impact of neutrosophy has increased rapidly in many areas of science and technology in recent years. Furthermore, numerous applications of the neutrosophic theory have become more usual. We aim to use neutrosophy to enhance Dai–Liao conjugate gradient (CG) iterative method. In particular, we suggest and explore a new neutrosophic logic system intended to compute the essential parameter t required in Dai–Liao CG iterations. Theoretical examination and numerical experiments signify the effectiveness of the introduced method for controlling t. By incorporation of the neutrosophy in the Dai–Liao conjugate gradient principle, we established novel Dai–Liao CG iterations for solving large-scale unconstrained optimization problems. Global convergence is proved under standard assumptions and with the use of the inexact line search. Finally, computational evidence shows the computational effectiveness of the proposed fuzzy neutrosophic Dai–Liao CG method.

A new hybrid conjugate gradient algorithm for unconstrained optimization

It is well known that conjugate gradient methods are useful for solving large-scale unconstrained nonlinear optimization problems. In this paper, we consider combining the best features of two conjugate gradient methods. In particular, we give a new conjugate gradient method, based on the hybridization of the useful DY (Dai-Yuan), and HZ (Hager-Zhang) methods. The hybrid parameters are chosen such that the proposed method satisfies the conjugacy and sufficient descent conditions. It is shown that the new method maintains the global convergence property of the above two methods. The numerical results are described for a set of standard test problems. It is shown that the performance of the proposed method is better than that of the DY and HZ methods in most cases.

Modified globally convergent Polak-Ribière-Polyak conjugate gradient methods with self-correcting property for large-scale unconstrained optimization

Modified globally convergent Polak-Ribière-Polyak conjugate gradient methods with self-correcting property for large-scale unconstrained optimization

A class of spectral three-term descent Hestenes-Stiefel conjugate gradient algorithms for large-scale unconstrained optimization and image restoration problems

Conjugate gradient methods are much effective and widely used for large-scale unconstrained optimization problems by their simple computation, low memory requirement and strong global convergence property. Spectral gradient methods are also effective for large-scale problems. In this paper, a class of new descent spectral three-term conjugate gradient algorithms are proposed which automatically have the sufficient descent property and satisfy the Dai-Liao conjugate condition. Under the Wolfe line search technique and some standard conditions, the proposed methods are globally convergent for strongly convex functions and general nonlinear functions with the help of the modified secant equations. In numerical part, 732 problems with dimensions varying from 1500 to 150000 and three image restoration problems with three noise levels are considered. Numerical results indicate that the proposed algorithms are more efficient, reliable and robust than the other methods for the testing problems.

A Modified Dai–Liao Conjugate Gradient Method Based on a Scalar Matrix Approximation of Hessian and Its Application

We introduce and investigate proper accelerations of the Dai–Liao (DL) conjugate gradient (CG) family of iterations for solving large-scale unconstrained optimization problems. The improvements are based on appropriate modifications of the CG update parameter in DL conjugate gradient methods. The leading idea is to combine search directions in accelerated gradient descent methods, defined based on the Hessian approximation by an appropriate diagonal matrix in quasi-Newton methods, with search directions in DL-type CG methods. The global convergence of the modified Dai–Liao conjugate gradient method has been proved on the set of uniformly convex functions. The efficiency and robustness of the newly presented methods are confirmed in comparison with similar methods, analyzing numerical results concerning the CPU time, a number of function evaluations, and the number of iterative steps. The proposed method is successfully applied to deal with an optimization problem arising in 2D robotic motion control.

Two families of hybrid conjugate gradient methods with restart procedures and their applications

In this paper, two families of hybrid conjugate gradient methods with restart procedures are proposed. Their hybrid conjugate parameters are yielded by projection or convex combination of the classical parameters. Moreover, their restart procedures are given uniformly, which are determined by the proposed hybrid conjugate parameters. The search directions of the presented families satisfy the sufficient descent condition. Under usual assumption and the weak Wolfe line search, the proposed families are proved to be globally convergent. Finally, choosing a specific parameter for each family to solve large-scale unconstrained optimization problems, convex constrained nonlinear monotone equations and image restoration problems. All the numerical results are reported and analysed, which show that the proposed families of hybrid conjugate gradient methods are promising.