Three-term Conjugate Gradient Method Research Articles

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

A hybrid BFGS-Like method for monotone operator equations with applications

In this paper, a hybrid three-term conjugate gradient (CG) method is proposed to solve constrained nonlinear monotone operator equations. The search direction is computed such that it is close to the direction obtained by the memoryless Broyden–Fletcher–Goldferb–Shanno (BFGS) method. Without any condition, the search direction is sufficiently descent and bounded. Moreover, based on some conditions, the search direction satisfy the conjugacy condition without using any line search. The global convergence of the method is established under mild assumptions. Comparison with existing methods is done to test the efficiency of the proposed method through some numerical experiments. Lastly, the applicability of the proposed method is shown.

A New Hybrid Three-Term HS-DY Conjugate Gradient In Solving Unconstrained Optimization Problems

Conjugate Gradient (CG) method is an interesting tool to solve optimization problems in many fields, such design, economics, physics and engineering. Until now, many CG methods have been developed to improve computational performance and have applied in the real-world problems. Combining two CG parameters with distinct denominators may result in non-optimal outcomes and congestion.In this paper, a new hybrid three-term CG method is proposed for solving unconstrained optimization problems. The hybrid threeterm search direction combines Hestenes-Stiefel (HS) and Dai-Yuan (DY) CG parameters which standardized by using a spectral to determine the suitable conjugate parameter choice and it satisfies the sufficient descent condition. Additionally, the global convergence was proved under standard Wolfe conditions and some suitable assumptions. Furthermore, the numerical experiments showed the proposed method is most robust and superior efficiency compared to some existing methods.

Extended modified three-term conjugate gradient method for large-scale nonlinear equations

<p>In this research paper, we introduce a novel gradient-free modified three-term conjugate gradient method designed to solve nonlinear equations subject to convex constraints. Our approach incorporates the projection scheme, which enhances the effectiveness of the proposed method. Building upon the modified three-term conjugate gradient method for solving M-tensor systems and ℓ1- norm-based nonsmooth optimization problems, our method can be regarded as an extension of their technique. By making mild assumptions, we establish the theoretical convergence properties of our iterative method. Through extensive numerical experiments, we demonstrate that our proposed approach is not only highly efficient but also outperforms other existing methods in terms of performance and accuracy.</p>

A Three-term Conjugate Gradient Method with a Random Parameter for Large-scale Unconstrained Optimization and its Application in Regression Model

A Three-term Conjugate Gradient Method with a Random Parameter for Large-scale Unconstrained Optimization and its Application in Regression Model

A New Scaled Three-Term Conjugate Gradient Algorithms For Unconstrained Optimization

Since optimization problems are getting more complicated, new ways to solve them must be thought of, or existing methods must be improved. In this research, we expand the different parameters of the three-term conjugate gradient method to work out unconstrained optimization problems. Our new CG approach meets the conditions of sufficient descent, and global convergence. In addition, we describe some numerical results that imply comparisons to relevant methodologies in the existing research literature.

A truncated three-term conjugate gradient method with complexity guarantees with applications to nonconvex regression problem

In this paper, a truncated three-term conjugate gradient method is proposed for nonconvex unconstrained optimization. At each iteration, the search direction generated by this method is sufficiently descent independent of any line search. We investigate its global convergence with Wolfe line search under the standard assumptions. Moreover, we present its complexity analysis under the objective function with the Lipschitz continuously gradient and Armijo line search which implies that the proposed method is also globally convergent with Armijo line search. A remarkable point about its complexity is that the proposed method possesses the same complexity order as the classical gradient descent method without any restart strategy. Finally, the proposed method is applied to solve the nonconvex robust regression problems. Numerical results show that the proposed method is efficient.

An adaptive modified three-term conjugate gradient method with global convergence

In this paper, an adaptive three-term conjugate gradient method is proposed based on the Perry family by using a modified secant condition. The new method depends on a positive parameter which has a critical role in the efficiency of the algorithm. We propose an adaptive choice for this parameter to control the condition number of the direction matrix by minimizing the distance between the smallest and largest eigenvalue. Global convergence of the new algorithm is proved for general functions, under standard assumptions. Experimental results verify the expected numerical behavior of our algorithm.

A new three-term conjugate gradient method with adaptively alternative two-side approximating strategy that generates descent search direction

A new three-term conjugate gradient method with adaptively alternative two-side approximating strategy that generates descent search direction

A new family of hybrid three-term conjugate gradient method for unconstrained optimization with application to image restoration and portfolio selection

<abstract><p>The conjugate gradient (CG) method is an optimization method, which, in its application, has a fast convergence. Until now, many CG methods have been developed to improve computational performance and have been applied to real-world problems. In this paper, a new hybrid three-term CG method is proposed for solving unconstrained optimization problems. The search direction is a three-term hybrid form of the Hestenes-Stiefel (HS) and the Polak-Ribiére-Polyak (PRP) CG coefficients, and it satisfies the sufficient descent condition. In addition, the global convergence properties of the proposed method will also be proved under the weak Wolfe line search. By using several test functions, numerical results show that the proposed method is most efficient compared to some of the existing methods. In addition, the proposed method is used in practical application problems for image restoration and portfolio selection.</p></abstract>

A Three-Term Conjugate Gradient Method for Non-Convex Functions with Applications for Heat Transfer

The problem of unconstrained optimization (UOP) has recently gained a great deal of attention from researchers around the globe due to its numerous real-life applications. The conjugate gradient (CG) method is among the most widely used algorithms for solving UOP because of its good convergence properties and low memory requirements. This study investigates the performance of a modified CG coefficient for optimization functions, proof of sufficient descent, and global convergence of the new CG method under suitable, standard Wolfe conditions. Computational results on several benchmark problems are presented to validate the robustness and efficacy of the new algorithm. The proposed method was also applied to solve function estimations in inverse heat transfer problems. Another interesting feature possessed by the proposed modification is the ability to solve problems on a large scale and use different dimensions. Based on the theoretical and computational efficiency of the new method, we can conclude that the new coefficient can be a better alternative for solving unconstrained optimization and real-life application problems. KEYWORDS Computational efficiency, global convergence, inverse heat, low memory, optimization problems, theoretical efficiency

A new three-term conjugate gradient method for unconstrained optimization and its applications in image restoration

A new three-term conjugate gradient method for unconstrained optimization and its applications in image restoration

SIGNAL RECOVERY WITH CONSTRAINED MONOTONE NONLINEAR EQUATIONS THROUGH AN EFFECTIVE THREE-TERM CONJUGATE GRADIENT METHOD

In this paper, we introduce a three-term conjugate gradient-type projection method for solving constrained monotone nonlinear equations. In this method, we firstly undertake the transformation of the relative matrices proposed by Yao and Ning. Secondly, we obtain the new relative matrices involving two parameters. Subsequently, we construct a relationship for the two parameters via the quasi-Newton equation and obtain the parameters by simplifying maximum eigenvalue of the new relative matrices. Finally, combining the three-term conjugate gradient method with projection technique, we establish an efficient three-term conjugate gradient-type projection algorithm. Meanwhile, we also give some theoretical analysis about the global convergence and R-linear convergence of the proposed algorithm under quite reasonable technical assumptions. Performance comparisons show that our proposed method is competitive and efficient for solving large-scale nonlinear monotone equations with convex constraints. Furthermore, the presented algorithm is also applied to recovery of a sparse signal in compressive sensing, and obtain practical, efficient and competitive performance in comparing with state-of-the-art algorithms.

A three-term conjugate gradient algorithm with restart procedure to solve image restoration problems

Based on the Liu–Storey conjugate gradient method, a three-term conjugate gradient method with restart procedure is proposed for unconstrained optimization. The designed restart condition is chosen according to the Liu–Storey conjugate parameter, and the restart direction consists of the first and third items in the non-restart direction, which can be picked flexibly. Without depending on any line search, the resulting direction satisfies the sufficient descent condition. Moreover, the proposed method possesses the global convergence under usual assumptions and the weak Wolfe line search. Finally, medium–large-scale numerical experiments are conducted for solving unconstrained optimization and image restoration problems, and the numerical results show that the proposed method is promising and practical, even compared with the state-of-the-art methods.

Variable three-term conjugate gradient method for training artificial neural networks

Artificial neural networks (ANNs) have been widely adopted as general computational tools both in computer science as well as many other engineering fields. Stochastic gradient descent (SGD) and adaptive methods such as Adam are popular as robust optimization algorithms used to train the ANNs. However, the effectiveness of these algorithms is limited because they calculate a search direction based on a first-order gradient. Although higher-order gradient methods such as Newton’s method have been proposed, they require the Hessian matrix to be semi-definite, and its inversion incurs a high computational cost. Therefore, in this paper, we propose a variable three-term conjugate gradient (VTTCG) method that approximates the Hessian matrix to enhance search direction and uses a variable step size to achieve improved convergence stability. To evaluate the performance of the VTTCG method, we train different ANNs on benchmark image classification and generation datasets. We also conduct a similar experiment in which a grasp generation and selection convolutional neural network (GGS-CNN) is trained to perform intelligent robotic grasping. After considering a simulated environment, we also test the GGS-CNN with a physical grasping robot. The experimental results show that the performance of the VTTCG method is superior to that of four conventional methods, including SGD, Adam, AMSGrad, and AdaBelief.

A New Spectral Three-Term Conjugate Gradient Method with Random Parameter Based on Modified Secant Equation and Its Application to Low-Carbon Supply Chain Optimization

In this work, considering the advantages of spectral conjugate gradient method and quasi-Newton method, a spectral three-term conjugate gradient method with random parameter is proposed. The parameter in the search direction of the new method is determined by minimizing the Frobenius norm of difference between search direction matrix and self-scaled memoryless BFGS matrix based on modified secant equation. Then, the search direction satisfying the sufficient descent condition is obtained. The global convergence of new method is proved under appropriate assumptions. Numerical experiments show that our method has better performance by comparing with the up-to-date method. Furthermore, the new method has been successfully applied to the optimization of low-carbon supply chain.

A new three-term conjugate gradient method with application to regression analysis

Conjugate gradient (CG) method is well-known for its ability to solve unconstrained optimization (UO.) problems. This article presenting a new CG method with sufficient descent conditions which improves the former method developed by Rvaie, Mustafa, Ismail and Leong (RMIL). The efficacy of the proposed method has been demonstrated through simulations on the Kijang Emas pricing regression problem. The daily data between January 2021 to May 2021 were obtained from Malaysian Ministry of Health and Bank Negara Malaysia. The dependent variable for this study was the Kijang Emas price, and the independent variables were the coronavirus disease (COVID-19) measures (i.e., new cases, R-naught, death cases, new recovered). Data collected were analyzed on its correlation and coefficient determinant, and the influences of COVID-19 on Kijang Emas price was examined through multiple linear regression model. Findings revealed that the suggested technique outperformed the existing CG algorithms in terms of computing efficiency.

A new three-term conjugate gradient method for training neural networks with global convergence

Conjugate gradient methods (CG) constitute excellent neural network training methods that are simplicity, flexibility, numerical efficiency, and low memory requirements. In this paper, we introduce a new three-term conjugate gradient method, for solving optimization problems and it has been tested on artificial neural networks (ANN) for training a feed-forward neural network. The new method satisfied the descent condition and sufficient descent condition. Global convergence of the new (NTTCG) method has been tested. The results of numerical experiences on some wellknown test function shown that our new modified method is very effective, by relying on the number of functions evaluation and number of iterations, also included the numerical results for training feed-forward neural networks with other well-known method in this field.

A Sufficient Descent Property to Improving a ThreeTerm Conjugate Gradient Algorithm

The nonlinear conjugate gradient (NLCGM) methods have received attention because due to their simplicity, low memory requirements, and global convergent property, which allows them to be used directly to solve large-scale nonlinear unconstrained optimization problems. We suggested a modification to the β KMAR k formula, applied with three-term conjugate gradient method that is both simple and effective, denoted by (TTKMAR), which has a sufficient descent property (SDP) and ensures global convergence (GCP) when we use any line search. The numerical efficiency of TTKMAR was assessed using a variety of standard test functions. TTCGM has been demonstrated to be more numerically efficient than two-term CG methods. This paper also quantifies the difference between TTCGM and two-term methods of performance. As a result, we compare our new modification to an efficient two-term and TTCGM in the numerical results. Finally, we conclude that our proposed modification.

Global convergence of a modified spectral three-term CG algorithm for nonconvex unconstrained optimization problems

Spectral conjugate gradient methods are an efficient family for solving unconstrained optimization problems that have been widely studied in recent decades. In this regard, Li et al. (2019) proposed a spectral three-term conjugate gradient method and proved the global convergence of this algorithm for uniformly convex functions. Our main motivation in this paper is to develop the convergence properties of this method such that the new method possesses suitable convergence for general nonlinear functions. To do end, we introduce a modified spectral conjugate gradient method based on the CG method by Li et al. (2019). We show that the new method fulfills the sufficient descent property without any line search. The new algorithm is globally convergent for general nonlinear functions without the convexity assumption on the objective function. The numerical results indicate that the behavior of the new algorithm is not only effective, but also promising versus other conjugate gradient methods dealing with unconstrained optimization problems of the CUTEst library.