Classical Conjugate Gradient Method Research Articles

A delayed weighted gradient method for strictly convex quadratic minimization

In this paper is developed an accelerated version of the steepest descent method by a two-step iteration. The new algorithm uses information with delay to define the iterations. Specifically, in the first step, a prediction of the new test point is calculated by using the gradient method with the exact minimal gradient steplength and then, a correction is computed by a weighted sum between the prediction and the predecessor iterate of the current point. A convergence result is provided. In order to compare the efficiency and effectiveness of the proposal, with similar methods existing in the literature, numerical experiments are performed. The numerical comparison of the new algorithm with the classical conjugate gradient method shows that our method is a good alternative to solve large-scale problems.

A conjugate gradient algorithm for large-scale nonlinear equations and image restoration problems

Nonlinear systems present a quite complicate problem. As the number of dimensions increases, it becomes more difficult to find the solution of the problem. In this paper, a modified conjugate gradient method is designed that has a sufficient descent property and trust region property. It is interesting that the formula for search direction makes full use of the property of convex combination between the deepest descent algorithm and the classical LS conjugate gradient (CG) method. The global convergence property is established under certain conditions, and the numerical results show that the modified method is effective for normal nonlinear equations and image restoration problems.

A comparison on classical-hybrid conjugate gradient method under exact line search

One of the popular approaches in modifying the Conjugate Gradient (CG) Method is hybridization. In this paper, a new hybrid CG is introduced and its performance is compared to the classical CG method which are Rivaie-Mustafa-Ismail-Leong (RMIL) and Syarafina-Mustafa-Rivaie (SMR) methods. The proposed hybrid CG is evaluated as a convex combination of RMIL and SMR method. Their performance are analyzed under the exact line search. The comparison performance showed that the hybrid CG is promising and has outperformed the classical CG of RMIL and SMR in terms of the number of iterations and central processing unit per time.

A Combination of FR and HS Coefficient in Conjugate Gradient Method for Unconstrained Optimization

The conjugate gradient (CG) method is one of the most popular methods for solving large-scale problems of unconstrained optimization. In this paper, a new CG method based on combination of two classical CG methods of Fletcher-Reeves (FR), and Hestence-Stiefel (HS) is proposed. This method possess the global convergence properties and the sufficient descent condition. The tests of the new CG method by using MATLAB are measured in terms of central processing unit (CPU) time and iteration numbers with strong Wolfe-Powell inexact line search. Results presented have shown that the new CG method performs better compare to other CG methods.

Sparse approximate solutions to stochastic Galerkin equations

In this Note, we formulate a sparse Krylov-based algorithm for solving large-scale linear systems of algebraic equations arising from the discretization of randomly parametrized (or stochastic) elliptic partial differential equations (SPDEs). We analyze the proposed sparse conjugate gradient (CG) algorithm within the framework of inexact Krylov subspace methods, prove its convergence and study its abstract computational cost. Numerical studies conducted on stochastic diffusion models show that the proposed sparse CG algorithm outperforms the classical CG method when the sought solutions admit a sparse representation in a polynomial chaos basis. In such cases, the sparse CG algorithm recovers almost exactly the sparsity pattern of the exact solutions, which enables accelerated convergence. In the case when the SPDE solution does not admit a sparse representation, the convergence of the proposed algorithm is very similar to the classical CG method.

An adaptive family of projection methods for constrained monotone nonlinear equations with applications

An adaptive class of conjugate gradient methods is proposed in this paper, which all possess descent property under strong-wolfe line search. The adaptive parameter in the search direction is determined by minimizing the distance between relative matrix and self-scaling memoryless BFGS update by Oren in the Frobenius norm. Two formulas of the adaptive parameter are further obtained, which are presented as those given by Oren and Luenberger (1973/74) and respectively Oren and Spedicato (1976). By projection technology, two adaptive projection algorithms are developed for solving monotone nonlinear equations with convex constraints. Some numerical comparisons show that these two algorithms are efficient. Last, the proposed algorithms are used to recover a sparse signal from incomplete and contaminated sampling measurements; the results are promising.

Discontinuous Galerkin Methods for Compressible and Incompressible Flows on Space–Time Adaptive Meshes: Toward a Novel Family of Efficient Numerical Methods for Fluid Dynamics

In this work the numerical discretization of the partial differential governing equations for compressible and incompressible flows is dealt within the discontinuous Galerkin (DG) framework along space–time adaptive meshes. Two main numerical frameworks can be distinguished: (1) fully explicit ADER-DG methods on collocated grids for compressible fluids (2) spectral semi-implicit and spectral space–time DG methods on edge-based staggered grids for the incompressible Navier–Stokes equations. In this work, the high-resolution properties of the aforementioned numerical methods are significantly enhanced within a 'cell-by-cell' Adaptive Mesh Refinement (AMR) implementation together with time accurate local time stepping (LTS). It is a well known fact that a major weakness of high order DG methods lies in the difficulty of limiting discontinuous solutions, which generate spurious oscillations, namely the so-called 'Gibbs phenomenon'. Over the years, several attempts have been made to cope with this problem and different kinds of limiters have been proposed. In this work the nonlinear stabilization of the scheme is sequentially and locally introduced only for troubled cells on the basis of a multidimensional optimal order detection (MOOD) criterion. ADER-DG is a novel, communication-avoiding family of algorithms, which achieves high order of accuracy in time not via the standard multi-stage Runge–Kutta (RK) time discretization like most other DG schemes, but at the aid of an element-local predictor stage. In practice the method first produces a so-called candidate solution by using a high order accurate unlimited DG scheme. Then, in those cells where at least one of the chosen admissibility criteria is violated the computed candidate solution is detected as troubled and is locally rejected. Next, the numerical solution of the previous time step is scattered onto cell averages on a suitable sub-grid in order to preserve the natural sub-cell resolution of the DG scheme. Then, a more reliable numerical solution is recomputed a posteriori by employing a more robust but still very accurate ADER-WENO finite volume scheme on the sub-grid averages within that troubled cell. Finally, a high order DG polynomial is reconstructed back from the evolved sub-cell averages. In the ADER-DG framework several PDE system are investigated, ranging from the Euler equations of compressible gas dynamics, over the viscous and resistive magneto-hydrodynamics (MHD), to special and general relativistic MHD. Indeed, the adopted formalism is quite general, leading to a novel family of adaptive ADER-DG schemes suitable for hyperbolic systems of partial differential equations in which the numerical fluxes also depend on the gradient of the state vector because of the parabolic nature of diffusive terms. The presented results show clearly that the high-resolution and shock-capturing capabilities of the news schemes are significantly enhanced within the cell-by-cell AMR implementation together with time accurate LTS. A special treatment has been followed for the incompressible Navier–Stokes equations. In fact, the elliptic character of the incompressible Navier–Stokes equations introduces an important difficulty in their numerical solution: whenever the smallest physical or numerical perturbation arises in the fluid flow then it will instantaneously affect the entire computational domain. Thus, a semi-implicit approach has been used. The main advantage of making use of a semi-implicit discretization is that the numerical stability can be obtained for large time-steps without leading to an excessive computational demand. In this context, we derived two new families of spectral semi-implicit and spectral space–time DG methods for the solution of the two and three dimensional Navier–Stokes equations on edge-based adaptive staggered Cartesian grids. The discrete solutions of pressure and velocity are expressed in the form of piecewise polynomials along different meshes. While the pressure is defined on the control volumes of the main grid, the velocity components are defined on edge-based dual control volumes, leading to a spatially staggered mesh. In the first family, high order of accuracy is achieved only in space, while a simple semi-implicit time discretization is derived by introducing an implicitness factor $$\theta \in [0.5,1]$$ for the pressure gradient in the momentum equation. The real advantages of the staggering arise after substituting the discrete momentum equation into the weak form of the continuity equation. In fact, the resulting linear system for the pressure is symmetric and positive definite and either block penta-diagonal (in 2D) or block hepta-diagonal (in 3D). As a consequence, the pressure system can be solved very efficiently by means of a classical matrix-free conjugate gradient method. The resulting algorithm is stable, computationally very efficient, and at the same time arbitrary high order accurate in both space and time. This new numerical method has been thoroughly validated for approximation polynomials of degree up to $$N = 12$$, using a large set of non-trivial test problems in two and three space dimensions, for which either analytical, numerical or experimental reference solutions exist.

A new class of conjugate gradient methods for unconstrained smooth optimization and absolute value equations

In this paper, we introduce a new three-term conjugate gradient (NTTCG) method to solve unconstrained smooth optimization problems. NTTCG is based on conjugate gradient methods proposed by Dai and Yuan (SIAM J Optim 10:177–182, 1999) and Polak and Ribiere (Rev Francaise Inform Rech Oper 3(16):35–43, 1969). The descent property of the direction generated by NTTCG in each iteration is established. Under some standard assumptions, the global convergence results of the new methods are investigated. The extension of this algorithm is proposed to solve absolute value equations (AVE), called three-term conjugate subgradient (NTTCS) method. Numerical experiments are reported for unconstrained CUTEst problems and AVE.

Higher order curvature information and its application in a modified diagonal Secant method

ABSTRACTA secant equation (quasi-Newton) has one of the most important rule to find an optimal solution in nonlinear optimization. Curvature information must satisfy the usual secant equation to ensure positive definiteness of the Hessian approximation. In this work, we present a new diagonal updating to improve the Hessian approximation with a modifying weak secant equation for the diagonal quasi-Newton (DQN) method. The gradient and function evaluation are utilized to obtain a new weak secant equation and achieve a higher order accuracy in curvature information in the proposed method. Modified DQN methods based on the modified weak secant equation are globally convergent. Extended numerical results indicate the advantages of modified DQN methods over the usual ones and some classical conjugate gradient methods.

New Hybrid Conjugate Gradient Method with Global Convergence Properties under Exact Line Search

Conjugate Gradient (CG) method is a very useful technique for solving large-scale nonlinear optimization problems. In this paper, we propose a new formula for 12خ²k"> , which is a hybrid of PRP and WYL methods. This method possesses sufficient descent and global convergence properties when used with exact line search. Numerical results indicate that the new formula has higher efficiency compared with other classical CG methods.

A class of conjugate gradient methods for convex constrained monotone equations

The recent designed non-linear conjugate gradient method of Dai and Kou [SIAM J Optim. 2013;23:296–320] is very efficient currently in solving large-scale unconstrained minimization problems due to its simpler iterative form, lower storage requirement and its closeness to the scaled memoryless BFGS method. Just because of these attractive properties, this method was extended successfully to solve higher dimensional symmetric non-linear equations in recent years. Nevertheless, its numerical performance in solving convex constrained monotone equations has never been explored. In this paper, combining with the projection method of Solodov and Svaiter, we develop a family of non-linear conjugate gradient methods for convex constrained monotone equations. The proposed methods do not require the Jacobian information of equations, and even they do not store any matrix in each iteration. They are potential to solve non-smooth problems with higher dimensions. We prove the global convergence of the class of the proposed methods and establish its R-linear convergence rate under some reasonable conditions. Finally, we also do some numerical experiments to show that the proposed methods are efficient and promising.

Semi-implicit discontinuous Galerkin methods for the incompressible Navier–Stokes equations on adaptive staggered Cartesian grids

In this paper a new high order semi-implicit discontinuous Galerkin method (SI-DG) is presented for the solution of the incompressible Navier–Stokes equations on staggered space-time adaptive Cartesian grids (AMR) in two and three space dimensions. The pressure is written in the form of piecewise polynomials on the main grid, which is dynamically adapted within a cell-by-cell AMR framework. According to the time dependent main grid, different face-based spatially staggered dual grids are defined for the piece-wise polynomials of the respective velocity components. Although the resulting adaptive staggered grids are more complex than classical uniform Cartesian meshes, the numerical scheme can still be written in a rather compact form. Thanks to the use of a tensor-product formulation for the definition of the nodal basis in the d-dimensional space (d=2,3), all the discrete operators can be efficiently written as a combination of linear one-dimensional operators acting in the d space directions separately.Arbitrary high order of accuracy is achieved in space, while a very simple semi-implicit time discretization is obtained via an explicit discretization of the nonlinear convective terms, and an implicit discretization of the pressure gradient in the momentum equation and of the divergence of the velocity field in the continuity equation. The real advantages of the staggered grid arise in the solution of the Schur complement associated with the saddle point problem of the discretized incompressible Navier–Stokes equations, i.e. after substituting the discrete momentum equations into the discrete continuity equation. This leads to a linear system for only one unknown, the scalar pressure. Indeed, the resulting linear pressure system is shown to be symmetric and positive-definite. In order to avoid a quadratic stability condition for the parabolic terms given by the viscous stress tensor, an implicit discretization is also used for the diffusive terms in the momentum equation. A particular feature of our staggered DG approach is that the viscous stress tensor is discretized on the dual mesh. This corresponds to the use of a lifting operator, but on the staggered grid. Both linear systems for pressure and velocity are very efficiently solved by means of a classical matrix-free conjugate gradient method, for which fast convergence is observed. Moreover, it should be noticed that all test cases shown in this paper have been performed without the use of any preconditioner. Due to the explicit discretization of the nonlinear convective terms, the final algorithm is stable within the classical CFL-type time step restriction for explicit DG methods based on the fluid velocity. Moreover, by using a suitable local time stepping technique based on the ADER-DG framework, each element can use its optimal local time step also within the AMR context.The new space-time adaptive staggered DG scheme has been thoroughly verified for polynomial degrees up to N=9 for a large set of non-trivial test problems in two and three space dimensions, for which analytical, numerical or experimental reference solutions exist. The high order of accuracy of the scheme is shown by means of a numerical convergence table obtained by comparing the numerical solution against a smooth analytical solution.To the knowledge of the authors, this is the first staggered semi-implicit DG scheme for the incompressible Navier–Stokes equations on space-time adaptive meshes in two and three space dimensions.

A conjugate gradient like method for p-norm minimization in functional spaces

We develop an iterative algorithm to recover the minimum p-norm solution of the functional linear equation $$Ax = b,$$ where $$A: \mathcal {X}\longrightarrow \mathcal {Y}\,$$ is a continuous linear operator between the two Banach spaces $$\mathcal {X}= L^p$$ , $$1< p < 2$$ , and $$\mathcal {Y}= L^r$$ , $$r > 1$$ , with $$x \in \mathcal {X}$$ and $$b \in \mathcal {Y}$$ . The algorithm is conceived within the same framework of the Landweber method for functional linear equations in Banach spaces proposed by Schopfer et al. (Inverse Probl 22:311–329, 2006). Indeed, the algorithm is based on using, at the n-th iteration, a linear combination of the steepest current “descent functional” $$A^* J \left( b - A x_n \right) $$ and the previous descent functional, where J denotes a duality map of the Banach space $$\mathcal {Y}$$ . In this regard, the algorithm can be viewed as a generalization of the classical conjugate gradient method on the normal equations in Hilbert spaces. We demonstrate that the proposed iterative algorithm converges strongly to the minimum p-norm solution of the functional linear equation $$Ax = b$$ and that it is also a regularization method, by applying the discrepancy principle as stopping rule. According to the geometrical properties of $$L^p$$ spaces, numerical experiments show that the method is fast, robust in terms of both restoration accuracy and stability, promotes sparsity and reduces the over-smoothness in reconstructing edges and abrupt intensity changes.

Derivative-free restrictively preconditioned conjugate gradient path method without line search technique for solving linear equality constrained optimization

In this paper, we propose a derivative-free restrictively preconditioned conjugate gradient path method without line search technique for solving linear equality constrained optimization. This method originates from the classical conjugate gradient method and its restrictively preconditioned variant. The preconditioned conjugate gradient path is obtained by solving extended system of the preconditioned equation with restrictively preconditioned conjugate gradient method which is a more robust and effective technique to solve large sparse linear system. The global convergence and local superlinear convergence rate of the proposed method are established under some reasonable conditions. Finally, the numerical results are reported to show the effectiveness of the proposed method.

A scaled conjugate gradient method for nonlinear unconstrained optimization

We propose a new optimization problem which combines the good features of the classical conjugate gradient method using some penalty parameter, and then, solve it to introduce a new scaled conjugate gradient method for solving unconstrained problems. The method reduces to the classical conjugate gradient algorithm under common assumptions, and inherits its good properties. We prove the global convergence of the method using suitable conditions. Numerical results show that the new method is efficient and robust.

Spectral semi-implicit and space–time discontinuous Galerkin methods for the incompressible Navier–Stokes equations on staggered Cartesian grids

In this paper two new families of arbitrary high order accurate spectral discontinuous Galerkin (DG) finite element methods are derived on staggered Cartesian grids for the solution of the incompressible Navier–Stokes (NS) equations in two and three space dimensions. The discrete solutions of pressure and velocity are expressed in the form of piecewise polynomials along different meshes. While the pressure is defined on the control volumes of the main grid, the velocity components are defined on edge-based dual control volumes, leading to a spatially staggered mesh. Thanks to the use of a nodal basis on a tensor-product domain, all discrete operators can be written efficiently as a combination of simple one-dimensional operators in a dimension-by-dimension fashion.In the first family, high order of accuracy is achieved only in space, while a simple semi-implicit time discretization is derived by introducing an implicitness factor θ∈[0.5,1] for the pressure gradient in the momentum equation. The real advantages of the staggering arise after substituting the discrete momentum equation into the weak form of the continuity equation. In fact, the resulting linear system for the pressure is symmetric and positive definite and either block penta-diagonal (in 2D) or block hepta-diagonal (in 3D). As a consequence, the pressure system can be solved very efficiently by means of a classical matrix-free conjugate gradient method. From our numerical experiments we find that the pressure system appears to be reasonably well-conditioned, since in all test cases shown in this paper the use of a preconditioner was not necessary. This is a rather unique feature among existing implicit DG schemes for the Navier–Stokes equations. In order to avoid a stability restriction due to the viscous terms, the latter are discretized implicitly using again a staggered mesh approach, where the viscous stress tensor is also defined on the dual mesh.The second family of staggered DG schemes proposed in this paper achieves high order of accuracy also in time by expressing the numerical solution in terms of piecewise space–time polynomials. In order to circumvent the low order of accuracy of the adopted fractional stepping, a simple iterative Picard procedure is introduced, which leads to a space–time pressure-correction algorithm. In this manner, the symmetry and positive definiteness of the pressure system are not compromised. The resulting algorithm is stable, computationally very efficient, and at the same time arbitrary high order accurate in both space and time. These features are typically not easy to obtain all at the same time for a numerical method applied to the incompressible Navier–Stokes equations. The new numerical method has been thoroughly validated for approximation polynomials of degree up to N=11, using a large set of non-trivial test problems in two and three space dimensions, for which either analytical, numerical or experimental reference solutions exist.

Diagonal quasi-Newton method via variational principle under generalized Frobenius norm

In this work, we present a new class of diagonal quasi-Newton methods for solving large-scale unconstrained optimization problems. The methods are derived by means of variational principle under the generalized Frobenius norm. We show global convergence of our methods under the standard line search with Armijo condition. Numerical results are carried out in standard test problems and clearly indicate vast superiority over some classical conjugate gradient methods.

A class of adaptive Dai–Liao conjugate gradient methods based on the scaled memoryless BFGS update

Minimizing the distance between search direction matrix of the Dai–Liao method and the scaled memoryless BFGS update in the Frobenius norm, and using Powell’s nonnegative restriction of the conjugate gradient parameters, a one-parameter class of nonlinear conjugate gradient methods is proposed. Then, a brief global convergence analysis is made with and without convexity assumption on the objective function. Preliminary numerical results are reported; they demonstrate a proper choice for the parameter of the proposed class of conjugate gradient methods may lead to promising numerical performance.

A class of descent four–term extension of the Dai–Liao conjugate gradient method based on the scaled memoryless BFGS update

Hybridizing the three–term conjugate gradient method proposed by Zhang et al. and the nonlinear conjugate gradient method proposed by Dai and Liao based on the scaled memoryless BFGS update, a one–parameter class of four–term conjugate gradient methods is proposed. It is shown that the suggested class of conjugate gradient methods possesses the sufficient descent property, without convexity assumption on the objective function. A brief global convergence analysis is made for uniformly convex objective functions. Results of numerical comparisons are reported. They demonstrate efficiency of a method of the proposed class in the sense of the Dolan–More performance profile.

Fast Density Inversion Solution for Full Tensor Gravity Gradiometry Data

We modify the classical preconditioned conjugate gradient method for full tensor gravity gradiometry data. The resulting parallelized algorithm is implemented on a cluster to achieve rapid density inversions for various scenarios, overcoming the problems of computation time and memory requirements caused by too many iterations. The proposed approach is mainly based on parallel programming using the Message Passing Interface, supplemented by Open Multi-Processing. Our implementation is efficient and scalable, enabling its use with large-scale data. We consider two synthetic models and real survey data from Vinton Dome, US, and demonstrate that our solutions are reliable and feasible.