Generalized Conjugate Gradient Research Articles

An engineering approach to generalized conjugate gradient methods and beyond

This is an elementary introductory paper (rather a tutorial) for generalized conjugate gradient (CG) methods. The emphasis is on “understanding” the approach of the now rather sophisticated CG-type methods and beyond. Practical aspects like normalization and smoothing are discussed. Then the development of different types of methods with their characteristic properties is presented. For a model of the Navier-Stokes equations that varies between strong and weak ellipticity, the methods are compared and their typical properties are demonstrated.

An Optimization Strategy for Maximizing Coordinate Measuring Machine Productivity, Part 2: Problem Formulation, Solution, and Experimental Results

In this, the second of two papers focused on optimizing CMM productivity, a strategy is presented for obtaining operating speeds that maximize CMM throughput. The strategy involves posing the objective and constraints in terms of a nonlinear programming problem. The objective is to minimize measuring time, and constraints are on measurement quality, available motor power and other machine specific characteristics. Solutions to various measurement scenarios are given, computed with a generalized conjugate gradient method that uses a golden section acceptable search method. Also, a sensitivity study is presented that utilizes the optimization strategy to determine how change to a CMM’s design will affect overall productivity. In the final section, experimental results utilizing the optimally computed trajectories for both single axis and multiple axis motion are given. In addition, an example is presented to demonstrate the implementation of the optimization strategy in an actual measuring application. In the example, a connecting-rod is measured at three speeds: the machine’s default speed, a speed constrained by available motor power, and the optimized speed (constrained additionally by measurement quality). The experiment demonstrates a 27 percent reduction in time for the optimized speed compared to the default speed using the Bartlett test for comparing variances in the dimensional data obtained.

The use of conjugate gradients methods with a segregated finite volume procedure for solving transient, incompressible Navier–Stokes equations

AbstractPISO (pressure‐implicit with splitting of operators) is an algorithm devised to solve the transient incompressible Navier–Stokes and energy equations. In that it uses separate equations for each of two pressure corrections, velocity components and transported scalars, PISO is known as a segregated solution procedure. Preconditioned, generalized conjugate gradient (GCG) methods in combination with schemes for vectorization and storage minimization are described in the paper for the purpose of solving both the symmetric and non‐symmetric algebraic equation systems that result from the PISO algorithm. Of particular interest in the paper is a comparison of the present procedure with a procedure described by Chin et al. (1992) in which preconditioned conjugate gradient methods are used to solve the Jacobian matrix used in the Newton iteration of the fully coupled non‐linear equations. The two methods are similar in that both techniques are based on a discretization using staggered control volumes with the power‐law method for modelling advection‐diffusion transport.

Iterative techniques for rough surface scattering problems

The problem of wave scattering at a randomly rough surface gives rise to a non-Hermitian linear system, which corresponds to the discretization of a linear Fredholm integral equation of the first or second kind. In this paper, we describe the use of various iterative methods for the solution of the real linear non-symmetric system, corresponding to the complex system of the scattering problem, and the regions of convergence of the methods in terms of the basic parameters of the problem. It has been found that Generalized Conjugate Gradient method is the only approach valid for very rough surfaces, h l . 1.0 say (here h l is the ratio of the RMS surface height to surface correlation length). The regions of validity for the iterative methods in terms of convergence and energy conservation are also found and it is shown that this depends not only on h l and kl (here k is the wavenumber of the incident field) but also on the density of mesh points in the discretization. Some convergence properties are discussed by analyzing the spectral radius of the iterative matrix. It is found for small h l that the spectral radius is linearly proportional to h l and approximately proportional to √ kl for large kl. It is also found numerically that when h l · √kl is greater than one, the iterative matrix for the linear integral equation of the second kind treated here has spectral radius greater than one, and therefore, the Generalized Conjugate Gradient method is not guaranteed to converge since the corresponding linear real system of the second kind may not have positive definite symmetric part. Finally, we present a comparison of iterative techniques for the linear integral equations of the first and second kinds, and find that when such iterative methods are to be employed, it is more efficient to use the second kind integral equation formulation than the first kind formulation.

Preconditioned cg‐like methods for solving non‐linear convection—diffusion equations

The paper presents the numerical performance of the preconditioned generalized conjugate gradient (PGCG) methods in solving non‐linear convection — diffusion equations. Three non‐linear systems which describe a non‐isothermal chemical reactor, the chemically driven convection in a porous medium and the incompressible steady flow past a sphere are the test problems. The standard second order accurate centred finite difference scheme is used to discretize the models equations. The discrete approximations are solved with a double iterative process using the Newton method as outer iteration and the PGCG algorithm as inner iteration. Three PGCG techniques, which emerge to be the best performing, are tested. Laplace‐type operators are employed for preconditioning. The results show that the convergence of the PGCG methods depends strongly on the convection—diffusion ratio. The most robust algorithm is GMRES. But even with GMRES non‐convergence occurs when the convection—diffusion ratio exceeds a limit value. Thi...

A note on preconditioned iterative methods for least squares problems

In this note new preconditioners are introduced for least squares problems with the result that new smaller reduced systems are derived. Then the SOR method, the generalized conjugate gradient method and the adjusted generalized conjugate gradient method can be applied to solve the problem with less computational work.

On nonlinear generalized conjugate gradient methods

The Generalized Conjugate Gradient method (see [1]) is an iterative method for nonsymmetric linear systems. We obtain generalizations of this method for nonlinear systems with nonsymmetric Jacobians. We prove global convergence results.

A generalized conjugate gradient model for the mild slope equation

In this paper the mild slope equation is solved by a generalized conjugate gradient method. The present model has a fast convergence rate, relatively small storage requirement and is easy to apply to complicated non-rectangular regions. Experimental data of wave propagating over a elliptic shoal obtained by Berkhoff et al. (1982) are used to test the model.

An evolution equation for water waves

An evolution equation is developed from the mild slope equation by using the perturbation method. For large times the solutions of the model will approach results of the elliptic mild slope equation. The alternating direction implicit method is used to solve the equation and the numerical scheme is unconditional stable. Another model based on wave amplitude and phase function is derived straight-forward from the time dependent parabolic model and it can be applied to a large area. Experimental data obtained by Berkhoff et al. (1982) are used to test the model. Comparison of the present model results for the case of an elliptic shoal against the Generalized Conjugate Gradient model results has been made and good agreement has been achieved.

Error-Minimizing Krylov Subspace Methods

Iterative methods for the solution of linear systems are usually controlled by the observation of the norm of the residual. In reality, the error should be controlled, but the error is not available. The residuals and the errors are connected by the condition number of the system matrix. If the system is well conditioned, the decrease of the errors is closely connected to the decrease of the residuals. For these cases, Krylov subspace methods that minimize the residuals in the Euclidean norm or in the energy norm are powerful solution techniques. If the system is ill conditioned, the residuals can decrease while the errors increase. For these systems, arising even from very simple and commonly used differential equations, iterative methods that minimize the residuals may require a large number of iterations to reduce the errors. The user may be misled to stop the iteration too early by small residuals. Two families of error-minimizing Krylov subspace methods are proposed to overcome these difficulties. Each of them is suited for different problem types. Principles for the design of generalized cg methods that minimize the error are derived from the geometric convergence behavior of generalized cg methods. These methods use the transposed system matrix multiplied by the system matrix as the iteration matrix. By this technique a fast convergence is achieved for matrices with clustered singular values and scattered eigenvalues. A class of Krylov subspace methods minimizing the error by using the simple transposed matrix as the iteration matrix is proposed. Various realization possibilities are inherent in these generalized minimum error methods. The methods are analyzed theoretically. Common and related properties with generalized conjugate gradient methods are presented. These techniques should be preferred if the eigenvalues are more clustered than the singular values. The first promising tests for one distinct method are presented.

Properties of generalized conjugate gradient methods

AbstractThe solution of linear systems has considerable importance for the computation of problems resulting from engineering, physics, chemistry, computer science, mathematics, medicine and economics. The calculation of costly and time‐consuming problems, e.g. crash tests, simulation of the human lung and skin, calculation of electrical and magnetical fields, thermal analysis and fluid dynamics, to name only a few, has become possible with the recent developments of advanced computer architectures and iterative solvers. Generalized conjugate gradient (CG) methods are the most important iterative solvers because they converge very quickly under certain conditions. Therefore they are widely used and in a rapid further development.The purpose of this paper is to present new results for the convergence of generalized CG methods. A convergence result for non‐symmetric and non‐positive definite matrices is given that includes the classical theory for symmetric, positive definite matrices as a special case.The norm of the residuals resulting from CG methods may oscillate heavily. Different remedies for smoothing this sequence have been proposed, for example by van der Vorst. Schönauer introduced in the 1980s a smoothing algorithm to get a norm nonincreasing function of the iteration index. For this algorithm a complete theoretical analysis is given. A surprising result is obtained showing that the smoothing algorithm is in a sense optimal. Convergence estimates are derived therefrom. A geometric interpretation of the smoothing algorithm is given showing the propagation of the errors.It should be stressed that a smooth convergence of the residuals is not equivalent to a smooth convergence of the errors which is the proper aim. A class of error minimizing methods can be easily derived from the theory.

Two-Level Local Refinement Preconditioners for Nonsymmetric and Indefinite Elliptic Problems

Preconditioners of optimal order for nonselfadjoint and indefinite elliptic boundary value problems discretized on grids with local refinement are constructed. The proposed technique utilizes solution of a discrete problem on a uniform coarse grid; then, the reduced problem is handled by a generalized conjugate gradient (GCG) method. The reduced problem is coercive if the initial coarse mesh is sufficiently fine and is local, solving only for the unknowns on the subdomains where local refinement has been introduced. The reduced problem can be preconditioned by a preconditioner for the symmetric positive definite matrix arising from the symmetric and coercive principal part of the original bilinear form restricted to the subdomains containing local refinement. This problem also utilizes a uniform grid. In the numerical tests, the recent algebraic multilevel (AMLI) preconditioners [Axelsson and Vassilevski, SIAM j. Numer. Anal., 27 (1990), pp. 1569–1590; Saad and Schultz, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 856–869], which are of optimal order for selfadjoint and coercive elliptic problems, were used.

Global convergence of three terms conjugate gradient methods∗

This paper considers the general conjugate gradient algorithms for unconstrained optimization whose search directions are defined by , [UM0002] where t(p) denotes the iteration index at which the p-th restarting occurs. It is obviously seen that Beale method (1972) and Powell restart method (1977) can be considered as two special cases. A global convergence theorem is proved and a new three terms conjugate gradient method is proposed.

Conjugate Gradient Acceleration of the EM Algorithm

The EM algorithm is a very popular and widely applicable algorithm for the computation of maximum likelihood estimates. Although its implementation is generally simple, the EM algorithm often exhibits slow convergence and is costly in some areas of application. Past attempts to accelerate the EM algorithm have most commonly been based on some form of Aitken acceleration. Here we propose an alternative method based on conjugate gradients. The key, as we show, is that the EM step can be viewed (approximately at least) as a generalized gradient, making it natural to apply generalized conjugate gradient methods in an attempt to accelerate the EM algorithm. The proposed method is relatively simple to implement and can handle Problems with a large number of parameters, an important feature of most EM algorithms. To demonstrate the effectiveness of the proposed acceleration method, we consider its application to several Problems in each of the following areas: estimation of a covariance matrix from incomplete mu...

On mesh independence and Newton-type methods

Mesh-independent convergence of Newton-type methods for the solution of nonlinear partial differential equations is discussed. First, under certain local smoothness assumptions, it is shown that by properly relating the mesh parameters $H$ and $h$ for a coarse and a fine discretization mesh, it suffices to compute the solution of the nonlinear equation on the coarse mesh and subsequently correct it once using the linearized (Newton) equation on the fine mesh. In this way the iteration error will be of the same order as the discretization error. The proper relation is found to be $H=h^1^/^\alpha$where in the ideal case, $\alpha=4$. This means that in practice the coarse mesh is very coarse. To solve the coarse mesh problem it is shown that under a Hölder continuity assumption, a truncated and approximate generalized conjugate gradient method with search directions updated from an (inexact) Newton direction vector, converges globally, i.e. independent of the initial vector. Further, it is shown that the number of steps before the superlinear rate of convergence sets in is bounded, independent of the mesh parametr.

Application of preconditioned conjugate gradient-like methods to reactor kinetics

Several conjugate gradient-like (CG-like) methods are applied to solve the nonsymmetric linear systems of equations derived from the time-dependent two-dimensional two-energy-group neutron diffusion equations by a finite difference approximation. The CG-like methods are: the generalized conjugate residual method; the generalized conjugate gradient least-square method; the generalized minimal residual method (GMRES); the conjugate gradient square method; and a variant of bi-conjugate gradient method (Bi-CGSTAB). In order to accelerate these CG-like methods, several preconditioning techniques are investigated. Two of the preconditioners are based on pointwise incomplete factorization: the incomplete LU (ILU) and the modified incomplete LU (MILU) decompositions. Another two, based on the block tridiagonal structure of the coefficient matrix, are blockwise and modified blockwise incomplete factorizations, BILU and MBILU. Finally, the last two are the alternating-direction implicit and symmetric successive overrelaxation (SSOR) preconditioners, which are derived from the basic iterative schemes. Comparisons are made by using CG-like methods combined with different preconditioners to solve a sequence of time-step reactor transient problems. We also compare the numerical results of CG-like methods with that of the line SOR method and the hybrid SSOR-SI Krylov subspace method, which is the Chebyshev semi-iteration method with the preconditioner SSOR, and with the GMRES method to estimate the extreme eigenvalues for the acceleration parameters. Numerical tests indicate that preconditioned BI-CGSTAB with the preconditioner MBILU requires less CPU time and number of iterations than other methods in reactor kinetics computation. Another advantage shown is that the preconditioneed CG-like methods are less sensitive to the time-step size used than the Chebyshev semi-iteration method and line SOR method. Comparison between CG-like methods indicates that the CGS, Bi-CGSTAB and GMRES methods are, on average, better than the other methods in reactor kinetics computation. Moreover, numerical results indicate that a good preconditioner is more important than the choice of CG-like methods. Numerical results also show that the MILU decomposition based on the conventional row-sum criterion has difficulty yielding a convergent solution in reactor kinetics computation. An improved version of the MILU decomposition is introduced by using Stone's idea for reactor kinetics computation.

Simulations of the wind field in Athens with the nonhydrostatic mesoscale model MEMO

MEMO is a fully vectorized nonhydrostatic mesoscale model using terrain-following coordinates. The numerical solution is based on second-order discretization applied on a staggered grid which is allowed to be non-equidistant in all directions. Special care is taken that conservative propeprties are preserved within the discrete model equations. The discrete pressure equation is solved with a direct elliptic solver in conjuction with a generalized conjugate gradient method. Advective terms are treated with an explicit, monotonicity-preserving discretization scheme with only small implicit diffusion. Turbulent diffusion is described using an one-equation turbulence model, while at roughness height similarity theory is applied. An efficient scheme is applied to calculate radiative transfer. The algebraic surface heat budget equation and an one dimensional heat conduction equation are solved to obtain the surface temperature over land and the soil temperature. In the frame of the APSIS exercise A the model MEMO was used to simulate the mesoscale flow in the Athens basin on May 25, 1990. Being in satisfactory agreement with observations, the results indicate that weak pressure gradients accompanied by warm advection aloft may lead to stagnant conditions and thus to severe air pollution episodes in Athens.

Performance of certain iterative methods in solving implicit difference schemes for 2‐D Navier–Stokes equations

AbstractThe performance of three iterative methods, local SOR, preconditioned generalized conjugate gradient methods such as ORTHOMIN(1) and the GCG–LS method with preconditioning matrices derived by incomplete (modified and unmodified) pointwise factorizations of the matrix corresponding to implicit finite difference schemes for the 2‐D Navier–Stokes equations, is compared. The relative merits of these methods in terms of CPU time, average number of iterations with respect to the Reynolds number and discretization parameters are emphasized.

A Black Box Generalized Conjugate Gradient Solver with Inner Iterations and Variable-Step Preconditioning

The generalized conjugate gradient method proposed by Axelsson is studied in the case when a variable-step preconditioning is used. This can be the case when the preconditioned system is solved app...

GENERALIZED CONJUGATE GRADIENT METHOD FOR THE EFFICIENT SOLUTION OF THREE-DIMENSIONAL FLUID FLOW PROBLEMS

A generalized conjugate gradient (CG) method was examined for use in the numerical calculation of recirculating turbulent three-dimensional fluid flows. The proposed algorithm was found to be a very efficient solution procedure in a large number of practical applications. Furthermore, the results obtained from two other CG-like solution methods that are entirely free of recursion clearly indicate the advantages of the recommended partly recursive method even for vector computers.