Conjugate Gradient Type Methods Research Articles

SIGNAL RECOVERY WITH CONSTRAINED MONOTONE NONLINEAR EQUATIONS THROUGH AN EFFECTIVE THREE-TERM CONJUGATE GRADIENT METHOD

In this paper, we introduce a three-term conjugate gradient-type projection method for solving constrained monotone nonlinear equations. In this method, we firstly undertake the transformation of the relative matrices proposed by Yao and Ning. Secondly, we obtain the new relative matrices involving two parameters. Subsequently, we construct a relationship for the two parameters via the quasi-Newton equation and obtain the parameters by simplifying maximum eigenvalue of the new relative matrices. Finally, combining the three-term conjugate gradient method with projection technique, we establish an efficient three-term conjugate gradient-type projection algorithm. Meanwhile, we also give some theoretical analysis about the global convergence and R-linear convergence of the proposed algorithm under quite reasonable technical assumptions. Performance comparisons show that our proposed method is competitive and efficient for solving large-scale nonlinear monotone equations with convex constraints. Furthermore, the presented algorithm is also applied to recovery of a sparse signal in compressive sensing, and obtain practical, efficient and competitive performance in comparing with state-of-the-art algorithms.

A Simple Conjugate Gradient Type Method for Solving Large-scale Systems of Nonlinear Equations

Abstract Quasi Newton’s method is among the most promising algorithm for solving systems of nonlinear equations. In this family, Broyden’s method update is the famous. This paper presents a simple conjugate gradient (CG) method for solving large-scale systems of nonlinear equations via memoryless Broyden’s approach. The attractive attribute of this method is due to its low memory requirements, global convergence properties and simple implementation. Under suitable conditions, the proposed method converges globally. Numerical performance of the proposed method are compared with the well-known conjugate gradients (CG) methods using benchmarks problems. The report demonstrate that the proposed method is reliable, efficient and competitive.

A new non-linear conjugate gradient algorithm for destructive cure rate model and a simulation study: illustration with negative binomial competing risks

In this paper, we propose a new estimation methodology based on a projected non-linear conjugate gradient (PNCG) algorithm with an efficient line search technique. We develop a general PNCG algorithm for a survival model incorporating a proportion cure under a competing risks setup, where the initial number of competing risks are exposed to elimination after an initial treatment (known as destruction). In the literature, expectation maximization (EM) algorithm has been widely used for such a model to estimate the model parameters. Through an extensive Monte Carlo simulation study, we compare the performance of our proposed PNCG with that of the EM algorithm and show the advantages of our proposed method. Through simulation, we also show the advantages of our proposed methodology over other optimization algorithms (including other conjugate gradient type methods) readily available as R software packages. To show these, we assume the initial number of competing risks to follow a negative binomial distribution although our general algorithm allows one to work with any competing risks distribution. Finally, we apply our proposed algorithm to analyze a well-known melanoma data.

A diagonal PRP-type projection method for convex constrained nonlinear monotone equations

<p style='text-indent:20px;'>Iterative methods for nonlinear monotone equations do not require the differentiability assumption on the residual function. This special property of the methods makes them suitable for solving large-scale nonsmooth monotone equations. In this work, we present a diagonal Polak-Ribi<inline-formula><tex-math id="M1">\begin{document}$ \grave{e} $\end{document}</tex-math></inline-formula>re-Polyak (PRP) conjugate gradient-type method for solving large-scale nonlinear monotone equations with convex constraints. The search direction is a combine form of a multivariate (<i>diagonal</i>) spectral method and a modified PRP conjugate gradient method. Proper safeguards are devised to ensure positive definiteness of the diagonal matrix associated with the search direction. Based on Lipschitz continuity and monotonicity assumptions the method is shown to be globally convergent. Numerical results are presented by means of comparative experiments with recently proposed multivariate spectral Dai-Yuan-type (J. Ind. Manag. Optim. 13 (2017) 283-295) and Wei-Yao-Liu-type (Int. J. Comput. Math. 92 (2015) 2261-2272) conjugate gradient methods.

A Nonmonotone Hestenes ans Stiefel Conjugate Gradient Algorithm for Nonsmooth Convex Optimization

Here, we propose a practical method for solving nonsmooth convex problems by using conjugate gradient type methods. We present a modified HS conjugate gradient method, as one of the most remarkable methods to solve smooth and large-scale optimization problems. In the case that we have a nonsmooth convex problem, by way of the Moreau-Yosida regularization, we convert the nonsmooth objective function to a smooth function and then we use our method, by making use of a nonmonotone line search, for solving a nonsmooth convex optimization problem. We prove that our algorithm converges to an optimal solution under standard condition. Our algorithm inherits the performance of HS conjugate gradient method.

Weighted Conjugate Gradient-Type Methods for Solving Quadrature Discretization of Fredholm Integral Equations of the First Kind

A variant of conjugate gradient-type methods, called weighted conjugate gradient (WCG), is given to solve quadrature discretization of various first-kind Fredholm integral equations with continuous kernels. The WCG-type methods use a new inner product instead of the Euclidean one arising from discretization of $$L^2$$ -inner product by the quadrature formula. On this basis, the proposed algorithms generate a sequence of vectors which are approximations of solution at the quadrature points. Numerical experiments on a few model problems are used to illustrate the performance of the new methods compared to the CG-type methods.

Multigrid optimization for DNS-based optimal control in turbulent channel flows

The use of PDE-constrained optimization techniques in combination with transient three-dimensional turbulent flow simulations such as Direct Numerical Simulation (DNS) or Large-Eddy Simulation (LES) involves large computational cost and memory resources. To date, the minimization of a DNS-based cost functional is typically achieved by applying classical single-grid gradient-based iterative methods of quasi-Newton or non-linear conjugate gradient type. In the current study, a multigrid optimization (MG/OPT) strategy is investigated in order to speed up gradient-based algorithms designed for large scale optimization problems. The method employs a hierarchy of optimization problems defined on different representation levels. It aims to reduce the computational resources associated with the cost functional improvement on the finest level. We apply the MG/OPT method in the context of direct numerical simulations of a fully developed channel flow problem. The performance of the multigrid optimization technique is compared against the single-grid optimization method in terms of equivalent function and gradient evaluations. Also the influence of the optimization problem properties and algorithmic parameters are investigated. It is found that, in some cases, the MG/OPT method accelerates the single-grid damped L-BFGS method by a factor of four.

English

We consider solving complex symmetric linear systems with multiple right-hand sides. We assume that the coefficient matrix has indefinite real part and positive definite imaginary part. We propose a new block conjugate gradient type method based on the Schur complement of a certain 2-by-2 real block form. The algorithm of the proposed method consists of building blocks that involve only real arithmetic with real symmetric matrices of the original size. We also present the convergence property of the proposed method and an efficient algorithmic implementation. In numerical experiments, we compare our method to a complex-valued direct solver, and a preconditioned and nonpreconditioned block Krylov method that uses complex arithmetic.

Improved Schur complement preconditioners for block-Toeplitz systems with small size blocks

In this paper, we employ the preconditioned conjugate gradient method with the Improved Schur complement preconditioners for Hermitian positive definite block-Toeplitz systems with small size blocks. Schur complement preconditioners have been proved to be an effective method for such block-Toeplitz systems (Ching et al. 2007). The modification is based on Taylor expansion approximation. We prove that the matrices preconditioned by improved Schur preconditioners have more clustered spectra compared to that of the Schur complement preconditioners. Hence, preconditioned conjugate gradient type methods will converge faster. Numerical examples are given to demonstrate the efficiency of the proposed method.

THE HYPERBOLIC QUADRATIC EIGENVALUE PROBLEM

The hyperbolic quadratic eigenvalue problem (HQEP) was shown to admit Courant–Fischer type min–max principles in 1955 by Duffin and Cauchy type interlacing inequalities in 2010 by Veselić. It can be regarded as the closest analog (among all kinds of quadratic eigenvalue problem) to the standard Hermitian eigenvalue problem (among all kinds of standard eigenvalue problem). In this paper, we conduct a systematic study on the HQEP both theoretically and numerically. On the theoretical front, we generalize Wielandt–Lidskii type min–max principles and, as a special case, Fan type trace min/max principles and establish Weyl type and Wielandt–Lidskii–Mirsky type perturbation results when an HQEP is perturbed to another HQEP. On the numerical front, we justify the natural generalization of the Rayleigh–Ritz procedure with existing principles and our new optimization principles, and, as consequences of these principles, we extend various current optimization approaches—steepest descent/ascent and nonlinear conjugate gradient type methods for the Hermitian eigenvalue problem—to calculate a few extreme eigenvalues (of both positive and negative type). A detailed convergence analysis is given for the steepest descent/ascent methods. The analysis reveals the intrinsic quantities that control convergence rates and consequently yields ways of constructing effective preconditioners. Numerical examples are presented to demonstrate the proposed theory and algorithms.

GENERIC APPROXIMATE SPARSE INVERSE MATRIX TECHNIQUES

During the last decades explicit preconditioning methods have gained interest among the scientific community, due to their efficiency for solving large sparse linear systems in conjunction with Krylov subspace iterative methods. The effectiveness of explicit preconditioning schemes relies on the fact that they are close approximants to the inverse of the coefficient matrix. Herewith, we propose a Generic Approximate Sparse Inverse (GenASPI) matrix algorithm based on ILU(0) factorization. The proposed scheme applies to matrices of any structure or sparsity pattern unlike the previous dedicated implementations. The new scheme is based on the Generic Approximate Banded Inverse (GenAbI), which is a banded approximate inverse used in conjunction with Conjugate Gradient type methods for the solution of large sparse linear systems. The proposed GenASPI matrix algorithm, is based on Approximate Inverse Sparsity patterns, derived from powers of sparsified matrices and is computed with a modified procedure based on the GenAbI algorithm. Finally, applicability and implementation issues are discussed and numerical results along with comparative results are presented.

A Polak–Ribière–Polyak method for solving large-scale nonlinear systems of equations and its global convergence

A derivative-free conjugate gradient type method for solving large-scale nonlinear systems of equations is presented. In the iterative method, the search direction is based on the Polak–Ribière–Polyak (PRP) conjugate gradient method, and the steplength is determined by a suitable nonmonotone line search. Under appropriate conditions, the global convergence of the proposed method is established. The method is suitable to large-scale problems for the lower storage requirement. It is shown from the numerical results that the method is practically effective.

On the convergence properties of the modified Polak--Ribiere--Polyak method with the standard Armijo line search

Zhang et al. [IMA J. Numer. Anal., 26 (2006) 629--640] proposed a modified Polak--Ribiere--Polyak method for non-convex optimization and proved its global convergence with some backtracking type line search. We further study its convergence properties. Under the standard Armijo line search condition, we show that the modified Polak--Ribiere--Polyak method has better global convergence property and locally \(R\)-linear convergence rate for non-convex minimization. Some preliminary numerical results are also reported to show its efficiency. References C. Li, A conjugate gradient type method for the nonnegative constraints optimization problems, J. Appl. Math. , Volume 2013, Article ID 986317, 6 pages. D. Li and B. Tian, $n$-step quadratic convergence of the mprp method with a restart strategy, J. Comput. Appl. Math. , 235 (2011) 4978--4990. J. J. More, B. S. Garbow and K. H. Hillstrom, Testing unconstrained optimization software, ACM Trans. Math. Softw. , 7 (1981) 17--41. E. Polak and G. Ribiere, Note sur la convergence de methodes de directions conjuguees, Rev. Fr. Inform. Rech. Oper. , 16 (1969) 35--43. B. T. Polyak, The conjugate gradient method in extreme problems, USSR Comput. Math. Math. Phys. , 9 (1969) 94--112. L. Zhang, W. Zhou and D. Li, A descent modified Polak--Ribiere--Polyak conjugate gradient method and its global convergence, IMA J. Numer. Anal. , 26 (2006) 629--640. H. Zhu, Y. Xiao and S. Wu, Large sparse signal recovery by conjugate gradient algorithm based on smoothing technique, Comput. Math. Appl. , 66 (2013) 24--32.

Performance of Preconditioned Linear Solvers Based on Minimum Residual for Complex Symmetric Linear Systems

Fast computation of linear systems is essential for reducing the elapsed time when using finite element analysis. The incomplete Cholesky conjugate orthogonal conjugate gradient method is widely used as a linear solver for complex symmetric systems derived from the edge-based FEM in the frequency domain. On the other hand, the performance of the preconditioned minimized residual method based on the three-term recurrence (MRTR) formula of the conjugate gradient-type method has been demonstrated on various symmetric sparse linear systems obtained from edge-based FEM formulated in the magnetostatic and time domain. This paper shows for the first time the performance of the preconditioned conjugate orthogonal MRTR method applied to complex symmetric linear systems.

Block conjugate gradient type methods for the approximation of bilinear form [formula omitted

In this paper, we consider computing the approximation of block bilinear form CHA−1B, where square matrix A is large sparse, and B and C are rectangular matrices with the same size. We propose block conjugate gradient (CG) type methods for the approximation of CHA−1B, in which approximation Xk of linear systems AX=B does not need to be computed explicitly and only the approximation ηk of CHA−1B, which is mathematically equivalent to CHXk, will be calculated instead. Numerical results show the effectiveness of our proposed methods.

A NOTE ON PARALLEL FINITE DIFFERENCE APPROXIMATE INVERSE PRECONDITIONING ON MULTICORE SYSTEMS USING POSIX THREADS

New parallel computational techniques are introduced for the parallelization of explicit finite difference (FD) approximate inverse matrix methods, based on Portable Operating System Interface for UniX (POSIX) threads, for multicore systems. Parallelization of the Optimized Banded Generalized Approximate Inverse Matrix (OBGAIM) algorithm is achieved based on the concept of the "fish bone" approach with the use of a thread pool pattern. Theoretical estimates on speedups and efficiency are also presented. Additionally, new parallel computational techniques are proposed for the parallelization of explicit preconditioned biconjugate conjugate gradient type methods, based on POSIX threads, for multicore systems. For parallelization purposes a replication of the parallel explicit preconditioned biconjugate conjugate gradient-STAB (PEPBICG-STAB) method was assigned on each created thread, with different index bands and with proper synchronization points on inner products and matrix-vector multiplications. Theoretical estimates on speedups and efficiency are also presented. Finally, numerical results for the performance of the Parallel Fish Bone OBGAIM (PaFiBo-OBGAIM) algorithm and the PEPBICG-STAB method for solving classical two-dimensional boundary value problems on multicore computer systems are presented, which are favorably compared to corresponding results from multiprocessor systems. The implementation issues of the proposed method are also discussed using POSIX threads on multicore systems.

A modified Polak–Ribière–Polyak conjugate gradient algorithm for nonsmooth convex programs

The conjugate gradient (CG) method is one of the most popular methods for solving smooth unconstrained optimization problems due to its simplicity and low memory requirement. However, the usage of CG methods is mainly restricted to solving smooth optimization problems so far. The purpose of this paper is to present efficient conjugate gradient-type methods to solve nonsmooth optimization problems. By using the Moreau–Yosida regulation (smoothing) approach and a nonmonotone line search technique, we propose a modified Polak–Ribière–Polyak (PRP) CG algorithm for solving a nonsmooth unconstrained convex minimization problem. Our algorithm possesses the following three desired properties. (i) The search direction satisfies the sufficient descent property and belongs to a trust region automatically; (ii) the search direction makes use of not only the gradient information but also the function value information; and (iii) the algorithm inherits an important property of the well-known PRP method: the tendency to turn towards the steepest descent direction if a small step is generated away from the solution, preventing a sequence of tiny steps from happening. Under standard conditions, we show that the algorithm converges globally to an optimal solution. Numerical experiment shows that our algorithm is effective and suitable for solving large-scale nonsmooth unconstrained convex optimization problems.

A Conjugate Gradient Type Method for the Nonnegative Constraints Optimization Problems

We are concerned with the nonnegative constraints optimization problems. It is well known that the conjugate gradient methods are efficient methods for solving large-scale unconstrained optimization problems due to their simplicity and low storage. Combining the modified Polak-Ribière-Polyak method proposed by Zhang, Zhou, and Li with the Zoutendijk feasible direction method, we proposed a conjugate gradient type method for solving the nonnegative constraints optimization problems. If the current iteration is a feasible point, the direction generated by the proposed method is always a feasible descent direction at the current iteration. Under appropriate conditions, we show that the proposed method is globally convergent. We also present some numerical results to show the efficiency of the proposed method.

Conjugate gradient type methods for the nondifferentiable convex minimization

We proposed implementable conjugate gradient type methods for solving a possibly nondifferentiable convex minimization problem by converting the original objective function to a once continuously differentiable function by way of the Moreau–Yosida regularization. The proposed methods make use of approximate function and gradient values of the Moreau–Yosida regularization instead of the corresponding exact values. The global convergence is established under mild conditions.

On the GPGPU parallelization issues of finite element approximate inverse preconditioning

During the past decades, explicit finite element approximate inverse preconditioning methods have been extensively used for efficiently solving sparse linear systems on multiprocessor systems. The effectiveness of explicit approximate inverse preconditioning schemes relies on the use of efficient preconditioners that are close approximants to the coefficient matrix and are fast to compute in parallel. New parallel computational techniques are proposed for the parallelization of the Optimized Banded Generalized Approximate Inverse Finite Element Matrix ( OBGAIFEM) algorithm, based on the concept of the “fish bone” computational approach, and for the Explicit Preconditioned Conjugate Gradient type methods on a General Purpose Graphics Processing Unit ( GPGPU). The proposed parallel methods have been implemented using Compute Unified Device Architecture (CUDA) developed by NVIDIA. Finally, numerical results for the performance of the finite element explicit approximate inverse preconditioning for solving characteristic two dimensional boundary value problems on a massive multiprocessor interface on a GPU are presented. The CUDA implementation issues of the proposed methods are also discussed.