Truncated Conjugate Gradient Method Research Articles

Fast convergence of trust-regions for non-isolated minima via analysis of CG on indefinite matrices

AbstractTrust-region methods (TR) can converge quadratically to minima where the Hessian is positive definite. However, if the minima are not isolated, then the Hessian there cannot be positive definite. The weaker Polyak–Łojasiewicz (PŁ) condition is compatible with non-isolated minima, and it is enough for many algorithms to preserve good local behavior. Yet, TR with an exact subproblem solver lacks even basic features such as a capture theorem under PŁ. In practice, a popular inexact subproblem solver is the truncated conjugate gradient method (tCG). Empirically, TR-tCG exhibits superlinear convergence under PŁ. We confirm this theoretically. The main mathematical obstacle is that, under PŁ, at points arbitrarily close to minima, the Hessian has vanishingly small, possibly negative eigenvalues. Thus, tCG is applied to ill-conditioned, indefinite systems. Yet, the core theory underlying tCG is that of CG, which assumes a positive definite operator. Accordingly, we develop new tools to analyze the dynamics of CG in the presence of small eigenvalues of any sign, for the regime of interest to TR-tCG.

Analysis of the Truncated Conjugate Gradient Method for Linear Matrix Equations

The matrix-oriented version of the conjugate gradient (CG) method can be used to approximate the solution to certain linear matrix equations. To limit memory consumption, low-rank reduction of the factored iterates is often employed, possibly leading to disruption of the regular convergence behavior. We analyze the properties of the method in the matrix regime and identify the quantities that are responsible for early termination, usually stagnation, when truncation is in effect. Moreover, we illustrate relations between CG and a projection technique directly applied to the same matrix equation.

Riemannian optimization model order reduction method for general linear port-Hamiltonian systems

Abstract This paper presents a Riemannian optimal model order reduction method for general linear stable port-Hamiltonian systems based on the Riemannian trust-region method. We consider the $\mathcal{H}_2$ optimal model order reduction problem of the general linear port-Hamiltonian systems. The problem is formulated as an optimization problem on the product manifold, which is composed of the set of skew symmetric matrices, the manifold of the positive definite matrices, the manifold of the positive semidefinite matrices with fixed rank and the Euclidean space. To solve the optimal problem, the Riemannian geometry of the product manifold is given. Moreover, the Riemannian gradient and the Riemannian Hessian of the objective function are derived. Furthermore, we propose the Riemannian trust-region method for the optimization problem and introduce the truncated conjugate gradient method to solve the trust-region subproblem. Finally, the numerical experiments illustrate the efficiency of the proposed method.

Truncated trust region method for nonlinear inverse problems and application in full-waveform inversion

We present a general truncated trust region method to solve large-scale nonlinear inverse problems. The truncated trust region method can serve as an implicit regularization method, and it can take advantage of the second-order derivative information of the misfit functional. The convergence of the truncated trust region method is provided under some smoothness assumptions. To improve the computational efficiency and reduce the memory requirement, we develop a second-order adjoint-state method to efficiently estimate matrix–vector products, and solve the trust region subproblem using the truncated conjugate gradient method with the matrix-free strategy. The full-waveform inversion problem is used to test the numerical performance of the proposed method. Numerical results show that the truncated trust region method can perform better than conventional methods (e.g. nonlinear conjugate gradient method and L-BFGS) for highly nonlinear inverse problems, in terms of inverting resolution.

HT-AWGM: a hierarchical Tucker–adaptive wavelet Galerkin method for high-dimensional elliptic problems

This paper is concerned with the construction, analysis, and realization of a numerical method to approximate the solution of high-dimensional elliptic partial differential equations. We propose a new combination of an adaptive wavelet Galerkin method (AWGM) and the well-known hierarchical tensor (HT) format. The arising HT-AWGM is adaptive both in the wavelet representation of the low-dimensional factors and in the tensor rank of the HT representation. The point of departure is an adaptive wavelet method for the HT format using approximate Richardson iterations and an AWGM for elliptic problems. HT-AWGM performs a sequence of Galerkin solves based upon a truncated preconditioned conjugate gradient (PCG) algorithm in combination with a tensor-based preconditioner. Our analysis starts by showing convergence of the truncated conjugate gradient method. The next step is to add routines realizing the adaptive refinement. The resulting HT-AWGM is analyzed concerning convergence and complexity. We show that the performance of the scheme asymptotically depends only on the desired tolerance with convergence rates depending on the Besov regularity of low-dimensional quantities and the low-rank tensor structure of the solution. The complexity in the ranks is algebraic with powers of four stemming from the complexity of the tensor truncation. Numerical experiments show the quantitative performance.

A derivative-free algorithm for non-linear optimization with linear equality constraints

ABSTRACTWe propose a derivative-free algorithm for solving linear equality constrained non-linear optimization problems, named LECOA. In each iteration of the algorithm, the objective function is approximated by a quadratic model constructed from interpolation points. The choice of the points leaves some degree of freedom in the model taken up by minimizing the Frobenius norm of the change to the Hessian matrix of the model. The new iterate is generally generated by minimizing the model in a trust-region using a null space truncated conjugate gradient method. Numerical results are presented which show that the proposed algorithm competes against some algorithms in the literature. Experiments show that starting with the point that minimizes the infinity norm subject to the linear equality constraints gives excellent results. A limit-type global convergence of the proposed algorithm is proved under some reasonable assumptions.

A fast Total Variation-based iterative algorithm for digital breast tomosynthesis image reconstruction

In this work, we propose a fast iterative algorithm for the reconstruction of digital breast tomosynthesis images. The algorithm solves a regularization problem, expressed as the minimization of the sum of a least-squares term and a weighted smoothed version of the Total Variation regularization function. We use a Fixed Point method for the solution of the minimization problem, requiring the solution of a linear system at each iteration, whose coefficient matrix is a positive definite approximation of the Hessian of the objective function. We propose an efficient implementation of the algorithm, where the linear system is solved by a truncated Conjugate Gradient method. We compare the Fixed Point implementation with a fast first order method such as the Scaled Gradient Projection method, that does not require any linear system solution. Numerical experiments on a breast phantom widely used in tomographic simulations show that both the methods recover microcalcifications very fast while the Fixed Point is more efficient in detecting masses, when more time is available for the algorithm execution.

Bioluminescence Tomography–Guided Radiation Therapy for Preclinical Research

In preclinical radiation research, it is challenging to localize soft tissue targets based on cone beam computed tomography (CBCT) guidance. As a more effective method to localize soft tissue targets, we developed an online bioluminescence tomography (BLT) system for small-animal radiation research platform (SARRP). We demonstrated BLT-guided radiation therapy and validated targeting accuracy based on a newly developed reconstruction algorithm. The BLT system was designed to dock with the SARRP for image acquisition and to be detached before radiation delivery. A 3-mirror system was devised to reflect the bioluminescence emitted from the subject to a stationary charge-coupled device (CCD) camera. Multispectral BLT and the incomplete variables truncated conjugate gradient method with a permissible region shrinking strategy were used as the optimization scheme to reconstruct bioluminescent source distributions. To validate BLT targeting accuracy, a small cylindrical light source with high CBCT contrast was placed in a phantom and also in the abdomen of a mouse carcass. The center of mass (CoM) of the source was recovered from BLT and used to guide radiation delivery. The accuracy of the BLT-guided targeting was validated with films and compared with the CBCT-guided delivery. In vivo experiments were conducted to demonstrate BLT localization capability for various source geometries. Online BLT was able to recover the CoM of the embedded light source with an average accuracy of 1 mm compared to that with CBCT localization. Differences between BLT- and CBCT-guided irradiation shown on the films were consistent with the source localization revealed in the BLT and CBCT images. In vivo results demonstrated that our BLT system could potentially be applied for multiple targets and tumors. The online BLT/CBCT/SARRP system provides an effective solution for soft tissue targeting, particularly for small, nonpalpable, or orthotopic tumor models.

Incomplete variables truncated conjugate gradient method for signal reconstruction in compressed sensing

Compressed sensing (CS) has stirred great interests in many fields of science, due to its ability to capture most information of compressible signals at a rate significantly below the Nyquist rate. Reconstructing the signal from random measurements is an important topic in CS. In this paper, a new algorithm— Incomplete variables Truncated Conjugate Gradient method (ITCG) is proposed to reconstruct the signal by solving a programming with ℓ1 norm. By adjusting the parameters of ITCG, two specific algorithms are presented, i.e. ITCG-vs for very sparse reconstruction and ITCG-nvs for not very sparse reconstruction. To make full use of the sparse nature of signals, ITCG can reconstruct them efficiently. The experiments show that the two algorithms of ITCG (especially ITCG-nvs) are much faster than competing methods in sparse reconstruction. In addition, it has been shown that ITCG-vs can converge after finite iterations under some decent conditions.

Cone beam x‐ray luminescence computed tomography: A feasibility study

The appearance of x-ray luminescence computed tomography (XLCT) opens new possibilities to perform molecular imaging by x ray. In the previous XLCT system, the sample was irradiated by a sequence of narrow x-ray beams and the x-ray luminescence was measured by a highly sensitive charge coupled device (CCD) camera. This resulted in a relatively long sampling time and relatively low utilization of the x-ray beam. In this paper, a novel cone beam x-ray luminescence computed tomography strategy is proposed, which can fully utilize the x-ray dose and shorten the scanning time. The imaging model and reconstruction method are described. The validity of the imaging strategy has been studied in this paper. In the cone beam XLCT system, the cone beam x ray was adopted to illuminate the sample and a highly sensitive CCD camera was utilized to acquire luminescent photons emitted from the sample. Photons scattering in biological tissues makes it an ill-posed problem to reconstruct the 3D distribution of the x-ray luminescent sample in the cone beam XLCT. In order to overcome this issue, the authors used the diffusion approximation model to describe the photon propagation in tissues, and employed the sparse regularization method for reconstruction. An incomplete variables truncated conjugate gradient method and permissible region strategy were used for reconstruction. Meanwhile, traditional x-ray CT imaging could also be performed in this system. The x-ray attenuation effect has been considered in their imaging model, which is helpful in improving the reconstruction accuracy. First, simulation experiments with cylinder phantoms were carried out to illustrate the validity of the proposed compensated method. The experimental results showed that the location error of the compensated algorithm was smaller than that of the uncompensated method. The permissible region strategy was applied and reduced the reconstruction error to less than 2 mm. The robustness and stability were then evaluated from different view numbers, different regularization parameters, different measurement noise levels, and optical parameters mismatch. The reconstruction results showed that the settings had a small effect on the reconstruction. The nonhomogeneous phantom simulation was also carried out to simulate a more complex experimental situation and evaluated their proposed method. Second, the physical cylinder phantom experiments further showed similar results in their prototype XLCT system. With the discussion of the above experiments, it was shown that the proposed method is feasible to the general case and actual experiments. Utilizing numerical simulation and physical experiments, the authors demonstrated the validity of the new cone beam XLCT method. Furthermore, compared with the previous narrow beam XLCT, the cone beam XLCT could more fully utilize the x-ray dose and the scanning time would be shortened greatly. The study of both simulation experiments and physical phantom experiments indicated that the proposed method was feasible to the general case and actual experiments.

A feasible direction method for image restoration

In this work, a feasible direction method is proposed for computing the regularized solution of image restoration problems by simply using an estimate of the noise present on the data. The problem is formulated as an optimization problem with one quadratic constraint. The proposed method computes a feasible search direction by inexactly solving a trust region subproblem with the truncated Conjugate Gradient method of Steihaug. The trust region radius is adjusted to maintain feasibility and a line-search globalization strategy is employed. The global convergence of the method is proved. The results of image denoising and deblurring are presented in order to illustrate the effectiveness and efficiency of the proposed method.

A truncated conjugate gradient method with an inexact Gauss-Newton technique for solving nonlinear systems

In this paper, a truncated conjugate gradient method with an inexact Gauss-Newton technique is proposed for solving nonlinear systems. The iterative direction is obtained by the conjugate gradient method solving the inexact Gauss-Newton equation. Global convergence and local superlinear convergence rate of the proposed algorithm are established under some reasonable conditions. Finally, some numerical results are presented to illustrate the effectiveness of the proposed algorithm.

Block relaxation and majorization methods for the nearest correlation matrix with factor structure

We propose two numerical methods, namely the alternating block relaxation method and the alternating majorization method, for the problem of nearest correlation matrix with factor structure, which is highly nonconvex. In the block relaxation method, the subproblem is of the standard trust region problem, which is solved by Steighaug's truncated conjugate gradient method or by the exact trust region method. In the majorization method, the subproblem has a closed-form solution. We then apply the majorization method to the case where nonnegative factors are required. The numerical results confirm that the proposed methods work quite well and are competitive against the best available methods.

Sparse reconstruction for quantitative bioluminescence tomography based on the incomplete variables truncated conjugate gradient method

In this paper, we present an incomplete variables truncated conjugate gradient (IVTCG) method for bioluminescence tomography (BLT). Considering the sparse characteristic of the light source and insufficient surface measurement in the BLT scenarios, we combine a sparseness-inducing (ℓ1 norm) regularization term with a quadratic error term in the IVTCG-based framework for solving the inverse problem. By limiting the number of variables updated at each iterative and combining a variable splitting strategy to find the search direction more efficiently, it obtains fast and stable source reconstruction, even without a priori information of the permissible source region and multispectral measurements. Numerical experiments on a mouse atlas validate the effectiveness of the method. In vivo mouse experimental results further indicate its potential for a practical BLT system.

Using truncated conjugate gradient method in trust-region method with two subproblems and backtracking line search

A trust-region method with two subproblems and backtracking line search for solving unconstrained optimization is proposed. At every iteration, we use the truncated conjugate gradient method or its variation to solve one of the two subproblems approximately. Backtracking line search is carried out when the trust-region trail step fails. We show that this method have the same convergence properties as the traditional trust-region method based on the truncated conjugate gradient method. Numerical results show that this method is as reliable as the traditional one and more efficient in respect of iterations, CPU time and evaluations. Mathematical subject classification: Primary: 65K05; Secondary: 90C30.

The bounded smooth reformulation and a trust region algorithm for semidefinite complementarity problems

In this paper, we present a new merit function for the semidefinite complementarity problem (SDCP) by extending the bounded smooth reformulation for variational inequality problems. We prove that the merit function has a bounded level set and any stationary point of it is a global minimizer without the assumption of monotonicity. Moreover, we present a trust region algorithm for solving the minimization problem with semidefinite constraints. The trust region subproblem is solved by the truncated conjugate gradient method and the global convergence is established even without requiring the existence of an accumulation point of the generated sequence.

A Truncated Newton Method for the Solution of Large-Scale Inequality Constrained Minimization Problems

A new active set Newton-type algorithm for the solution of inequality constrained minimization problems is proposed. The algorithm possesses the following favorable characteristics: (i) global convergence under mild assumptionss (ii) superlinear convergence of primal variables without strict complementaritys (iii) a Newton-type direction computed by means of a truncated conjugate gradient method. Preliminary computational results are reported to show viability of the approach in large scale problems having only a limited number of constraints.

Constrained optimal control of Navier–Stokes flow by semismooth Newton methods

We propose and analyze a semismooth Newton-type method for the solution of a pointwise constrained optimal control problem governed by the time-dependent incompressible Navier–Stokes equations. The method is based on a reformulation of the optimality system as an equivalent nonsmooth operator equation. We analyze the flow control problem and prove q-superlinear convergence of the method. In the numerical implementation, adjoint techniques are combined with a truncated conjugate gradient method. Numerical results are presented that support our theoretical results and confirm the viability of the approach.

A trust region-CG algorithm for deblurring problem in atmospheric image reconstruction

In this paper we solve large scale ill-posed problems, particularly the image restoration problem in atmospheric imaging sciences, by a trust region-CG algorithm. Image restoration involves the removal or minimization of degradation (blur, clutter, noise, etc.) in an image using a priori knowledge about the degradation phenomena. Our basic technique is the so-called trust region method, while the subproblem is solved by the truncated conjugate gradient method, which has been well developed for well-posed problems. The trust region method, due to its robustness in global convergence, seems to be a promising way to deal with ill-posed problems.

Computational methods for a large-scale inverse problem arising in atmosphericoptics

The inverse problem of interest is the reconstruction of an astronomical objectand wavefront aberrations due to atmospheric turbulence from a sequence of shorttime exposure (speckle) phase diversity images obtained from a ground-basedtelescope. A regularized least squares approach is taken, and two numericaltechniques are applied to the resulting unconstrained optimization problem. Thefirst technique is the limited memory BFGS method with line search globalization.The second is a Newton/trust region iteration in which the trust regionsubproblem is solved using a truncated conjugate gradient method. Anumerical comparison based on real data is presented, and the roles ofpreconditioning and cost functional reduction (elimination of the object) areexamined.