Conjugate Gradient Descent Method Research Articles

A three-term conjugate gradient descent method with some applications

The stationary point of optimization problems can be obtained via conjugate gradient (CG) methods without the second derivative. Many researchers have used this method to solve applications in various fields, such as neural networks and image restoration. In this study, we construct a three-term CG method that fulfills convergence analysis and a descent property. Next, in the second term, we employ a Hestenses-Stiefel CG formula with some restrictions to be positive. The third term includes a negative gradient used as a search direction multiplied by an accelerating expression. We also provide some numerical results collected using a strong Wolfe line search with different sigma values over 166 optimization functions from the CUTEr library. The result shows the proposed approach is far more efficient than alternative prevalent CG methods regarding central processing unit (CPU) time, number of iterations, number of function evaluations, and gradient evaluations. Moreover, we present some applications for the proposed three-term search direction in image restoration, and we compare the results with well-known CG methods with respect to the number of iterations, CPU time, as well as root-mean-square error (RMSE). Finally, we present three applications in regression analysis, image restoration, and electrical engineering.

Multi-sensor remote sensing image alignment based on fast algorithms

Abstract Remote sensing image technology to the ground has important guiding significance in disaster assessment and emergency rescue deployment. In order to realize the fast automatic registration of multi-sensor remote sensing images, the remote sensing image block registration idea is introduced, and the image reconstruction is processed by using the conjugate gradient descent (CGD) method. The scale-invariant feature transformation (SIFT) algorithm is improved and optimized by combining the function-fitting method. By this way, it can improve the registration accuracy and efficiency of multi-sensor remote sensing images. The results show that the average peak signal-to-noise ratio of the image processed by the CGD method is 25.428. The average root mean square value is 17.442. The average image processing time is 6.093 s. These indicators are better than the passive filter algorithm and the gradient descent method. The average accuracy of image registration of the improved SIFT registration method is 96.37%, and the average image registration time is 2.14 s. These indicators are significantly better than the traditional SIFT algorithm and speeded-up robust features algorithm. It is proved that the improved SIFT registration method can effectively improve the accuracy and operation efficiency of multi-sensor remote sensing image registration methods. The improved SIFT registration method effectively solves the problems of low accuracy and long time consumption of traditional multi-sensor remote sensing image fast registration methods. While maintaining high registration accuracy, it improves the image registration speed and provides technical support for a rapid disaster assessment after major disasters such as earthquakes and floods. And it has an important value for the development of the efficient post-disaster rescue deployment.

Riemannian conjugate gradient descent method for fixed multi rank third-order tensor completion

The goal of tensor completion is to fill in missing entries of a partially known tensor under a low-rank constraint. In this paper, we study low rank third-order tensor completion problems by using Riemannian optimization methods on the smooth manifold. Here the tensor rank is defined to be a multi-rank under the Discrete Cosine Transform-related transform tensor-tensor product. With suitable incoherence conditions on the underlying low multi-rank rank tensor, we show that the proposed Riemannian optimization method is guaranteed to converge to the underlying low multi-rank tensor with a high probability. Number of sampling entries required for convergence are also derived. Numerical examples of synthetic data and real data sets are reported to demonstrate that the performance of the proposed method is better than that of tensor-based method using the Tucker-rank model in terms of computational time, and that of the matrix-based completion method in terms of number of sampling entries.

Low-rank traffic matrix completion with marginal information

Accurate spatio-temporal traffic data is crucial to intelligent transportation systems. Missing traffic data is an important problem to solve. Low-rank matrix completion provides an effective way to find the missing data. The completion aims to obtain a low-rank matrix that can approximate the known entries as far as possible. Meanwhile, some linear constraint marginal information of the matrix can also be observed in the real application. In this paper, we utilize such marginal information to largely improve the performance of common matrix completion algorithms and propose an alternating direction method of multipliers (ADMM) and conjugate gradient descent method (CGD) based SoftImpute alternative least square (ALS) algorithm. We analyze their convergence rates and prove that the model can always converge to a first-order stationary point. We also utilize ADMM and CGD to largely accelerate the subproblem and make its complexity of each iteration at the same level as the popular SoftImpute-ALS matrix completion algorithm. Furthermore, this algorithm can be used in distributed computation, suitable for large-scale problems. In the numerical experiments, we demonstrate its outstanding matrix completion performance and high speed in several traffic matrix datasets.

Efficient implementation of x-ray ghost imaging based on a modified compressive sensing algorithm

Towards efficient implementation of x-ray ghost imaging (XGI), efficient data acquisition and fast image reconstruction together with high image quality are preferred. In view of radiation dose resulted from the incident x-rays, fewer measurements with sufficient signal-to-noise ratio (SNR) are always anticipated. Available methods based on linear and compressive sensing algorithms cannot meet all the requirements simultaneously. In this paper, a method based on a modified compressive sensing algorithm with conjugate gradient descent method (CGDGI) is developed to solve the problems encountered in available XGI methods. Simulation and experiments demonstrate the practicability of CGDGI-based method for the efficient implementation of XGI. The image reconstruction time of sub-second implicates that the proposed method has the potential for real-time XGI.

Global optimization with a new method based on the gradient direction vector without constrained derivable objective functions

Gradient descent is a popular method in optimization. This method can be used in the case of a derivable and without constraint function. But it encountered several obstacles (the choice of initial point and the type of objective function, the solution, global or local). In this work, we propose an optimization method that is used to calculate the global optimization of mono-objective optimization functions that can be derived without constraints. This method is inspired by the gradient descent method. Their principle is to use a new entity based on the gradient direction instead of gradient. Unlike the gradient method and other methods inspired by gradient descent (conjugate gradient method and Newton method) where the choice of initial point is made in a random way. In our case, the choice of initial point is fixed at the beginning and also the form of the objective function curve is not a problem. Finally, we will end by applying our method to some examples (nonlinear function, quadratic function (not-convex, convex).

A gradient based numerical approach to simulate elastic rod

An efficient numerical method based on a Sobolev gradient is presented to find the equilibrium shapes of an elastic rod resistant to bending and twisting. A Sobolev preconditioning operator is introduced based on the energy of the elastic rod that preconditions the standard conjugate gradient method and results in a speedy convergence. It outperforms the usual conjugate gradient and steepest descent methods without requiring expensive calculation of the dense Hessian associated with the total energy of the rod. The method has also been validated to predict the Michell’s instability in a closed rod with a twist.

Two-versions of descent conjugate gradient methods for large-scale unconstrained optimization

<p>The conjugate gradient methods are noted to be exceedingly valuable for solving large-scale unconstrained optimization problems since it needn't the storage of matrices. Mostly the parameter conjugate is the focus for conjugate gradient methods. The current paper proposes new methods of parameter of conjugate gradient type to solve problems of large-scale unconstrained optimization. A Hessian approximation in a diagonal matrix form on the basis of second and third-order Taylor series expansion was employed in this study. The sufficient descent property for the proposed algorithm are proved. The new method was converged globally. This new algorithm is found to be competitive to the algorithm of fletcher-reeves (FR) in a number of numerical experiments.</p>

Making Landsat 5, 7 and 8 reflectance consistent using MODIS nadir-BRDF adjusted reflectance as reference

The Landsat satellite data applications have been broadened by the improvement in data consistency, e.g., advances in calibration and atmospheric correction algorithms. However, there are still inter-sensor inconsistencies of surface reflectance caused by differences in sensor spectral response and residuals in calibration and atmospheric correction. Such inconsistencies tend to entangle with the land surface change, especially for the subtle changes such as vegetation responses to climate change. We propose a Bayesian approach to reduce inter-sensor inconsistencies by making Landsat 5, 7 and 8 reflectance consistent with MODIS nadir BRDF-adjusted reflectance (NBAR). The Bayesian MODIS-consistent Landsat reflectance should be similar to the MODIS NBAR when it is degraded to the MODIS resolution considering the point spread function (PSF). The MODIS NBAR PSF is modelled as a Gaussian function with empirically derived and band specific parameters. The relationships between the MODIS-consistent Landsat reflectance and the two available reflectance (i.e., the MODIS NBAR and observed Landsat reflectance) are built in the Bayesian framework. The Bayesian MODIS-consistent Landsat reflectance is then derived using the computationally efficient conjugate gradient descent method. The algorithm was applied to Landsat 5, 7 and 8 surface reflectance over five study areas in Brazil, China, France, Russia and USA, and compared with a linear regression-based reflectance adjustment method. Visual difference was evident in the MODIS NBAR and the observed Landsat reflectance and the relative root-mean-squared differences (rRMSD) ranged from 3.16% to 67.72% and R2 from 0.037 to 0.934. Both the Bayesian and linearly MODIS-consistent Landsat reflectance are visually more consistent with the MODIS NBAR. The average rRMSD and R2 improvements of the Bayesian MODIS-consistent Landsat reflectance over the observed Landsat reflectance were 21.36% and 0.19, 3.38% and 0.11, 5.25% and 0.07, 3.14% and 0.11, 3.49% and 0.08, and 5.72% and 0.09 for the blue, green, red, near infrared, and two shortwave infrared bands, respectively. The linearly MODIS-consistent Landsat reflectance had less rRMSD improvements: 18.26%, 1.46%, 3.02%, 1.86%, 1.97% and 3.36% for the six bands respectively, and had no R2 improvements. The linear regression derived using the whole image pixels has low performance in adjusting some extreme reflectance values that are not well represented in the image. In contrast, the Bayesian method derives the reflectance image that is consistent to MODIS at pixel level.

Cosmology without window functions: Quadratic estimators for the galaxy power spectrum

Conventional algorithms for galaxy power-spectrum estimation measure the true spectrum convolved with a survey window function, which, for parameter inference, must be compared with a similarly convolved theory model. In this work, we directly estimate the unwindowed power spectrum multipoles using quadratic estimators akin to those introduced in the late 1990s. Under Gaussian assumptions, these are optimal and free from the leading-order effects of pixelization and non-Poissonian shot noise. They may be straightforwardly computed given the survey dataset and a suite of simulations of known cosmology. We implement the pixel-based maximum-likelihood estimator and a simplification based on the Feldman-Kaiser-Peacock weighting scheme, both of which can be computed via FFTs and conjugate-gradient descent methods. Furthermore, the estimators allow direct computation of spectrum coefficients in an arbitrary linear compression scheme, without needing to first bin the statistic. Applying the technique to a subset of the BOSS DR12 galaxies, we find that the pixel-based quadratic estimators give statistically consistent power spectra, compressed coefficients, and cosmological parameters to those obtained with the usual windowed approaches. Because of the sample's low number density and compact window function, the optimal weighting scheme gives little improvement over the simplified form; this may change for dense surveys or those focusing on primordial non-Gaussianity. The technique is shown to be efficient and robust, and shows significant potential for measuring the windowless power spectrum and bispectrum in the presence of weak non-Gaussianity.

Deterministic control of stochastic reaction-diffusion equations

<p style='text-indent:20px;'>We consider the control of semilinear stochastic partial differential equations (SPDEs) via deterministic controls. In the case of multiplicative noise, existence of optimal controls and necessary conditions for optimality are derived. In the case of additive noise, we obtain a representation for the gradient of the cost functional via adjoint calculus. The restriction to deterministic controls and additive noise avoids the necessity of introducing a backward SPDE. Based on this novel representation, we present a probabilistic nonlinear conjugate gradient descent method to approximate the optimal control, and apply our results to the stochastic Schlögl model. We also present some analysis in the case where the optimal control for the stochastic system differs from the optimal control for the deterministic system.</p>

Learning temporal-spatial consistency correlation filter for visual tracking

Discriminant correlation filter-based tracking approaches, which adopt a circular shift operator on the tracking target object (the only accurate positive sample) to obtain training data and rely on the potential sample periodic extension hypothesis that enables model training and detection, can be efficiently accomplished through FFT. However, real background information is not modeled during the total learning process. The background-aware correlation filter (BACF) tracking algorithm uses a binary matrix to acquire real positive and negative samples using a dense sampling method to model the targets appearance. However, the BACF algorithm does not consider temporal and spatial consistency information, and when a target undergoes an abrupt change, the learned correlation filter will drift to the background. To solve this problem, in this paper, we introduce temporal and spatial consistency constraints into the baseline BACF framework and propose a learning temporal-spatial consistency correlation filter (TSCF) tracking algorithm. This enables the correlation filter to learn to adapt to the appearance of mutation between successive frames. The temporal consistency constraint smooths the multi-channel correlation filter in the time series, and the spatial consistency constraint smooths the multi-channel correlation filter in spatial distribution, thus making the energy distribution more uniform of the correlation filter learned. In this paper, the TSCF model has closed solutions and the conjugate gradient descent method is used to approximate the optimal solution of a system of closed solutions. The optimization process can then be transformed into the Fourier domain using cyclic matrix properties to quickly obtain a solution, which effectively reduces the cost of calculating large matrices. In this paper, our TSCF algorithm increases distance precision by 5.5$%$ and raises the AUC by 4.3$%$ compared to the baseline BACF algorithm on the TB100 public database. The distance precision achieves 0.879 and the AUC reaches 0.663 on the TB100 database making use of only hand-crafted features. The TSCF algorithm proposed in this paper can be applied to challenging conditions such as short time occlusion, out-of-plane rotation, in-plane rotation, and so on, thus demonstrating its robustness and effectiveness.

A globally convergent hybrid conjugate gradient method with strong Wolfe conditions for unconstrained optimization

In this paper, we develop a new hybrid conjugate gradient method that inherits the features of the Liu and Storey (LS), Hestenes and Stiefel (HS), Dai and Yuan (DY) and Conjugate Descent (CD) conjugate gradient methods. The new method generates a descent direction independently of any line search and possesses good convergence properties under the strong Wolfe line search conditions. Numerical results show that the proposed method is robust and efficient.

Super-Resolution mmWave Channel Estimation for Generalized Spatial Modulation Systems

We propose super-resolution multiple-input multiple-output channel estimators for generalized spatial modulation-based millimeter-wave systems. Utilizing the inherent spatial sparsity of millimeter-wave channels, channel estimation problem is formulated using atomic norm minimization that enhances sparsity in the continuous angles of arrival and departure. Both pilot-assisted and data-aided channel estimators are developed, with the former one formulated as a convex problem and the latter one as a nonconvex problem. To efficiently solve these formulated channel estimation problems, we develop nonconvex factorization-based conjugate gradient descent methods to restrict search space into low-rank matrices. Superior channel estimation performance of the proposed algorithms compared to the state-of-the-art compressed-sensing-based estimators is demonstrated by simulation results.

Conjugate gradient-based Takagi-Sugeno fuzzy neural network parameter identification and its convergence analysis

Model identification is divided into two parts: structure identification and parameter identification, and the parameter identification is actually an optimization process. For improving the optimization performance, in this paper, we firstly present a novel conjugate gradient descent method with a modified Armijo-type line search technique to train a Takagi-Sugeno fuzzy neural network model. Numerical simulations are implemented to demonstrate the efficiency of the proposed algorithm. According to the experimental comparisons that are evaluated over 15 classification and 3 regression problems, the advantages of the given method are superior to its another two counterparts. To complement the simulation results and help in establishing a robust fuzzy neural network model, we strictly prove two deterministic convergent behaviors of the presented algorithm, i.e., weak and strong convergence results. They indicate the gradient of the target function with respect to network weights converges to zero and the parameter sequence approaches a fixed optimal point, respectively.

Multi-Target Position and Velocity Estimation Using OFDM Communication Signals

In this paper, we consider a passive radar system that estimates the positions and velocities of multiple moving targets by using OFDM signals transmitted by a totally un-coordinated and un-synchronizated illuminator and multiple receivers. It is assumed that data demodulation is performed separately based on the direct-path signal, and the error-prone estimated data symbols are made available to the passive radar receivers, which estimate the positions and velocities of the targets in two stages. First, we formulate a problem of joint estimation of the delay-Doppler of reflectors and the demodulation errors, by exploiting two types of sparsities of the system, namely, the numbers of reflectors (i.e., targets and clutters) and demodulation errors are both small. This problem is non-convex and a conjugate gradient descent method is proposed to solve it. Then in the second stage we determine the positions and velocities of targets based on the estimated delay-Doppler in the first stage. For the second stage, two methods are proposed: the first is based on numerically solving a set of nonlinear equations, while the second is based on the neural network, which is more efficient. The performance of the proposed algorithms is evaluated through extensive simulations.

A class of descent conjugate gradient methods for solving optimization problems

A class of descent conjugate gradient methods for solving optimization problems

Semi-Blind Millimeter-Wave Channel Estimation Using Atomic Norm Minimization

Exploiting the inherent sparsity of millimeter-wave channels, the channel estimation problem is formulated as an atomic norm minimization that enhances sparsity in the continuous angles of arrival and departure. A semi-blind channel estimator is developed to track the time-varying channel dynamics, which is formulated as a non-convex problem. To solve the formulated channel estimation problem, we develop a computationally efficient conjugate gradient descent method based on non-convex factorization which restricts the search space to low-rank matrices. Simulation results are presented to illustrate the superior channel estimation performance of the proposed algorithms compared with the existing compressed-sensing-based estimators with finely quantized angle grids.

Predicting compressive strength of blended cement concrete with ANNs

Predicting the compressive strength of concrete is important to assess the load-carrying capacity of a structure. However, the use of blended cements to accrue the technical, economic and environmental benefits has increased the complexity of prediction models. Artificial Neural Networks (ANNs) have been used for predicting the compressive strength of ordinary Portland cement concrete, i.e., concrete produced without the addition of supplementary cementing materials. In this study, models to predict the compressive strength of blended cement concrete prepared with a natural pozzolan were developed using regression models and single- and 2-phase learning ANNs. Back-propagation (BP), Levenberg-Marquardt (LM) and Conjugate Gradient Descent (CGD) methods were used for training the ANNs. A 2-phase learning algorithm is proposed for the first time in this study for predictive modeling of the compressive strength of blended cement concrete. The output of these predictive models indicates that the use of a 2-phase learning algorithm will provide better results than the linear regression model or the traditional single-phase ANN models.

Optimal response to non-equilibrium disturbances under truncated Burgers–Hopf dynamics

We model and compute the average response of truncated Burgers–Hopf dynamics to finite perturbations away from the Gibbs equipartition energy spectrum using a dynamical optimization framework recently conceptualized in a series of papers. Non-equilibrium averages are there approximated in terms of geodesic paths in probability space that ‘best-fit’ the Liouvillean dynamics over a family of quasi-equilibrium trial densities. By recasting the geodesic principle as an optimal control problem, we solve numerically for the non-equilibrium responses using an augmented Lagrangian, non-linear conjugate gradient descent method. For moderate perturbations, we find an excellent agreement between the optimal predictions and the direct numerical simulations of the truncated Burgers–Hopf dynamics. In this near-equilibrium regime, we argue that the optimal response theory provides an approximate yet predictive counterpart to fluctuation-dissipation identities.