Spectral Gradient Method Research Articles

A Spectral Gradient Projection Method for Sparse Signal Reconstruction in Compressive Sensing

In this paper, a new spectral gradient direction is proposed to solve the l1 -regularized convex minimization problem. The spectral parameter of the proposed method is computed as a convex combination of two existing spectral parameters of some conjugate gradient method. Moreover, the spectral gradient method is applied to the resulting problem at each iteration without requiring the Jacobian matrix. Furthermore, the proposed method is shown to have converge globally under some assumptions. Numerically, the proposed method is efficient and robust in terms of its quality in reconstructing sparse signal and low computational cost compared to the existing methods.

A note on the spectral gradient projection method for nonlinear monotone equations with applications

In this work, we provide a note on the spectral gradient projection method for solving nonlinear equations. Motivated by recent extensions of the spectral gradient method for solving nonlinear monotone equations with convex constraints, in this paper, we note that choosing the search direction as a convex combination of two different positive spectral coefficients multiplied with the residual vector is more efficient and robust compared with the standard choice of spectral gradient coefficients combined with the projection strategy of Solodov and Svaiter (A globally convergent inexact newton method for systems of monotone equations. In: Reformulation: Nonsmooth. Piecewise Smooth, Semismooth and Smoothing Methods, pp 355–369. Springer, 1998). Under suitable assumptions, the convergence of the proposed method is established. Preliminary numerical experiments show that the method is promising. In this paper, the proposed method was used to recover sparse signal and restore blurred image arising from compressive sensing.

A robust data-driven AVO inversion with logarithm absolute error loss function

Amplitude variation with offset (AVO) inversion is a widely used approach to obtain reliable estimates of elastic parameter. Tikhonov and total variation regularization are commonly used methods to address ill-posed problem of AVO inversion. However, these model-driven methods are only for special geological structure such as smoothness or blockiness. In this letter, a robust data-driven-based regularization method with logarithm absolute error loss function (DDI-Log) for AVO inversion is proposed. In DDI-Log, the information of well-log data and the complex geology are considered in a sparse representation framework. In pre-stack seismic data, outlier noise can negatively influence inversion results. Thus, different from the previous data-driven inversion based on $$L _2$$ norm loss function, we extend the logarithm absolute error function as the loss function. In the iteration, a new spectral PRP conjugate gradient method is used to solve the large-scale optimization problem. The synthetic data and field data tests illustrate that the proposed approach is robust against outlier noise and that the resolution and accuracy of the solutions are improved.

Subsampled Nonmonotone Spectral Gradient Methods

Abstract This paper deals with subsampled spectral gradient methods for minimizing finite sums. Subsample function and gradient approximations are employed in order to reduce the overall computational cost of the classical spectral gradient methods. The global convergence is enforced by a nonmonotone line search procedure. Global convergence is proved provided that functions and gradients are approximated with increasing accuracy. R-linear convergence and worst-case iteration complexity is investigated in case of strongly convex objective function. Numerical results on well known binary classification problems are given to show the effectiveness of this framework and analyze the effect of different spectral coefficient approximations arising from the variable sample nature of this procedure.

无约束优化问题的牛顿谱梯度法

无约束优化问题的牛顿谱梯度法

Exact spectral-like gradient method for distributed optimization

Since the initial proposal in the late 80s, spectral gradient methods continue to receive significant attention, especially due to their excellent numerical performance on various large scale applications. However, to date, they have not been sufficiently explored in the context of distributed optimization. In this paper, we consider unconstrained distributed optimization problems where n nodes constitute an arbitrary connected network and collaboratively minimize the sum of their local convex cost functions. In this setting, building from existing exact distributed gradient methods, we propose a novel exact distributed gradient method wherein nodes’ step-sizes are designed according to the novel rules akin to those in spectral gradient methods. We refer to the proposed method as Distributed Spectral Gradient method. The method exhibits R-linear convergence under standard assumptions for the nodes’ local costs and safeguarding on the algorithm step-sizes. We illustrate the method’s performance through simulation examples.

A family of spectral gradient methods for optimization

We propose a family of spectral gradient methods, whose stepsize is determined by a convex combination of the long Barzilai-Borwein (BB) stepsize and the short BB stepsize. Each member of the family is shown to share certain quasi-Newton property in the sense of least squares. The family also includes some other gradient methods as its special cases. We prove that the family of methods is $R$-superlinearly convergent for two-dimensional strictly convex quadratics. Moreover, the family is $R$-linearly convergent in the any-dimensional case. Numerical results of the family with different settings are presented, which demonstrate that the proposed family is promising.

New hybrid algorithm based on nonmonotone spectral gradient and simultaneous perturbation

In this paper, we introduce a new hybrid method called nonmonotone spectral gradient and simultaneous perturbation (NSGSP). It combines the advantages of nonmonotone spectral gradient (NSG), and simultaneous perturbation (SP) methods. The main idea of our approach is to use the simultaneous perturbation (SP) method in order to get a non expensive estimate of the gradient, and exploit the good properties of the nonmonotone spectral gradient (NSG) method in order to compute an efficient line search. Several numerical experiments are provided. The results indicate that the new method is effective and outperforms most of other popular methods.

Structured Two-Point Stepsize Gradient Methods for Nonlinear Least Squares

In this paper, we present two choices of structured spectral gradient methods for solving nonlinear least squares problems. In the proposed methods, the scalar multiple of identity approximation of the Hessian inverse is obtained by imposing the structured quasi-Newton condition. Moreover, we propose a simple strategy for choosing the structured scalar in the case of negative curvature direction. Using the nonmonotone line search with the quadratic interpolation backtracking technique, we prove that these proposed methods are globally convergent under suitable conditions. Numerical experiment shows that the methods are competitive with some recently developed methods.

A Synthetical Two-Component Model with Peakon Solutions: One More Bi-Hamiltonian Case

Abstract Compatible pairs of Hamiltonian operators for the synthetical two-component model of Xia, Qiao, and Zhou are derived systematically by means of the spectral gradient method. A new two-component system, which is bi-Hamiltonian, is presented. For this new system, the construction of its peakon solutions is considered.

Multi-step spectral gradient methods with modified weak secant relation for large scale unconstrained optimization

In this paper, we aim to propose some spectral gradient methods via variational technique under log-determinant norm. The spectral parameters satisfy the modified weak secant relations that inspired by the multistep approximation for solving large scale unconstrained optimization. An executable code is developed to test the efficiency of the proposed method with spectral gradient method using standard weak secant relation as constraint. Numerical results are presented which suggest a better performance has been achieved.

A Two-Phase Gradient Method for Quadratic Programming Problems with a Single Linear Constraint and Bounds on the Variables

We propose a gradient-based method for quadratic programming problems with a single linear constraint and bounds on the variables. Inspired by the GPCG algorithm for bound-constrained convex quadratic programming [J.J. Mor\'e and G. Toraldo, SIAM J. Optim. 1, 1991], our approach alternates between two phases until convergence: an identification phase, which performs gradient projection iterations until either a candidate active set is identified or no reasonable progress is made, and an unconstrained minimization phase, which reduces the objective function in a suitable space defined by the identification phase, by applying either the conjugate gradient method or a recently proposed spectral gradient method. However, the algorithm differs from GPCG not only because it deals with a more general class of problems, but mainly for the way it stops the minimization phase. This is based on a comparison between a measure of optimality in the reduced space and a measure of bindingness of the variables that are on the bounds, defined by extending the concept of proportioning, which was proposed by some authors for box-constrained problems. If the objective function is bounded, the algorithm converges to a stationary point thanks to a suitable application of the gradient projection method in the identification phase. For strictly convex problems, the algorithm converges to the optimal solution in a finite number of steps even in case of degeneracy. Extensive numerical experiments show the effectiveness of the proposed approach.

An extended nonmonotone line search technique for large-scale unconstrained optimization

In this paper, an extended nonmonotone line search is proposed to improve the efficiency of the existing line searches. This line search is first proved to be an extension of the classical line search rules. On the one hand, under mild assumptions, global convergence and R-linear convergence are established for the new line search rule. On the other hand, by numerical experiments, it is shown that the line search can integrate the advantages of the existing methods in searching for a suitable step-size. Combined with the spectral step-size, a class of spectral gradient algorithms are developed and employed to solve a large number of benchmark test problems from CUTEst. Numerical results show that the new line search is promising in solving large-scale optimization problems, and outperforms the other similar ones as it is combined with a spectral gradient method.

Optimal control of viscous Burgers equation via an adaptive nonmonotone Barzilai–Borwein gradient method

ABSTRACTAn adaptive nonmonotone spectral gradient method for the solution of distributed optimal control problem (OCP) for the viscous Burgers equation is presented in a black-box framework. Regarding the implicit function theorem, the OCP is transformed into an unconstrained nonlinear optimization problem (UNOP). For solving UNOP, an adaptive nonmonotone Barzilai–Borwein gradient method is proposed in which to make a globalization strategy, first an adaptive nonmonotone strategy which properly controls the degree of nonmonotonicity is presented and then is incorporated into an inexact line search approach to construct a more relaxed line search procedure. Also an adjoint technique is used to effectively evaluate the gradient. The low memory requirement and the guaranteed convergence property make the proposed method quite useful for large-scale OCPs. The efficiency of the presented method is supported by numerical experiments.

Nonsmooth spectral gradient methods for unconstrained optimization

To solve nonsmooth unconstrained minimization problems, we combine the spectral choice of step length with two well-established subdifferential-type schemes: the gradient sampling method and the simplex gradient method. We focus on the interesting case in which the objective function is continuously differentiable almost everywhere, and it is often not differentiable at minimizers. In the case of the gradient sampling method, we also present a simple differentiability test that allows us to use the exact gradient direction as frequently as possible, and to build a stochastic subdifferential direction only if the test fails. The proposed spectral gradient sampling method is combined with a monotone line search globalization strategy. On the other hand, the simplex gradient method is a direct search method that only requires function evaluations to build an approximation to the gradient direction. In this case, the proposed spectral simplex gradient method is combined with a suitable nonmonotone line search strategy. For both scenarios, we discuss convergence properties and present numerical results on a set of nonsmooth test functions. These numerical results indicate that using a spectral step length can improve the practical performance of both methods.

Sparse Signal Inversion with Impulsive Noise by Dual Spectral Projected Gradient Method

We consider sparse signal inversion with impulsive noise. There are three major ingredients. The first is regularizing properties; we discuss convergence rate of regularized solutions. The second is devoted to the numerical solutions. It is challenging due to the fact that both fidelity and regularization term lack differentiability. Moreover, for ill-conditioned problems, sparsity regularization is often unstable. We propose a novel dual spectral projected gradient (DSPG) method which combines the dual problem of multiparameter regularization with spectral projection gradient method to solve the nonsmooth l1+l1 optimization functional. We show that one can overcome the nondifferentiability and instability by adding a smooth l2 regularization term to the original optimization functional. The advantage of the proposed functional is that its convex duality reduced to a constraint smooth functional. Moreover, it is stable even for ill-conditioned problems. Spectral projected gradient algorithm is used to compute the minimizers and we prove the convergence. The third is numerical simulation. Some experiments are performed, using compressed sensing and image inpainting, to demonstrate the efficiency of the proposed approach.

Nonmonotone Spectral Gradient Method for l_1-regularized Least Squares

In the paper, we investigate a linear constraint optimization reformulation to a more general form of the l_1 regularization problem and give some good properties of it. We first show that the equivalence between the linear constraint optimization problem and the l_1 regularization problem. Second, the KKT point of the linear constraint problem always exists since the constraints are linear; we show that the half constraints must be active at any KKT point. In addition, we show that the KKT points of the linear constraint problem are the same as the stationary points of the l_1 regularization problem. Based on the linear constraint optimization problem, we propose a nonomotone spectral gradient method and establish its global convergence. Numerical experiments with compressive sense problems show that our approach is competitive with several known methods for standard l_2-l_1 problem.

A trust region spectral method for large-scale systems of nonlinear equations

The spectral gradient method is one of the most effective methods for solving large-scale systems of nonlinear equations. In this paper, we propose a new trust region spectral method without gradient. The trust region technique is a globalization strategy in our method. The global convergence of the proposed algorithm is proved. The numerical results show that our new method is more competitive than the spectral method of La Cruz et al. (Math. Comput. 75(255):1429-1448, 2006) for large-scale nonlinear equations.

Multivariate spectral DY-type projection method for convex constrained nonlinear monotone equations

In this paper, we consider a multivariate spectral DY-type projection method for solving nonlinear monotone equations with convex constraints. The search direction of the proposed method combines those of the multivariate spectral gradient method and DY conjugate gradient method. With no need for the derivative information, the proposed method is very suitable to solve large-scale nonsmooth monotone equations. Under appropriate conditions, we prove the global convergence and R-linear convergence rate of the proposed method. The preliminary numerical results also indicate that the proposed method is robust and effective.

Hybrid nonmonotone spectral gradient method for the unconstrained minimization problem

Nonmonotone spectral gradient (NSG) techniques are considered for unconstrained optimization of differentiable functions. They combine a nonmonotone step length strategy, that is based on the Grippo–Lampariello–Lucidi nonmonotone line search, with the spectral gradient choice of steplength. In this paper, we present a hybrid nonmonotone spectral gradient method. In each iteration after finding an acceptable point by NSG method, we employ an extrapolation step to find a even better point by using particle swarm optimization. Through specific assumptions, we prove the global convergence of our proposed algorithm