Gradient-type Methods Research Articles

Streaming First‐Order Optimization Methods With Momentum for Dominant Eigenspace Estimation

ABSTRACTIn this article, we consider the dominant eigenspace recovery for PCA in the streaming scenario, and design several new algorithms for this problem by incorporating various momentum schemes into two classical gradient based streaming eigensolvers, which emerges as a frequently‐used trick in improving gradient type methods. Theoretically, we establish the convergence guarantees of the proposed first‐order momentum methods, and prove their decreasing variance under favorable conditions. Extensive numerical experiments are conducted to corroborate the result that momentum does yield variance reduction, and also highlight the advantages of lowering the stepsize sensitivity. In addition, our methods with appropriately chosen momentum yield a dramatically enhanced performance in terms of both the convergence speed and accuracy.

Comparison of the Computational Efficiency of Gradient-Type Methods in Optimal Control Problems

Comparison of the Computational Efficiency of Gradient-Type Methods in Optimal Control Problems

An augmented Lagrangian approach for cardinality constrained minimization applied to variable selection problems

To solve convex constrained minimization problems, that also include a cardinality constraint, we propose an augmented Lagrangian scheme combined with alternating projection ideas. Optimization problems that involve a cardinality constraint are NP-hard mathematical programs and typically very hard to solve approximately. Our approach takes advantage of a recently developed and analyzed continuous formulation that relaxes the cardinality constraint. Based on that formulation, we solve a sequence of smooth convex constrained minimization problems, for which we use projected gradient-type methods. In our setting, the convex constraint region can be written as the intersection of a finite collection of convex sets that are easy and inexpensive to project. We apply our approach to a variety of over and under determined constrained linear least-squares problems, with both synthetic and real data that arise in variable selection, and demonstrate its effectiveness.

The continuous stochastic gradient method: part I–convergence theory

In this contribution, we present a full overview of the continuous stochastic gradient (CSG) method, including convergence results, step size rules and algorithmic insights. We consider optimization problems in which the objective function requires some form of integration, e.g., expected values. Since approximating the integration by a fixed quadrature rule can introduce artificial local solutions into the problem while simultaneously raising the computational effort, stochastic optimization schemes have become increasingly popular in such contexts. However, known stochastic gradient type methods are typically limited to expected risk functions and inherently require many iterations. The latter is particularly problematic, if the evaluation of the cost function involves solving multiple state equations, given, e.g., in form of partial differential equations. To overcome these drawbacks, a recent article introduced the CSG method, which reuses old gradient sample information via the calculation of design dependent integration weights to obtain a better approximation to the full gradient. While in the original CSG paper convergence of a subsequence was established for a diminishing step size, here, we provide a complete convergence analysis of CSG for constant step sizes and an Armijo-type line search. Moreover, new methods to obtain the integration weights are presented, extending the application range of CSG to problems involving higher dimensional integrals and distributed data.

An approximate gradient-type method for nonlinear symmetric equations with convex constraints

Based on the projection operator and the pseudo-monotone property, an approximate gradient-type method is proposed to solve nonlinear symmetric equations with convex constraints. The proposed method does not need any precise gradient or Jacobian matrix information, and only uses some approximate substitutions based on the symmetric property of equations. Under some appropriate assumptions, the global convergence and R-linear convergence rate are proved. Numerical experiments show that the proposed method is stable and effective for the test problems.

Two Convergent Primal–Dual Hybrid Gradient Type Methods for Convex Programming with Linear Constraints

Two Convergent Primal–Dual Hybrid Gradient Type Methods for Convex Programming with Linear Constraints

New A hybrid conjugate gradient Fletcher-Reeves and Polak-Ribiere algorithm for unconstrained optimization

In this Research we developed a New Hybrid method of conjugate gradient type, this Method depends basically on combining Fletcher-Reeves and Polak-Ribiere algorithms by using spectral direction conjugate algorithm, which is developed by Yang Z & Kairong W [19]. The developed method becomes converged by assuming some hypothesis. The numerical results show the efficiency of the developed method for solving test Unconstrained Nonlinear Optimization problems.

New A hybrid Hestenes-Stiefel and Dai-Yuan conjugate gradient algorithms for unconstrained optimization

In this Research we developed a New Hybrid method of conjugate gradient type, this Method depends basically on combining Hestenes-Stiefel and Dai-Yuan algorithms by using spectral direction conjugate algorithm, which is developed by Yang Z & Kairong W [19]. The developed method becomes converged by assuming some hypothesis. The numerical results show the efficiency of the developed method for solving test Unconstrained Nonlinear Optimization problems.

SIGNAL RECOVERY WITH CONSTRAINED MONOTONE NONLINEAR EQUATIONS THROUGH AN EFFECTIVE THREE-TERM CONJUGATE GRADIENT METHOD

In this paper, we introduce a three-term conjugate gradient-type projection method for solving constrained monotone nonlinear equations. In this method, we firstly undertake the transformation of the relative matrices proposed by Yao and Ning. Secondly, we obtain the new relative matrices involving two parameters. Subsequently, we construct a relationship for the two parameters via the quasi-Newton equation and obtain the parameters by simplifying maximum eigenvalue of the new relative matrices. Finally, combining the three-term conjugate gradient method with projection technique, we establish an efficient three-term conjugate gradient-type projection algorithm. Meanwhile, we also give some theoretical analysis about the global convergence and R-linear convergence of the proposed algorithm under quite reasonable technical assumptions. Performance comparisons show that our proposed method is competitive and efficient for solving large-scale nonlinear monotone equations with convex constraints. Furthermore, the presented algorithm is also applied to recovery of a sparse signal in compressive sensing, and obtain practical, efficient and competitive performance in comparing with state-of-the-art algorithms.

Trace minimization method via penalty for linear response eigenvalue problems

<p style='text-indent:20px;'>In various applications, such as the computation of energy excitation states of electrons and molecules, and the analysis of interstellar clouds, the linear response eigenvalue problem, which is a special type of the Hamiltonian eigenvalue problem, is frequently encountered. However, traditional eigensolvers may not be applicable to this problem owing to its inherently large scale. In fact, we are usually more interested in computing some of the smallest positive eigenvalues. To this end, a trace minimum principle optimization model with orthogonality constraint has been proposed. On this basis, we propose an unconstrained surrogate model called trace minimization via penalty, and we establish its equivalence with the original constrained model, provided that the penalty parameter is larger than a certain threshold. By avoiding the orthogonality constraint, we can use a gradient-type method to solve this model. Specifically, we use the gradient descent method with Barzilai–Borwein step size. Moreover, we develop a restarting strategy for the proposed algorithm whereby higher accuracy and faster convergence can be achieved. This is verified by preliminary experimental results.</p>

Avoiding bad steps in Frank-Wolfe variants

The study of Frank-Wolfe (FW) variants is often complicated by the presence of different kinds of “good” and “bad” steps. In this article, we aim to simplify the convergence analysis of specific variants by getting rid of such a distinction between steps, and to improve existing rates by ensuring a non-trivial bound at each iteration. In order to do this, we define the Short Step Chain (SSC) procedure, which skips gradient computations in consecutive short steps until proper conditions are satisfied. This algorithmic tool allows us to give a unified analysis and converge rates in the general smooth non convex setting, as well as a linear convergence rate under a Kurdyka-Łojasiewicz (KL) property. While the KL setting has been widely studied for proximal gradient type methods, to our knowledge, it has never been analyzed before for the Frank-Wolfe variants considered in the paper. An angle condition, ensuring that the directions selected by the methods have the steepest slope possible up to a constant, is used to carry out our analysis. We prove that such a condition is satisfied, when considering minimization problems over a polytope, by the away step Frank-Wolfe (AFW), the pairwise Frank-Wolfe (PFW), and the Frank-Wolfe method with in face directions (FDFW).

A Frobenius Norm Regularization Method for Convolutional Kernel Tensors in Neural Networks.

The convolutional neural network is a very important model of deep learning. It can help avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network if the singular values of the Jacobian of a layer are bounded around 1 in the training process. We propose a new Frobenius norm penalty function for a convolutional kernel tensor to let the singular values of the corresponding transformation matrix be bounded around 1. We show how to carry out the gradient-type methods. This provides a potentially useful regularization method for the weights of convolutional layers.

Convergence of Inverse Volatility Problem Based on Degenerate Parabolic Equation

Based on the theoretical framework of the Black–Scholes model, the convergence of the inverse volatility problem based on the degenerate parabolic equation is studied. Being different from other inverse volatility problems in classical parabolic equations, we introduce some variable substitutions to convert the original problem into an inverse principal coefficient problem in a degenerate parabolic equation on a bounded area, from which an unknown volatility can be recovered and deficiencies caused by artificial truncation can be solved. Based on the optimal control framework, the problem is transformed into an optimization problem and the existence of the minimizer is established, and a rigorous mathematical proof is given for the convergence of the optimal solution. In the end, the gradient-type iteration method is applied to obtain the numerical solution of the inverse problem, and some numerical experiments are performed.

Градиентный метод с неточным оракулом для задач композитной невыпуклой оптимизации

In this paper, we develop new first-order method for composite non-convex minimization problems with simple constraints and inexact oracle. The objective function is given as a sum of `hard', possibly non-convex part, and `simple' convex part. Informally speaking, oracle inexactness means that, for the `hard' part, at any point we can approximately calculate the value of the function and construct a quadratic function, which approximately bounds this function from above. We give several examples of such inexactness: smooth non-convex functions with inexact Holder-continuous gradient, functions given by auxiliary uniformly concave maximization problem, which can be solved only approximately. For the introduced class of problems, we propose a gradient-type method, which allows to use different proximal setup to adapt to geometry of the feasible set, adaptively chooses controlled oracle error, allows for inexact proximal mapping. We provide convergence rate for our method in terms of the norm of generalized gradient mapping and show that, in the case of inexact Holder-continuous gradient, our method is universal with respect to Holder parameters of the problem. Finally, in a particular case, we show that small value of the norm of generalized gradient mapping at a point means that a necessary condition of local minimum approximately holds at that point.

Generalized Fractional Algebraic Linear System Solvers

This paper is devoted to the numerical computation of algebraic linear systems involving several matrix power functions; that is finding x solution to \(\sum _{\alpha \in \mathbb {R}}A^{\alpha }x=b\). These systems will be referred to as Generalized Fractional Algebraic Linear Systems (GFALS). In this paper, we derive several gradient methods for solving these very computationally complex problems, which themselves require the solution to intermiediate standard Fractional Algebraic Linear Systems (FALS) \(A^{\alpha }x=b\), with \(\alpha \in \mathbb {R_+}\). The latter usually require the solution to many classical linear systems \(Ax=b\). We also show that in some cases, the solution to a GFALS problem can be obtained as the solution to a first-order hyperbolic system of conservation laws. We also discuss the connections between this PDE-approach and gradient-type methods. The convergence analysis is addressed and some numerical experiments are proposed to illustrate and compare the methods which are proposed in this paper.

DATA-DRIVEN REGULARIZATION OF INVERSE PROBLEM FOR SEIR-HCD MODEL OF COVID-19 PROPAGATION IN NOVOSIBIRSK REGION

Abstract The inverse problem for SEIR-HCD model of COVID-19 propagation in Novosi- birsk region described by system of seven nonlinear ordinary differential equations (ODE) is numerical investigated. The inverse problem consists in identification of coefficients of ODE system (infection rate, portions of infected, hospitalized, mortality cases) and some ini- tial conditions (initial number of asymptomatic and symptomatic infectious) by additional measurements about daily diagnosed, critical and mortality cases of COVID-19. Due to ill-posedness of inverse problem the regularization is applied based on usage of additional information about antibodies IgG to COVID-19 and detailed mortality statistics. The inverse problem is reduced to a minimization problem of misfit function. We apply data-driven ap- proach based on combination of global (OPTUNA software) and gradient-type methods for solving the minimization problem. The numerical results show that adding new information and detailed statistics increased the forecasting scenario in 2 times.

On stochastic Kaczmarz type methods for solving large scale systems of ill-posed equations

In this article we investigate a family of stochastic gradient type methods for solving systems of linear ill-posed equations. The method under consideration is a stochastic version of the projective Landweber–Kaczmarz method in Leitão and Svaiter (2016 Inverse Problems 32 025004) (see also Leitão and Svaiter (2018 Numer. Funct. Anal. Optim. 39 1153–80)). In the case of exact data, mean square convergence to zero of the iteration error is proven. In the noisy data case, we couple our method with an a priori stopping rule and characterize it as a regularization method for solving systems of linear ill-posed operator equations. Numerical tests are presented for two linear ill-posed problems: (i) a Hilbert matrix type system with over 108 equations; (ii) a big data linear regression problem with real data. The obtained results indicate superior performance of the proposed method when compared with other well-established random iterations. Our preliminary investigation indicates that the proposed iteration is a promising alternative for computing stable approximate solutions of large scale systems of linear ill-posed equations.

Iterative regularization for constrained minimization formulations of nonlinear inverse problems

In this paper we study the formulation of inverse problems as constrained minimization problems and their iterative solution by gradient or Newton type methods. We carry out a convergence analysis in the sense of regularization methods and discuss applicability to the problem of identifying the spatially varying diffusivity in an elliptic PDE from different sets of observations. Among these is a novel hybrid imaging technology known as impedance acoustic tomography, for which we provide numerical experiments.

Numerical method for solving the piecewise constant source inverse problem of an elliptic equation from a partial boundary observation data

We present results of numerical investigation of the source term recovery in a boundary value problem for an elliptic equation. An additional information about the solution is considered as its normal derivative taken on a part of the boundary. Such source inverse problem is related with inverse gravimetry problem of determining an inhomogeneity from gravitational potential anomalies on the Earth’s surface. We propose an iterative method for numerical recovery of the source term on the base of minimization of the observation residual by a gradient type method. The numerical implementation is based on finite element approximation using the FEniCS scientific computing platform and the dolfin-adjoint package. The capabilities of the developed computational algorithm are illustrated by results of numerical solutions of two dimensional test problems.

Numerical algorithms for solving an elliptic optimal control problem with a power-law nonlinearity

The paper is devoted to the development and analysis of approximation-iteration algorithms based on the method of grids and the method of lines for solving an elliptic optimal control problem with a power-law nonlinearity. For the numerical solution of the main boundary value problem and the adjoint one, the second order of accuracy difference schemes are applied using the implicit method of simple iteration. Computational schemes of the method of lines for solving the above-mentioned elliptic boundary value problems are implemented in combination with the shooting method for the approximate solution of boundary value problems for the corresponding ordinary differential equations systems arising in the considered domain after lattice approximation. To minimize the objective functional, well-known gradient-type methods (gradient projection and conditional gradient methods) of constrained optimization are used. The essence of the proposed approximation-iteration approach consists in replacing the original extremal problem with a sequence of grid problems that approximate it on a set of refining grids, and applying an iterative gradient-type method to each of the "approximate" extremal problems. In this case, we propose to construct only a few approximations to the solution for each of the "approximate" problems and to take the last of these approximations, using piecewise linear interpolation, as the initial approximation in the iterative process for the next "approximate" problem. The sequence of the corresponding piecewise linear interpolants is considered as a sequence of approximations to the solution of the original extremal problem. The paper discusses the theoretical foundations of this combined approach, as well as its advantages over traditional methods using the example of solving a model optimal control problem