First-order Optimization Methods Research Articles

The complexity of first-order optimization methods from a metric perspective

The complexity of first-order optimization methods from a metric perspective

ЗАДАЧА ОПТИМАЛЬНОГО КЕРУВАННЯ ТОЧКАМИ НАВАНТАЖУВАННЯ ТА ЇХНІМИ ФУНКЦІЯМИ РЕАКЦІЇ ДЛЯ ПАРАБОЛІЧНОГО РІВНЯННЯ

We consider the problem of optimal control of loading points and the corresponding reaction functions described by a loaded parabolic equation. Optimality conditions for control actions are obtained. The objective functional gradient formulas contained in these conditions are used in the algorithm for numerically solving the problem of optimization of loading points and reaction functions based on first-order optimization methods. The results of numerical experiments are provided. Keywords: distributed-parameter system, loaded differential equation, necessary optimality condition, functional gradient.

On the Rod Heating Problem by Sources on the Class of Zonal Controls Using the Current and Past State at Measurement Points

We study an optimal feedback control problem for the rod heating process by means of lumped sources. The control actions are the powers of the sources, the values of which are defined on the class of zonal controls. The values of the parameters of zonal control actions are determined by subsets of the state space, to which belong the values of the process state at the measurement points at the current and past time moments. The posed problem is reduced to a parametric optimal control problem on determining a finite-dimensional vector of values of the parameters of zonal control actions. We have obtained optimality conditions for the values of the parameters of zonal control actions. These conditions contain formulas for the gradient of the objective functional with respect to the optimizable parameters. They make it possible to solve the reduced problem numerically with the application of efficient first-order optimization methods.

Https://www.bakumathj.org/archive/Vol2No2/j.bmj.033.pdf

In the paper, we consider an approach for numerical solution to the optimal feedback control problem for an object with distributed parameters on the basis of observation of the object’s phase state at its specific locations by the example of the rod heating process. The control actions are the power of the heat source, the values of which are defined on the class of “zonal” controls. The values of the parameters of zonal control actions are determined by subsets of the state space, to which belong the values of the process state at the measurement points at the current moment of time. The problem of determining zonal controls is reduced to a parametric optimal control problem on determining a finite-dimensional vector of values of the parameters of zonal control actions. We derive optimality conditions for the values of the parameters of zonal control actions. These conditions contain formulas for the gradient of the objective functional with respect to the optimizable parameters of zonal controls. They make it possible to solve the stated problem numerically with the application of efficient first-order optimization methods.

Multi-graph multi-label learning with novel and missing labels

Real-life objects typically contain complex structures, and the graph is a prevalent presentation for describing such objects. Multi-graph multi-label (MGML) learning is a supervised learning framework for learning complicated objects associated with multiple labels, where each object is described as a bag-of-graphs representation. However, existing methods are dependent on transforming graphs into instances, which prevents the structure information from being fully utilized; in addition, the adaptability of MGML is insufficient for the assumptions that the label set is in a closed environment and complete. In real applications, some novel labels may exist outside the fixed set, and some values of labels are missing. Existing MGML approaches are unable to deal with these novel and missing labels. In this paper, we propose a new multi-graph multi-label learning with novel and missing labels (Mgmlnm) algorithm, which is the first attempt to learn novel and missing labels in the MGML problem. To preserve the structure information, we exploit the specific graph kernels to generate an efficient graph representation. To learn multi-graph data with novel and missing labels, we propose a unified objective function, which has a two-perspective regularization term, a projection-similarity regularization term, and a bag-dependent regularization term, enabling novel and missing labels to be modeled simultaneously and exploring the complex relationship between bag and graph labels. Two first-order proximity optimization methods are then developed to solve the unified problem efficiently. A theoretical analysis of the generalization bound is provided for the unified objective. Experimental results demonstrate the effectiveness of the proposed algorithm.

Fast and Interpretable Dynamics for Fisher Markets via Block-Coordinate Updates

We consider the problem of large-scale Fisher market equilibrium computation through scalable first-order optimization methods. It is well-known that market equilibria can be captured using structured convex programs such as the Eisenberg-Gale and Shmyrev convex programs. Highly performant deterministic full-gradient first-order methods have been developed for these programs. In this paper, we develop new block-coordinate first-order methods for computing Fisher market equilibria, and show that these methods have interpretations as tâtonnement-style or proportional response-style dynamics where either buyers or items show up one at a time. We reformulate these convex programs and solve them using proximal block coordinate descent methods, a class of methods that update only a small number of coordinates of the decision variable in each iteration. Leveraging recent advances in the convergence analysis of these methods and structures of the equilibrium-capturing convex programs, we establish fast convergence rates of these methods.

An Efficient Underwater Navigation Method Using MPC with Unknown Kinematics and Non-Linear Disturbances

Many Autonomous Underwater Vehicles (AUVs) need to cope with hazardous underwater medium using a limited computational capacity while facing unknown kinematics and disturbances. However, most algorithms proposed for navigation in such conditions fail to fulfil all conditions at the same time. In this work, we propose an optimal control method, based on a receding horizon approach, namely MPC (Model Predictive Control). Our model also estimates the kinematics of the medium and its disturbances, using efficient tools that rely on the use of linear algebra and first-order optimization methods. We also test our ideas using an extensive set of simulations, which show that the proposed ideas are very competitive in terms of cost and computational efficiency in cases of total and partial observability.

On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

<p style='text-indent:20px;'>Second-order continuous-time dissipative dynamical systems with viscous and Hessian driven damping have inspired effective first-order algorithms for solving convex optimization problems. While preserving the fast convergence properties of the Nesterov-type acceleration, the Hessian driven damping makes it possible to significantly attenuate the oscillations. To study the stability of these algorithms with respect to perturbations, we analyze the behaviour of the corresponding continuous systems when the gradient computation is subject to exogenous additive errors. We provide a quantitative analysis of the asymptotic behaviour of two types of systems, those with implicit and explicit Hessian driven damping. We consider convex, strongly convex, and non-smooth objective functions defined on a real Hilbert space and show that, depending on the formulation, different integrability conditions on the perturbations are sufficient to maintain the convergence rates of the systems. We highlight the differences between the implicit and explicit Hessian damping, and in particular point out that the assumptions on the objective and perturbations needed in the implicit case are more stringent than in the explicit case.</p>

On the First-Order Optimization Methods in Deep Image Prior

Abstract Deep learning methods have state-of-the-art performances in many image restoration tasks. Their effectiveness is mostly related to the size of the dataset used for the training. Deep image prior (DIP) is an energy-function framework which eliminates the dependency on the training set, by considering the structure of a neural network as an handcrafted prior offering high impedance to noise and low impedance to signal. In this paper, we analyze and compare the use of different optimization schemes inside the DIP framework for the denoising task.

Neural Photo-Finishing

Image processing pipelines are ubiquitous and we rely on them either directly, by filtering or adjusting an image post-capture, or indirectly, as image signal processing (ISP) pipelines on broadly deployed camera systems. Used by artists, photographers, system engineers, and for downstream vision tasks, traditional image processing pipelines feature complex algorithmic branches developed over decades. Recently, image-to-image networks have made great strides in image processing, style transfer, and semantic understanding. The differentiable nature of these networks allows them to fit a large corpus of data; however, they do not allow for intuitive, fine-grained controls that photographers find in modern photo-finishing tools. This work closes that gap and presents an approach to making complex photo-finishing pipelines differentiable, allowing legacy algorithms to be trained akin to neural networks using first-order optimization methods. By concatenating tailored network proxy models of individual processing steps (e.g. white-balance, tone-mapping, color tuning), we can model a non-differentiable reference image finishing pipeline more faithfully than existing proxy image-to-image network models. We validate the method for several diverse applications, including photo and video style transfer, slider regression for commercial camera ISPs, photography-driven neural demosaicking, and adversarial photo-editing.

A Fully Distributed Hybrid Control Framework For Non-Differentiable Multi-Agent Optimization

This paper develops a fully distributed hybrid control framework for distributed constrained optimization problems. The individual cost functions are non-differentiable and convex. Based on hybrid dynamical systems, we present a distributed state-dependent hybrid design to improve the transient performance of distributed primal-dual first-order optimization methods. The proposed framework consists of a distributed constrained continuous-time mapping in the form of a differential inclusion and a distributed discrete-time mapping triggered by the satisfaction of local jump set. With the semistability theory of hybrid dynamical systems, the paper proves that the hybrid control algorithm converges to one optimal solution instead of oscillating among different solutions. Numerical simulations illustrate better transient performance of the proposed hybrid algorithm compared with the results of the existing continuous-time algorithms.

WARPd: A Linearly Convergent First-Order Primal-Dual Algorithm for Inverse Problems with Approximate Sharpness Conditions

Sharpness conditions directly control the recovery performance of restart schemes for first-order optimization methods without the need for restrictive assumptions such as strong convexity. However, they are challenging to apply in the presence of noise or approximate model classes (e.g., approximate sparsity). We provide a first-order method: weighted, accelerated, and restarted primal-dual (WARPd), based on primal-dual iterations and a novel restart-reweight scheme. Under a generic approximate sharpness condition, WARPd achieves stable linear convergence to the desired vector. Many problems of interest fit into this framework. For example, we analyze sparse recovery in compressed sensing, low-rank matrix recovery, matrix completion, TV regularization, minimization of $\|Bx\|_{l^1}$ under constraints ($l^1$-analysis problems for general $B$), and mixed regularization problems. We show how several quantities controlling recovery performance also provide explicit approximate sharpness constants. Numerical experiments show that WARPd compares favorably with specialized state-of-the-art methods and is ideally suited for solving large-scale problems. We also present a noise-blind variant based on a square-root LASSO decoder. Finally, we show how to unroll WARPd as neural networks. This approximation theory result provides lower bounds for stable and accurate neural networks for inverse problems and sheds light on architecture choices. Code and a gallery of examples are available online as a MATLAB package.

Seismic reflection waveform inversion based on Gauss–Newton optimization

Abstract Owing to the lack of long-offset signals and effective low-frequency components, it is extremely difficult for classical full-waveform inversion (FWI) to obtain the long-wavelength components of medium and deep elastic parameter models. Considering ray theory, reflection traveltime tomography is restricted by resolution, and the reflection waveform inversion (RWI) method of the wave equation, and its practicability, has received greater attention. However, previous studies into RWI have primarily used first-order optimization methods, in which convergence and resolution still need substantial improvement. Based on the second-order optimization inversion theory, we deduced the reflection wave-sensitive kernel, objective function gradient and Hessian operator of background and perturbation model. Additionally, we revealed the de-fuzzification effect of the Hessian matrix on the gradient of inversion and the working mechanism of background velocity updating on direction optimization and inversion resolution improvement. Experiments on the SEG/EAGE overthrust model showed that an RWI with the Gauss–Newton method using an approximate Hessian matrix significantly improved convergence and wide-spectrum modeling capability compared with the conjugate gradient method. On the streamer seismic data of the East China Sea, the second-order optimization RWI method surpassed the commonly used reflection traveltime tomography based on pre-stack depth migration. In addition, the RTM of inversion model can depict the complex fault system in the Changjiang sag with a high resolution, which improves the imaging quality of the deep basement.

Reflection-based traveltime and waveform inversion with second-order optimization

Reflection-based inversion that aims to reconstruct the low-to-intermediate wavenumbers of the subsurface model, can be a complementary to refraction-data-driven full-waveform inversion (FWI), especially for the deep target area where diving waves cannot be acquired at the surface. Nevertheless, as a typical nonlinear inverse problem, reflection waveform inversion may easily suffer from the cycle-skipping issue and have a slow convergence rate, if gradient-based first-order optimization methods are used. To improve the accuracy and convergence rate, we introduce the Hessian operator into reflection traveltime inversion (RTI) and reflection waveform inversion (RWI) in the framework of second-order optimization. A practical two-stage workflow is proposed to build the velocity model, in which Gauss-Newton RTI is first applied to mitigate the cycle-skipping problem and then Gauss-Newton RWI is employed to enhance the model resolution. To make the Gauss-Newton iterations more efficiently and robustly for large-scale applications, we introduce proper preconditioning for the Hessian matrix and design appropriate strategies to reduce the computational costs. The example of a real dataset from East China Sea demonstrates that the cascaded Hessian-based RTI and RWI have good potential to improve velocity model building and seismic imaging, especially for the deep targets.

Understanding Stochastic Optimization Behavior at the Layer Update Level (Student Abstract)

Popular first-order stochastic optimization methods for deep neural networks (DNNs) are usually either accelerated schemes (e.g. stochastic gradient descent (SGD) with momentum) or adaptive step-size methods (e.g. Adam/AdaMax, AdaBelief). In many contexts, including image classification with DNNs, adaptive methods tend to generalize poorly compared to SGD, i.e. get stuck in non-robust local minima; however, SGD typically converges slower. We analyze possible reasons for this behavior by modeling gradient updates as vectors of random variables and comparing them to probabilistic bounds to identify "meaningful" updates. Through experiments, we observe that only layers close to the output have "definitely non-random" update behavior. In the future, the tools developed here may be useful in rigorously quantifying and analyzing intuitions about why some optimizers and particular DNN architectures perform better than others.

Global Linear Convergence of Evolution Strategies on More than Smooth Strongly Convex Functions

Evolution strategies (ESs) are zeroth-order stochastic black-box optimization heuristics invariant to monotonic transformations of the objective function. They evolve a multivariate normal distribution, from which candidate solutions are generated. Among different variants, CMA-ES is nowadays recognized as one of the state-of-the-art zeroth-order optimizers for difficult problems. Despite ample empirical evidence that ESs with a step-size control mechanism converge linearly, theoretical guarantees of linear convergence of ESs have been established only on limited classes of functions. In particular, theoretical results on convex functions are missing, where zeroth-order and also first-order optimization methods are often analyzed. In this paper, we establish almost sure linear convergence and a bound on the expected hitting time of an ES family, namely, the $(1+1)_\kappa$-ES, which includes the (1+1)-ES with (generalized) one-fifth success rule and an abstract covariance matrix adaptation with bounded condition number, on a broad class of functions. The analysis holds for monotonic transformations of positively homogeneous functions and of quadratically bounded functions, the latter of which particularly includes monotonic transformation of strongly convex functions with Lipschitz continuous gradient. As far as the authors know, this is the first work that proves linear convergence of ES on such a broad class of functions.

Fast MATLAB evaluation of nonlinear energies using FEM in 2D and 3D: Nodal elements

Nonlinear energy functionals appearing in the calculus of variations can be discretized by the finite element (FE) method and formulated as a sum of energy contributions from local elements. A fast evaluation of energy functionals containing the first order gradient terms is a central part of this contribution. We describe a vectorized implementation using the simplest linear nodal (P1) elements in which all energy contributions are evaluated all at once without the loop over triangular or tetrahedral elements. Furthermore, in connection to the first-order optimization methods, the discrete gradient of energy functional is assembled in a way that the gradient components are evaluated over all degrees of freedom all at once. The key ingredient is the vectorization of exact or approximate energy gradients over nodal patches. It leads to a time-efficient implementation at higher memory-cost. Provided codes in MATLAB related to 2D/3D hyperelasticity and 2D p-Laplacian problem are available for download and structured in a way it can be easily extended to other types of vector or scalar forms of energies.

An Efficient Fisher Matrix Approximation Method for Large-Scale Neural Network Optimization.

Although the shapes of the parameters are not crucial for designing first-order optimization methods in large scale empirical risk minimization problems, they have important impact on the size of the matrix to be inverted when developing second-order type methods. In this article, we propose an efficient and novel second-order method based on the parameters in the real matrix space [Formula: see text] and a matrix-product approximate Fisher matrix (MatFisher) by using the products of gradients. The size of the matrix to be inverted is much smaller than that of the Fisher information matrix in the real vector space [Formula: see text]. Moreover, by utilizing the matrix delayed update and the block diagonal approximation techniques, the computational cost can be controlled and is comparable with first-order methods. A global convergence and a superlinear local convergence analysis are established under mild conditions. Numerical results on image classification with ResNet50, quantum chemistry modeling with SchNet, and data-driven partial differential equations solution with PINN illustrate that our method is quite competitive to the state-of-the-art methods.

ОБ ОДНОЙ ЗАДАЧЕ УПРАВЛЕНИЯ С ОБРАТНОЙ СВЯЗЬЮ ИСТОЧНИКАМИ НАГРЕВА СТЕРЖНЯ

In the work proposes an approach to solving the problem of synthesis of motion and power control of lumped sources with optimization of the locations of the points of the measurements. For specificity, the problem of feedback control of moving heat sources during rod heating is considered. The power and speed of point-wise sources are assigned depending on the state of the processes at the measurement points. The formulas for the gradient components of the objective functional, allowing for the numerical solution of the problem using of the first-order optimization methods are obtained.

Frank\u2013Wolfe and friends: a journey into projection-free first-order optimization methods

Invented some 65 years ago in a seminal paper by Marguerite Straus-Frank and Philip Wolfe, the Frank–Wolfe method recently enjoys a remarkable revival, fuelled by the need of fast and reliable first-order optimization methods in Data Science and other relevant application areas. This review tries to explain the success of this approach by illustrating versatility and applicability in a wide range of contexts, combined with an account on recent progress in variants, improving on both the speed and efficiency of this surprisingly simple principle of first-order optimization.