Second-order Optimization Methods Research Articles

Inexact Nonconvex Newton-Type Methods

The paper aims to extend the theory and application of nonconvex Newton-type methods, namely trust region and cubic regularization, to the settings in which, in addition to the solution of subproblems, the gradient and the Hessian of the objective function are approximated. Using certain conditions on such approximations, the paper establishes optimal worst-case iteration complexities as the exact counterparts. This paper is part of a broader research program on designing, analyzing, and implementing efficient second-order optimization methods for large-scale machine learning applications. The authors were based at UC Berkeley when the idea of the project was conceived. The first two authors were PhD students, the third author was a postdoc, all supervised by the fourth author.

Application of the second-order optimization methods to the stochastic programming problems with probability function

Application of the second-order optimization methods to the stochastic programming problems with probability function

Optimization methods for achieving high diffraction efficiency with perfect electric conducting gratings.

This work presents the implementation, numerical examples, and experimental convergence study of first- and second-order optimization methods applied to one-dimensional periodic gratings. Through boundary integral equations and shape derivatives, the profile of a grating is optimized such that it maximizes the diffraction efficiency for given diffraction modes for transverse electric polarization. We provide a thorough comparison of three different optimization methods: a first-order method (gradient descent); a second-order approach based on a Newton iteration, where the usual Newton step is replaced by taking the absolute value of the eigenvalues given by the spectral decomposition of the Hessian matrix to deal with non-convexity; and the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, a quasi-Newton method. Numerical examples are provided to validate our claims. Moreover, two grating profiles are designed for high efficiency in the Littrow configuration and then compared to a high efficiency commercial grating. Conclusions and recommendations, derived from the numerical experiments, are provided as well as future research avenues.

SPAN: A Stochastic Projected Approximate Newton Method

Second-order optimization methods have desirable convergence properties. However, the exact Newton method requires expensive computation for the Hessian and its inverse. In this paper, we propose SPAN, a novel approximate and fast Newton method. SPAN computes the inverse of the Hessian matrix via low-rank approximation and stochastic Hessian-vector products. Our experiments on multiple benchmark datasets demonstrate that SPAN outperforms existing first-order and second-order optimization methods in terms of the convergence wall-clock time. Furthermore, we provide a theoretical analysis of the per-iteration complexity, the approximation error, and the convergence rate. Both the theoretical analysis and experimental results show that our proposed method achieves a better trade-off between the convergence rate and the per-iteration efficiency.

Second-order MCSCF optimization revisited. I. Improved algorithms for fast and robust second-order CASSCF convergence.

A new improved implementation of the second-order multiconfiguration self-consistent field optimization method of Werner and Knowles [J. Chem. Phys. 82, 5053 (1985)] is presented. It differs from the original method by more stable and efficient algorithms for minimizing the second-order energy approximation in the so-called microiterations. Conventionally, this proceeds by alternating optimizations of the orbitals and configuration (CI) coefficients and is linearly convergent. The most difficult part is the orbital optimization, which requires solving a system of nonlinear equations that are often strongly coupled. We present a much improved algorithm for solving this problem, using an iterative subspace method that includes part of the orbital Hessian explicitly, and discuss different strategies for performing the uncoupled optimization in a most efficient manner. Second, we present a new solver in which the orbital-CI coupling is treated explicitly. This leads to quadratic convergence of the microiterations but requires many additional evaluations of reduced (transition) density matrices. In difficult optimization problems with a strong coupling of the orbitals and CI coefficients, it leads to much improved convergence of both the macroiterations and the microiterations. Third, the orbital-CI coupling is treated approximately using a quasi-Newton approach with Broyden-Fletcher-Goldfarb-Shanno updates of the orbital Hessian. It is demonstrated that this converges almost as well as the explicitly coupled method but avoids the additional effort for computing many transition density matrices. The performance of the three methods is compared for a set of 21 aromatic molecules, an Fe(ii)-porphine transition metal complex, as well as for the [Cu2O2(NH3) 6]2+, FeCl3, Co2(CO)6C2H2, and Al4O2 complexes. In all cases, faster and more stable convergence than with the original implementation is achieved.

A subsampled truncated-Newton method for multiparameter full-waveform inversion

Accounting for the Hessian in full-waveform inversion (FWI) can lead to higher convergence rates, improved resolution, and better mitigation of parameter trade-off in multiparameter problems. In spite of these advantages, the adoption of second-order optimization methods (e.g., truncated Newton [TN]) has been precluded by their high computational cost. We propose a subsampled TN (STN) algorithm for time-domain FWI with applications presented for the elastic isotropic case. By using uniform or nonuniform source subsampling during the computation of Hessian-vector products, we reduce the number of partial differential equation solves required per iteration when compared to the conventional TN algorithm. We evaluate the performance of STN through synthetic inversions on the Marmousi II and BP 2.5D models, using the limited-memory Broyden–Fletcher–Goldfarb–Shanno and TN algorithms as benchmarks. We determine that STN reaches a target misfit reduction at an overall cost comparable to first-order gradient methods, while retaining favorable convergence properties of TN methods. Furthermore, we evaluate an example in which nonuniform sampling outperforms uniform sampling in STN due to highly nonuniform source contributions to the Hessian.

Optim: A mathematical optimization package for Julia

Optim provides a range of optimization capabilities written in the Julia programming language (Bezanson et al. 2017). Our aim is to enable researchers, users, and other Julia packages to solve optimization problems without writing such algorithms themselves. The package supports optimization on manifolds, functions of complex numbers, and input types such as arbitrary precision vectors and matrices. We have implemented routines for derivative free, first-order, and second-order optimization methods. The user can provide derivatives themselves, or request that they are calculated using automatic differentiation or finite difference methods. The main focus of the package has currently been on unconstrained optimization, however, box-constrained optimization is supported, and a more comprehensive support for constraints is underway. Similar to Optim, the C library NLopt (Johnson 2008) contains a collection of nonlinear optimization routines. In Python, scipy.optimize supports many of the same algorithms as Optim does, and Pymanopt (Townsend, Niklas, and Weichwald 2016) is a toolbox for manifold optimization. Within the Julia community, the packages BlackBoxOptim.jl and Optimize.jl provide optimization capabilities focusing on derivative-free and large-scale smooth problems respectively. The packages Convex.jl and JuMP.jl (Dunning, Huchette, and Lubin 2017) define modelling languages for which users can formulate optimization problems. In contrast to the previously mentioned optimization codes, Convex and JuMP work as abstraction layers between the user and solvers from a other packages.

Emergence of Lévy Walks from Second-Order Stochastic Optimization.

In natural foraging, many organisms seem to perform two different types of motile search: directed search (taxis) and random search. The former is observed when the environment provides cues to guide motion towards a target. The latter involves no apparent memory or information processing and can be mathematically modeled by random walks. We show that both types of search can be generated by a common mechanism in which Lévy flights or Lévy walks emerge from a second-order gradient-based search with noisy observations. No explicit switching mechanism is required-instead, continuous transitions between the directed and random motions emerge depending on the Hessian matrix of the cost function. For a wide range of scenarios, the Lévy tail index is α=1, consistent with previous observations in foraging organisms. These results suggest that adopting a second-order optimization method can be a useful strategy to combine efficient features of directed and random search.

An improved isenthalpic flash algorithm based on maximization of entropy

An algorithm is proposed for conducting the phase-equilibrium calculation for a given mixture with a specified pressure, enthalpy, and composition. We use a second-order optimization method to maximize entropy, with the individual phase compositions and enthalpies as the independent variables. At any point during the iterative solution procedure, the direction of the iterative step can be checked by ensuring an increase in entropy. During the iterations we allow phases to be split further into sub-phases to tackle the issue of a phase having multiple stable roots for the specified enthalpy. We successfully apply the method to calculate two-phase thermodynamic equilibrium for many hydrocarbon mixtures, some of which are known to have posed issues related to narrow-boiling.

Performance of first- and second-order methods for $$\ell _1$$ ℓ 1 -regularized least squares problems

We study the performance of first- and second-order optimization methods for $$\ell _1$$l1-regularized sparse least-squares problems as the conditioning of the problem changes and the dimensions of the problem increase up to one trillion. A rigorously defined generator is presented which allows control of the dimensions, the conditioning and the sparsity of the problem. The generator has very low memory requirements and scales well with the dimensions of the problem.

Algorithmic Differentiation of Numerical Methods: Second-order Adjoint Solvers for Parameterized Systems of Nonlinear Equations

Adjoint mode algorithmic (also know as automatic) differentiation (AD) transforms implementations of multivariate vector functions as computer programs into first-order adjoint code. Its reapplication or combinations with tangent mode AD yields higher-order adjoint code. Second derivatives play an important role in nonlinear programming. For example, second-order (Newton-type) nonlinear optimization methods promise faster convergence in the neighborhood of the minimum through taking into account second derivative information. The adjoint mode is of particular interest in large-scale gradient-based nonlinear optimization due to the independence of its computational cost on the number of free variables. Part of the objective function may be given implicitly as the solution of a system of n parameterized nonlinear equations. If the system parameters depend on the free variables of the objective, then second derivatives of the nonlinear system's solution with respect to those parameters are required. The local computational overhead as well as the additional memory requirement for the computation of second-order adjoints of the solution vector with respect to the parameters by AD depends on the number of iterations performed by the nonlinear solver. This dependence can be eliminated by taking a symbolic approach to the differentiation of the nonlinear system.

Shape and cross-section optimization of plane trusses subjected to earthquake excitation using gradient and Hessian matrix calculations

ABSTRACTThis article describes a second-order shape and cross-section optimization method of plane truss subjected to earthquake excitation. The method is based on gradient and Hessian matrix calculation. First, the first and second derivatives of dynamic response with respect to design variables are calculated based on the Newmark method. Second, the inequality time-dependent constraint problem is converted into a sequence of appropriately formed unconstrained problems using the integral interior penalty function method. Then, the gradient and Hessian matrix of the integral interior penalty function are computed. Third, Marquardt's method is employed to solve the unconstrained problems. Finally, the new approach is validated through several case studies. The results show that the new optimization method is an efficient and effective approach for minimum weight design of truss structures.

Second-order Tangent Solvers for Systems of Parameterized Nonlinear Equations

Abstract Forward mode algorithmic differentiation transforms implementations of multivariate vector functions as computer programs into first directional derivative (also: first-order tangent) code. Its reapplication yields higher directional derivative (higher-order tangent) code. Second derivatives play an important role in nonlinear programming. For example, second-order (Newton-type) nonlinear optimization methods promise faster convergence in the neighborhood of the minimum through taking into account second derivative information. Part of the objective function may be given implicitly as the solution of a system of n parameterized nonlinear equations. If the system parameters depend on the free variables of the objective, then second derivatives of the nonlinear system's solution with respect to those parameters are required. The local computational overhead for the computation of second-order tangents of the solution vector with respect to the parameters by Algorithmic Differentiation depends on the number of iterations performed by the nonlinear solver. This dependence can be eliminated by taking a second-order symbolic approach to differentiation of the nonlinear system.

Fast full waveform inversion with source encoding and second-order optimization methods

Full waveform inversion (FWI) of 3-D data sets has recently been possible thanks to the development of high performance computing. However, FWI remains a computationally intensive task when high frequencies are injected in the inversion or more complex wave physics (viscoelastic) is accounted for. The highest computational cost results from the numerical solution of the wave equation for each seismic source. To reduce the computational burden, one well-known technique is to employ a random linear combination of the sources, rather that using each source independently. This technique, known as source encoding, has shown to successfully reduce the computational cost when applied to real data. Up to now, the inversion is normally carried out using gradient descent algorithms. With the idea of achieving a fast and robust frequency-domain FWI, we assess the performance of the random source encoding method when it is interfaced with second-order optimization methods (quasi-Newton l-BFGS, truncated Newton). Because of the additional seismic modelings required to compute the Newton descent direction, it is not clear beforehand if truncated Newton methods can indeed further reduce the computational cost compared to gradient algorithms. We design precise stopping criteria of iterations to fairly assess the computational cost and the speed-up provided by the source encoding method for each optimization method. We perform experiment on synthetic and real data sets. In both cases, we confirm that combining source encoding with second-order optimization methods reduces the computational cost compared to the case where source encoding is interfaced with gradient descent algorithms. For the synthetic data set, inspired from the geology of Gulf of Mexico, we show that the quasi-Newton l-BFGS algorithm requires the lowest computational cost. For the real data set application on the Valhall data, we show that the truncated Newton methods provide the most robust direction of descent.

Simultaneous Optimal Estimation of Mode Transition Times and Parameters Applied to Simple Traction Models

An optimization-based estimation method is presented to determine mode transition times and model parameters for hybrid systems. First- and second-order optimality conditions are derived, including cross-derivative terms between transition times and parameters. Second-order optimization methods are shown to provide superior convergence to correct values in simulation, and to values within expected ranges experimentally for traction estimation of a skid-steered vehicle.

An Optimization Rule for In Silico Identification of Targeted Overproduction in Metabolic Pathways

In an extension of previous work, here we introduce a second-order optimization method for determining optimal paths from the substrate to a target product of a metabolic network, through which the amount of the target is maximum. An objective function for the said purpose, along with certain linear constraints, is considered and minimized. The basis vectors spanning the null space of the stoichiometric matrix, depicting the metabolic network, are computed, and their convex combinations satisfying the constraints are considered as flux vectors. A set of other constraints, incorporating weighting coefficients corresponding to the enzymes in the pathway, are considered. These weighting coefficients appear in the objective function to be minimized. During minimization, the values of these weighting coefficients are estimated and learned. These values, on minimization, represent an optimal pathway, depicting optimal enzyme concentrations, leading to overproduction of the target. The results on various networks demonstrate the usefulness of the methodology in the domain of metabolic engineering. A comparison with the standard gradient descent and the extreme pathway analysis technique is also performed. Unlike the gradient descent method, the present method, being independent of the learning parameter, exhibits improved results.

A second-order unconstrained optimization method for canonical-ensemble density-functional methods

A second order converging method of ensemble optimization (SOEO) in the framework of Kohn-Sham Density-Functional Theory is presented, where the energy is minimized with respect to an ensemble density matrix. It is general in the sense that the number of fractionally occupied orbitals is not predefined, but rather it is optimized by the algorithm. SOEO is a second order Newton-Raphson method of optimization, where both the form of the orbitals and the occupation numbers are optimized simultaneously. To keep the occupation numbers between zero and two, a set of occupation angles is defined, from which the occupation numbers are expressed as trigonometric functions. The total number of electrons is controlled by a built-in second order restriction of the Newton-Raphson equations, which can be deactivated in the case of a grand-canonical ensemble (where the total number of electrons is allowed to change). To test the optimization method, dissociation curves for diatomic carbon are produced using different functionals for the exchange-correlation energy. These curves show that SOEO favors symmetry broken pure-state solutions when using functionals with exact exchange such as Hartree-Fock and Becke three-parameter Lee-Yang-Parr. This is explained by an unphysical contribution to the exact exchange energy from interactions between fractional occupations. For functionals without exact exchange, such as local density approximation or Becke Lee-Yang-Parr, ensemble solutions are favored at interatomic distances larger than the equilibrium distance. Calculations on the chromium dimer are also discussed. They show that SOEO is able to converge to ensemble solutions for systems that are more complicated than diatomic carbon.

A New Gradient Descent Approach for Local Learning of Fuzzy Neural Models

The majority of reported learning methods for Takagi-Sugeno-Kang (TSK) fuzzy neural models to date mainly focus on improvement of their accuracy. However, one of the key design requirements in building an interpretable fuzzy model is that each obtained rule consequent must match well with the system local behavior when all the rules are aggregated to produce the overall system output. This is one of the distinctive characteristics from black-box models such as neural networks. Therefore, how to find a desirable set of fuzzy partitions and, hence, identify the corresponding consequent models which can be directly explained in terms of system behavior, presents a critical step in fuzzy neural modeling. In this paper, a new learning approach considering both nonlinear parameters in the rule premises and linear parameters in the rule consequents is proposed. Unlike the conventional two-stage optimization procedure widely practiced in the field where the two sets of parameters are optimized separately, the consequent parameters are transformed into a dependent set on the premise parameters, thereby enabling the introduction of a new integrated gradient descent learning approach. Thus, a new Jacobian matrix is proposed and efficiently computed to achieve a more accurate approximation of the cost function by using the second-order Levenberg-Marquardt optimization method. Several other interpretability issues regarding the fuzzy neural model are also discussed and integrated into this new learning approach. Numerical examples are presented to illustrate the resultant structure of the fuzzy neural models and the effectiveness of the proposed new algorithm, and compared with the results from some well-known methods.

Second-Order Switching Time Optimization for Nonlinear Time-Varying Dynamic Systems

This technical note gives a method for calculating the first and second derivatives of a cost function with respect to switching times for systems with piecewise second-differentiable dynamics. Differential equations governing the linear and bilinear operators required for calculating the derivatives are presented. Example optimizations of linear and nonlinear systems are presented as evidence for the value of second-order optimization methods. One example converges in 33 iterations using second-order methods whereas the first-order algorithm requires over 30 000 iterations.

Separable Least Squares Identification of Wiener Box-Jenkins Models

AbstractThis paper presents an approach for the identification of a Wiener model, a dynamic linear system followed by a static nonlinearity, in the presence of colored measurement noise. A Box-Jenkins model structure is proposed where the process model consists of a recursive digital filter followed by a polynomial nonlinearity, while the noise model is represented by another recursive digital filter. The prediction error method is implemented using a separable least squares technique in order to estimate the parameters of the linear and nonlinear elements. The parameters of the digital filters are estimated using a second-order iterative optimization method since they appear nonlinearly in the output. After each iteration, the nonlinearity is fitted using linear regression. Monte-Carlo simulation is used to validate the algorithm.