Successive Quadratic Approximation Research Articles

Accelerating inexact successive quadratic approximation for regularized optimization through manifold identification

AbstractFor regularized optimization that minimizes the sum of a smooth term and a regularizer that promotes structured solutions, inexact proximal-Newton-type methods, or successive quadratic approximation (SQA) methods, are widely used for their superlinear convergence in terms of iterations. However, unlike the counter parts in smooth optimization, they suffer from lengthy running time in solving regularized subproblems because even approximate solutions cannot be computed easily, so their empirical time cost is not as impressive. In this work, we first show that for partly smooth regularizers, although general inexact solutions cannot identify the active manifold that makes the objective function smooth, approximate solutions generated by commonly-used subproblem solvers will identify this manifold, even with arbitrarily low solution precision. We then utilize this property to propose an improved SQA method, $$^{+}$$ + , that switches to efficient smooth optimization methods after this manifold is identified. We show that for a wide class of degenerate solutions, $$^{+}$$ + possesses superlinear convergence not only in iterations, but also in running time because the cost per iteration is bounded. In particular, our superlinear convergence result holds on problems satisfying a sharpness condition that is more general than that in existing literature. We also prove iterate convergence under a sharpness condition for inexact SQA, which is novel for this family of methods that could easily violate the classical relative-error condition frequently used in proving convergence under similar conditions. Experiments on real-world problems support that $$^{+}$$ + improves running time over some modern solvers for regularized optimization.

An inexact successive quadratic approximation method for a class of difference-of-convex optimization problems

An inexact successive quadratic approximation method for a class of difference-of-convex optimization problems

Modified successive approximation methods for the nonlinear eigenvalue problems

In this paper we present the convergence factor of the successive quadratic approximation method for solving the nonlinear eigenvalue problems explicitly. To reduce the computational cost, we propose the inexact versions of the successive approximation methods and analyze the corresponding convergence properties. Numerical results demonstrate the effectiveness of the modified successive approximation methods.

Global complexity analysis of inexact successive quadratic approximation methods for regularized optimization under mild assumptions

Successive quadratic approximations (SQA) are numerically efficient for minimizing the sum of a smooth function and a convex function. The iteration complexity of inexact SQA methods has been analyzed recently. In this paper, we present an algorithmic framework of inexact SQA methods with four types of line searches, and analyze its global complexity under milder assumptions. First, we show its well-definedness and some decreasing properties. Second, under the quadratic growth condition and a uniform positive lower bound condition on stepsizes, we show that the function value sequence and the iterate sequence are linearly convergent. Moreover, we obtain a o(1/k) complexity without the quadratic growth condition, improving existing O(1/k) complexity results. At last, we show that a local gradient-Lipschitz-continuity condition could guarantee a uniform positive lower bound for the stepsizes.

Inexact Successive quadratic approximation for regularized optimization

Successive quadratic approximations, or second-order proximal methods, are useful for minimizing functions that are a sum of a smooth part and a convex, possibly nonsmooth part that promotes regularization. Most analyses of iteration complexity focus on the special case of proximal gradient method, or accelerated variants thereof. There have been only a few studies of methods that use a second-order approximation to the smooth part, due in part to the difficulty of obtaining closed-form solutions to the subproblems at each iteration. In fact, iterative algorithms may need to be used to find inexact solutions to these subproblems. In this work, we present global analysis of the iteration complexity of inexact successive quadratic approximation methods, showing that an inexact solution of the subproblem that is within a fixed multiplicative precision of optimality suffices to guarantee the same order of convergence rate as the exact version, with complexity related in an intuitive way to the measure of inexactness. Our result allows flexible choices of the second-order term, including Newton and quasi-Newton choices, and does not necessarily require increasing precision of the subproblem solution on later iterations. For problems exhibiting a property related to strong convexity, the algorithms converge at global linear rates. For general convex problems, the convergence rate is linear in early stages, while the overall rate is O(1 / k). For nonconvex problems, a first-order optimality criterion converges to zero at a rate of $$O(1/\sqrt{k})$$ .

An inexact successive quadratic approximation method for L-1 regularized optimization

We study a Newton-like method for the minimization of an objective function $$\phi $$ that is the sum of a smooth function and an $$\ell _1$$l1 regularization term. This method, which is sometimes ...

Optimization of shock response within a military vehicle space frame

Space frames are usually used to enhance the structural strength of a vehicle while reducing its overall weight. The space frame of a military vehicle is subjected to significantly different loading than what is experienced in civilian vehicles, such as projectile impact or mine blast. In this work, a finite element (FE) model for the upper half of an armored vehicle with internal space frame is developed. The behavior of the vehicle is studied when subjected to a high impact load that simulates a projectile impact. The objective of this work is to minimize shocks at identified critical locations on the space frame while maintaining the overall structural integrity of the vehicle. Several variables that can affect shock propagation are identified including, the cross-sectional parameters of the internal space frame and outer armor. The optimization problem is solved using the Successive Heuristic Quadratic Approximation (SHQA) technique, which combines successive quadratic approximation with an adaptive random search while varying the bounds of the search space. The entire optimization process is carried out within the MATLAB environment. The results show that a significant reduction in the shock can be achieved using this approach.

Motion estimation for cardiac emission tomography by optical flow methods

This paper describes a new method for estimating the 3D, non-rigid object motion in a time sequence of images. The method is a generalization of a standard optical flow algorithm that is incorporated into a successive quadratic approximation framework. The method was evaluated for gated cardiac emission tomography using images obtained from a mathematical, 4D phantom and a physical, dynamic phantom. The results showed that the proposed method offers improved motion estimation accuracy relative to the standard optical flow method. Convergence of the proposed algorithm was evidenced with a monotonically decreasing objective function value with iteration. Practical application of the motion estimation method in cardiac emission tomography includes quantitative myocardial motion estimation and 4D, motion-compensated image reconstruction.

Convergence acceleration of the Newton-Raphson method using successive quadratic function approximation of residual

This paper presents new methods for determining a proper relaxation factor of the Newton-Raphson method to accelerate the convergence characteristics of a nonlinear finite-element analysis. In the methods, the squared residual of the Galerkin's approximation is successively approximated to a quadratic function using the gradients or Brent's method, and a relaxation factor is determined by minimizing the quadratic function until a quasioptimum relaxation factor is obtained. The presented methods are applied to the TEAM Workshop problem 13, and the results are compared with a conventional method.

Optimization of the Closure-Weld Region of Cylindrical Containers for Long-Term Corrosion Resistance Using the Successive Heuristic Quadratic Approximation Technique*

Welded cylindrical containers are susceptible to stress corrosion cracking (SCC) in the closure-weld area. An induction coil heating technique may be used to relieve the residual stresses in the closure-weld. This technique involves localized heating of the material by the surrounding coils. The material is then cooled to room temperature by quenching. A two-dimensional axisymmetric finite element model is developed to study the effects of induction coil heating and subsequent quenching. The finite element results are validated through an experimental test. The container design is tuned to maximize the compressive stress from the outer surface to a depth that is equal to the long-term general corrosion rate of the container material multiplied by the desired container lifetime. The problem is subject to several geometrical and stress constraints. Two different solution methods are implemented for this purpose. First, an off-the-shelf optimization software is used. The results however were unsatisfactory because of the highly nonlinear nature of the problem. The paper proposes a novel alternative: the Successive Heuristic Quadratic Approximation (SHQA) technique. This algorithm combines successive quadratic approximation with an adaptive random search within varying search space. SHQA promises to be a suitable search method for computationally intensive, highly nonlinear problems.

Nonlinear internal model control using quadratic prediction models

This study is aimed at development of a nonlinear IMC controller that can handle a larger class of nonlinear systems than hitherto, including the singular systems that exhibit change in the sign of the steady state gain. The construction of the model inverse is achieved through inversion of successive quadratic approximation of the nonlinear model operator. The controller synthesis problem is formulated as minimization of 2-norm of single step prediction error. Using the theory of solutions of the multidimensional quadratic equations, a closed form dead-beat type quadratic control law is evolved. The performance of the proposed control law is demonstrated by simulating control of two highly nonlinear systems and also a singular system. In the case of singular system, it has been shown that the proposed quadratic control law can be effectively used for handling a difficult robust control problem arising out of non-attainability of set point due to plant-model mismatch.

A nonlinear programming approach to the parameter design problem

The problem of determining an optimal set of values for the parameters of a manufactured product (and/or a manufacturing process), from the point of view of its robustness against various sources of noise, is commonly referred to as the the parameter design problem. We propose an algorithmic strategy for solving the parameter design problem, and present a specific implementation of this strategy which we refer to as the Successive Quadratic Variance Approximation Method (SQVAM). SQVAM is based on conventional optimization techniques, and assumes that the functional relationship between the input parameters and the performance characteristic of interest is either known or could be well approximated. An illustrative numerical example, discussing the design of a Wheatstone bridge, is then presented. This is the same example used by Taguchi to illustrate the orthogonal array method.

Numerical Computation of Optimal Controls in the Presence of Statedependent Time Lags

The present paper treats several numerical aspects of an algorithm designed for a very general class of optimal control problems with state - dependent time delays in control and state variables. The approach chosen deals with computation of parameterized controls by successive quadratic approximation. The emphasis of the paper is put on the key problem of solving the dynamical systems with delays and associated variational systems in an efficient, safe and stable way by a variable order and step multistep method REBUS, based on the concept of natural interpolation. As a numerical application, the optimal control of a chemical reactor from a perturbed to an equilibrium state is given.

Approximation programming of chemical processes—1: Optimization of flowtran® models

A FLOWTRAN® modelled ammonia synthesis loop has been optimized by Quadratic Approximation Programming (QAP),—a method based on successive quadratic approximation to the plant objective and constraint functions. The algorithm is easily adapted to FLOWTRAN and functions efficiently. In two examples, the total computing time for optimization is only about 65 times that for a single simulation. this is a significant improvement over earlier studies.