Characterizing the Rate of Convergence of the Augmented Lagrange Method for Nonlinear Programming

The convergence rate of the augmented Lagrangian method (ALM) for solving the nonlinear semidefinite optimization problem is studied. Under the Jacobian uniqueness conditions, when a multiplier vector (π,Y) and the penalty parameter σ are chosen such that σ is larger than a threshold σ*>0 and the ratio ∥(π,Y)−(π*,Y*)∥/σ is small enough, it is demonstrated that the convergence rate of the augmented Lagrange method is linear with respect to ∥(π,Y)−(π*,Y*)∥ and the ratio constant is proportional to 1/σ, where (π*,Y*) is the multiplier corresponding to a local minimizer. Furthermore, by analyzing the second-order derivative of the perturbation function of the nonlinear semidefinite optimization problem, we characterize the rate constant of local linear convergence of the sequence of Lagrange multiplier vectors produced by the augmented Lagrange method. This characterization shows that the sequence of Lagrange multiplier vectors has a Q-linear convergence rate when the sequence of penalty parameters {σk} has an upper bound and the convergence rate is superlinear when {σk} is increasing to infinity.

In this work, we are interested in nonlinear symmetric cone problems (NSCPs), which contain as special cases nonlinear semidefinite programming, nonlinear second-order cone programming, and the classical nonlinear programming problems. We explore the possibility of reformulating NSCPs as common nonlinear programs (NLPs), with the aid of squared slack variables. Through this connection, we show how to obtain second-order optimality conditions for NSCPs in an easy manner, thus bypassing a number of difficulties associated to the usual variational analytical approach. We then discuss several aspects of this connection. In particular, we show a “sharp” criterion for membership in a symmetric cone that also encodes rank information. Also, we discuss the possibility of importing convergence results from nonlinear programming to NSCPs, which we illustrate by discussing a simple augmented Lagrangian method for nonlinear symmetric cones. We show that, employing the slack variable approach, we can use the results for NLPs to prove convergence results, thus extending a special case (i.e., the case with strict complementarity) of an earlier result by Sun et al. [Sun D, Sun J, Zhang L (2008) The rate of convergence of the augmented Lagrangian method for nonlinear semidefinite programming. Math. Programming 114(2):349–391] for nonlinear semidefinite programs.

This paper describes the development of an augmented Lagrangian optimization method for the numerical simulation of the inflation process in the design of inflatable space structures. Although the Newton–Raphson scheme was proven to be efficient for solving many nonlinear problems, it can lead to lack of convergence when it is applied to the simulation of the inflation process. As a result, it is recommended to use an optimization algorithm to find the minimum energy configuration that satisfies the equilibrium equations characterizing the final shape of the inflated structure subject to an internal pressure. On top of that, given that some degrees of freedom may be linked, the optimum may be constrained, and specific optimization methods for constrained problems must be considered. The paper presents the formulation and the augmented Lagrangian method (ALM) developed in SAMCEF Mecano for inflatable structures analysis problems. The related quasi-unconstrained optimization problem is solved with a nonlinear conjugate gradient method. The Wolfe conditions are used in conjunction with a cubic interpolation for the line search. Equality constraints are considered and can be easily treated by the ALM formulation. Numerical applications present simulations of unconstrained and constrained inflation processes (i.e., where the motion of some nodes is ruled by a rigid body element restriction and/or problems including contact conditions).

Global placement dominates the circuit placement process in its solution quality and efficiency. With increasing design complexity and various design constraints, it is desirable to develop an efficient, high-quality global placement algorithm for modern large-scale circuit designs. In this paper, we first analyze the properties of four nonlinear optimization methods (the quadratic penalty method, the Lagrange multiplier method, and two augmented Lagrangian methods) for global placement, and then develop a generalized augmented Lagrangian method to solve this problem. Our proposed method preserves the advantages of the quadratic penalty method and the augmented Lagrangian method, and provides a smooth progress from the quadratic penalty method to the augmented Lagrangian method. We prove that the proposed generalized augmented Lagrangian method is globally convergent for the original global placement problem, even with different constraints. Compared with the other four popular optimization methods, experimental results show that our method achieves the best quality and is robust for handling different objectives. In particular, our generalized augmented Lagrangian formulation is theoretically sound and can solve generic large-scale constrained nonlinear optimization problems, which are widely used in many fields.

There are many important practical optimization problems whose feasible regions are not known to be nonempty or not, and optimizers of the objective function with the least constraint violation prefer to be found. A natural way for dealing with these problems is to extend the nonlinear optimization problem as the one optimizing the objective function over the set of points with the least constraint violation. This leads to the study of the shifted problem. This paper focuses on the constrained convex optimization problem. The sufficient condition for the closedness of the set of feasible shifts is presented and the continuity properties of the optimal value function and the solution mapping for the shifted problem are studied. Properties of the conjugate dual of the shifted problem are discussed through the relations between the dual function and the optimal value function. The solvability of the dual of the optimization problem with the least constraint violation is investigated. It is shown that, if the least violated shift is in the domain of the subdifferential of the optimal value function, then this dual problem has an unbounded solution set. Under this condition, the optimality conditions for the problem with the least constraint violation are established in term of the augmented Lagrangian. It is shown that the augmented Lagrangian method has the properties that the sequence of shifts converges to the least violated shift and the sequence of multipliers is unbounded. Moreover, it is proved that the augmented Lagrangian method is able to find an approximate solution to the problem with the least constraint violation and it has linear rate of convergence under an error bound condition. The augmented Lagrangian method is applied to an illustrative convex second-order cone constrained optimization problem with least constraint violation and numerical results verify our theoretical results.

The augmented Lagrangian method is a classical solution method for nonlinear optimization problems. At each iteration, it minimizes an augmented Lagrangian function that consists of the constraint functions and the corresponding Lagrange multipliers. If the Lagrange multipliers in the augmented Lagrangian function are close to the exact Lagrange multipliers at an optimal solution, the method converges steadily. Since the conventional augmented Lagrangian method uses inaccurate estimated Lagrange multipliers, it sometimes converges slowly. In this paper, we propose a novel augmented Lagrangian method that allows the augmented Lagrangian function and its minimization problem to have variable constraints at each iteration. This allowance enables the new method to get more accurate estimated Lagrange multipliers by exploiting Karush–Kuhn–Tucker points of the subproblems and consequently to converge more efficiently and steadily.

The paper analyzes the rate of local convergence of the augmented Lagrangian method for nonlinear second-order cone optimization problems. Under the constraint nondegeneracy condition and the strong second order sufficient condition, we demonstrate that the sequence of iterate points generated by the augmented Lagrangian method locally converges to a local minimizer at a linear rate, whose ratio constant is proportional to 1/τ with penalty parameter τ not less than a threshold \(\hat{\tau}>0\) . Importantly and interestingly enough, the analysis does not require the strict complementarity condition.

This paper is concerned with a novel deep learning method for variational problems with essential boundary conditions. To this end, we first reformulate the original problem into a minimax problem corresponding to a feasible augmented Lagrangian, which can be solved by the augmented Lagrangian method in an infinite dimensional setting. Based on this, by expressing the primal and dual variables with two individual deep neural network functions, we present an augmented Lagrangian deep learning method for which the parameters are trained by the stochastic optimization method together with a projection technique. Compared to the traditional penalty method, the new method admits two main advantages: i) the choice of the penalty parameter is flexible and robust, and ii) the numerical solution is more accurate in the same magnitude of computational cost. As typical applications, we apply the new approach to solve elliptic problems and (nonlinear) eigenvalue problems with essential boundary conditions, and numerical experiments are presented to show the effectiveness of the new method.

The augmented Lagrangian method (ALM) has been widely used to solve nonlinear programming problems. Although there has been extensive research in the algorithmic aspect, the box constraints are typically preserved during the minimization of the augmented Lagrangian function. In this paper, we propose an equivalent reformulation of the problem by splitting the variables in the constraint. We then establish the augmented Lagrangian function based on projection operators to propose an ALM framework, where the subproblem is an unconstrained optimization problem. For the ALM subproblem, we utilize efficient techniques to approximate the Hessian matrix of the augmented Lagrangian function and design an inexact semismooth Newton method as the solver. We also provide convergence and complexity properties of the algorithm. Finally, we conduct numerical experiments and comparisons with the interior point and sequential quadratic programming methods on nonlinear programming instances from the CUTEr (constrained and unconstrained testing environment) dataset. The results demonstrate the effectiveness of the proposed ALM.

We investigate in this paper global convergence properties of the augmented Lagrangian method for nonlinear semidefinite programming (NLSDP). Four modified augmented Lagrangian methods for solving NLSDP based on different algorithmic strategies are proposed. Possibly infeasible limit points of the proposed methods are characterized. It is proved that feasible limit points that satisfy the Mangasarian-Fromovitz constraint qualification are KKT points of NLSDP without requiring the boundedness condition of the multipliers. Preliminary numerical results are reported to compare the performance of the modified augmented Lagrangian methods.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

The augmented Lagrangian method (ALM) is one of the most successful first-order methods for convex programming with linear equality constraints. To solve the two-block separable convex minimization problem, we always use the parallel splitting ALM method. In this paper, we will show that no matter how small the step size and the penalty parameter are, the convergence of the parallel splitting ALM is not guaranteed. We propose a new convergent parallel splitting ALM (PSALM), which is the regularizing ALM’s minimization subproblem by some simple proximal terms. In application this new PSALM is used to solve video background extraction problems and our numerical results indicate that this new PSALM is efficient.

Many combinatorial optimization problems and engineering problems can be modeled as boolean quadratic programming (BQP) problems. In this paper, two augmented Lagrangian methods (ALM) are discussed for the solution of BQP problems based on a class of continuous functions. After convexification, the BQP is reformulated as an equivalent augmented Lagrangian function, and then solved by two ALM algorithms. Within this ALM algorithm, L-BFGS is called for the solution of unconstrained nonlinear programming problem. Experiments are performed on max-cut problem, 0–1 quadratic knapsack problem and image deconvolution, which indicate that ALM method is promising for solving large scale BQP by the quality of near optimal solution with low computational time.

The objective of this work is to develop a general methodology for determination of the optimal freshwater inflows into bays and estuaries to balance freshwater demands with the harvest of various types of estuarine resources. The methodology is based upon solving a large‐scale nonlinear programming problem formulated in an optimal‐control framework. Constraints of the optimization problem include regression equations for harvest of the various species that express fishery harvest as a function of the quantity of freshwater inflow. The stochastic element of the problem (i.e. the uncertainty associated with the regression equations for harvest) is considered by expressing constraints in a chance‐constrained formulation. A nonlinear programming optimizer is interfaced with a hydrodynamic transport model to implicitly solve the hydrodynamic‐salinity constraint equations for salinity levels. An augmented Lagrangian method is introduced to incorporate the salinity constraints into the objective so that the problem size for the optimizer is significantly reduced. A computer model OPTFLOW has been developed by interfacing a simulator for the hydrodynamic‐salinity (HYD‐SAL) with a nonlinear optimizer (GRG2) to apply the methodology by the present writers in an accompanying companion paper. An efficient approximation scheme is developed for evaluation of the objective function and its reduced gradient to reduce the computational effort dramatically. The new methodology can provide a very useful tool for decision makers to quantitatively analyze various water‐management strategies.

Dynamical systems approaches to constrained optimization often rely on a penalization term to reach feasible points, at the cost of slower convergence. However, we show one can construct a discrete-time system that maintains the same convergence guarantees without requiring such penalization term. We demonstrate that the sequential homotopy method, namely taking projected backward Euler steps on a projected gradient/anti-gradient augmented Lagrangian flow, matches with the classical augmented Lagrangian method without the multiplier estimate update. Then, we introduce a time-scaled flow and provide an interpretation of augmented Lagrangian methods as discrete-time dynamical systems. This approach inspires a simple yet effective method for nonlinear programming. We report on numerical results for equality-constrained problems.