Proximal Augmented Lagrangian Research Articles

A Corrected Inexact Proximal Augmented Lagrangian Method with a Relative Error Criterion for a Class of Group-Quadratic Regularized Optimal Transport Problems

A Corrected Inexact Proximal Augmented Lagrangian Method with a Relative Error Criterion for a Class of Group-Quadratic Regularized Optimal Transport Problems

QPALM-OCP: A Newton-Type Proximal Augmented Lagrangian Solver Tailored for Quadratic Programs Arising in Model Predictive Control

QPALM-OCP: A Newton-Type Proximal Augmented Lagrangian Solver Tailored for Quadratic Programs Arising in Model Predictive Control

On proximal augmented Lagrangian based decomposition methods for dual block-angular convex composite programming problems

On proximal augmented Lagrangian based decomposition methods for dual block-angular convex composite programming problems

A Proximal Augmented Lagrangian Method for Linearly Constrained Nonconvex Composite Optimization Problems

A Proximal Augmented Lagrangian Method for Linearly Constrained Nonconvex Composite Optimization Problems

Convex and Nonconvex Risk-Based Linear Regression at Scale

The value at risk (VaR) and the conditional value at risk (CVaR) are two popular risk measures to hedge against the uncertainty of data. In this paper, we provide a computational toolbox for solving high-dimensional sparse linear regression problems under either VaR or CVaR measures, the former being nonconvex and the latter convex. Unlike the empirical risk (neutral) minimization models in which the overall losses are decomposable across data, the aforementioned risk-sensitive models have nonseparable objective functions so that typical first order algorithms are not easy to scale. We address this scaling issue by adopting a semismooth Newton-based proximal augmented Lagrangian method of the convex CVaR linear regression problem. The matrix structures of the Newton systems are carefully explored to reduce the computational cost per iteration. The method is further embedded in a majorization–minimization algorithm as a subroutine to tackle the nonconvex VaR-based regression problem. We also discuss an adaptive sieving strategy to iteratively guess and adjust the effective problem dimension, which is particularly useful when a solution path associated with a sequence of tuning parameters is needed. Extensive numerical experiments on both synthetic and real data demonstrate the effectiveness of our proposed methods. In particular, they are about 53 times faster than the commercial package Gurobi for the CVaR-based sparse linear regression with 4,265,669 features and 16,087 observations. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Funding: This work was supported in part by the NSF, the Division of Computing and Communication Foundations [Grant 2153352], the National Natural Science Foundation of China [Grant 12271187], and the Hong Kong Research Grant Council [Grant 15304019]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.1282 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0012 ) at ( http://dx.doi.org/10.5281/zenodo.7483279 ).

An accelerated inexact dampened augmented Lagrangian method for linearly-constrained nonconvex composite optimization problems

This paper proposes and analyzes an accelerated inexact dampened augmented Lagrangian (AIDAL) method for solving linearly-constrained nonconvex composite optimization problems. Each iteration of the AIDAL method consists of: (i) inexactly solving a dampened proximal augmented Lagrangian (AL) subproblem by calling an accelerated composite gradient (ACG) subroutine; (ii) applying a dampened and under-relaxed Lagrange multiplier update; and (iii) using a novel test to check whether the penalty parameter of the AL function should be increased. Under several mild assumptions involving the dampening factor and the under-relaxation constant, it is shown that the AIDAL method generates an approximate stationary point of the constrained problem in $$\mathcal{O}(\varepsilon ^{-5/2}\log \varepsilon ^{-1})$$ iterations of the ACG subroutine, for a given tolerance $$\varepsilon >0$$ . Numerical experiments are also given to show the computational efficiency of the proposed method.

Model predictive optimization for imitation learning from demonstrations

“Motion generation by imitating” enables a robot to generate its trajectory in a new environment. Research works on dynamic movement primitives (DMP) has reported promising results, with good imitation effect and convergence to the target. However, DMP still has issues such as learning from multiple demonstrations for different initial conditions and achieving obstacle avoidance considering the distribution and motion of obstacles. One of the effective solutions is combining DMP and model predictive control (MPC). The imitation process was transformed into a receding horizon planning procedure, letting the robot to learn more from nearer demonstrations. It is solved as an optimization problem with obstacles modeled as constraints. However, its drawback includes the heavy computation burden, which can be even aggravated in a multi-obstacle scenario where complicated constraints occur. Thus, in this paper, we propose an enhanced MPDMP+ method that combines the advantages of MPC with potential function for both multi-demonstration imitation and multi-obstacle avoidance effect. A proximal augmented Lagrangian method is proposed to solve the optimization problem. This proposed method has a faster convergence rate and small errors. We conducted the simulation and robot experiments for imitation learning for obstacle avoidance scenarios. Our results illustrate the superior performance of the proposed method.

Iteration Complexity of an Inner Accelerated Inexact Proximal Augmented Lagrangian Method Based on the Classical Lagrangian Function

This paper establishes the iteration complexity of an inner accelerated inexact proximal augmented Lagrangian (IAIPAL) method for solving linearly constrained smooth nonconvex composite optimization problems that is based on the classical augmented Lagrangian (AL) function. More specifically, each IAIPAL iteration consists of inexactly solving a proximal AL subproblem by an accelerated composite gradient (ACG) method followed by a classical Lagrange multiplier update. Under the assumption that the domain of the composite function is bounded and the problem has a Slater point, it is shown that IAIPAL generates an approximate stationary solution in ACG iterations where is a tolerance for both stationarity and feasibility. Moreover, the above bound is derived without assuming that the initial point is feasible. Finally, numerical results are presented to demonstrate the strong practical performance of IAIPAL.

Iteration Complexity of a Proximal Augmented Lagrangian Method for Solving Nonconvex Composite Optimization Problems with Nonlinear Convex Constraints

This paper proposes and analyzes a proximal augmented Lagrangian (NL-IAPIAL) method for solving constrained nonconvex composite optimization problems, where the constraints are smooth and convex with respect to the order given by a closed convex cone. Each NL-IAPIAL iteration consists of inexactly solving a proximal augmented Lagrangian subproblem by an accelerated composite gradient method followed by a Lagrange multiplier update. Under some mild assumptions, a complexity bound for NL-IAPIAL to obtain an approximate stationary solution of the problem is also derived. Numerical experiments are also given to illustrate the computational efficiency of the proposed method. Funding: This work was supported by the Natural Sciences and Engineering Research Council of Canada [Grant PGSD3-516700-2018]; Conselho Nacional de Desenvolvimento Científico e Tecnológico [Grant 312559/2019-4]; Fundação de Amparo à Pesquisa do Estado de Goiás; Office of Naval Research [Grant N00014-18-1-2077]; Exascale Computing Project [Grant 17-SC-20-SC]; Air Force Office of Scientific Research [Grant FA9550-22-1-0088]; UT-Battelle, LLC [Grant DE-AC05-00OR22725].

QPPAL: A Two-phase Proximal Augmented Lagrangian Method for High-dimensional Convex Quadratic Programming Problems

In this article, we aim to solve high-dimensional convex quadratic programming (QP) problems with a large number of quadratic terms, linear equality, and inequality constraints. To solve the targeted QP problem to a desired accuracy efficiently, we consider the restricted-Wolfe dual problem and develop a two-phase Proximal Augmented Lagrangian method (QPPAL), with Phase I to generate a reasonably good initial point to warm start Phase II to obtain an accurate solution efficiently. More specifically, in Phase I, based on the recently developed symmetric Gauss-Seidel (sGS) decomposition technique, we design a novel sGS-based semi-proximal augmented Lagrangian method for the purpose of finding a solution of low to medium accuracy. Then, in Phase II, a proximal augmented Lagrangian algorithm is proposed to obtain a more accurate solution efficiently. Extensive numerical results evaluating the performance of QPPAL against existing state-of-the-art solvers Gurobi, OSQP, and QPALM are presented to demonstrate the high efficiency and robustness of our proposed algorithm for solving various classes of large-scale convex QP problems. The MATLAB implementation of the software package QPPAL is available at https://blog.nus.edu.sg/mattohkc/softwares/qppal/ .

QPALM: a proximal augmented lagrangian method for nonconvex quadratic programs

We propose QPALM, a nonconvex quadratic programming (QP) solver based on the proximal augmented Lagrangian method. This method solves a sequence of inner subproblems which can be enforced to be strongly convex and which therefore admit a unique solution. The resulting steps are shown to be equivalent to inexact proximal point iterations on the extended-real-valued cost function, which allows for a fairly simple analysis where convergence to a stationary point at an \(R\)-linear rate is shown. The QPALM algorithm solves the subproblems iteratively using semismooth Newton directions and an exact linesearch. The former can be computed efficiently in most iterations by making use of suitable factorization update routines, while the latter requires the zero of a monotone, one-dimensional, piecewise affine function. QPALM is implemented in open-source C code, with tailored linear algebra routines for the factorization in a self-written package LADEL. The resulting implementation is shown to be extremely robust in numerical simulations, solving all of the Maros-Meszaros problems and finding a stationary point for most of the nonconvex QPs in the Cutest test set. Furthermore, it is shown to be competitive against state-of-the-art convex QP solvers in typical QPs arising from application domains such as portfolio optimization and model predictive control. As such, QPALM strikes a unique balance between solving both easy and hard problems efficiently.

On a primal-dual Newton proximal method for convex quadratic programs

This paper introduces QPDO, a primal-dual method for convex quadratic programs which builds upon and weaves together the proximal point algorithm and a damped semismooth Newton method. The outer proximal regularization yields a numerically stable method, and we interpret the proximal operator as the unconstrained minimization of the primal-dual proximal augmented Lagrangian function. This allows the inner Newton scheme to exploit sparse symmetric linear solvers and multi-rank factorization updates. Moreover, the linear systems are always solvable independently from the problem data and exact linesearch can be performed. The proposed method can handle degenerate problems, provides a mechanism for infeasibility detection, and can exploit warm starting, while requiring only convexity. We present details of our open-source C implementation and report on numerical results against state-of-the-art solvers. QPDO proves to be a simple, robust, and efficient numerical method for convex quadratic programming.

An augmented Lagrangian method with constraint generation for shape-constrained convex regression problems

Shape-constrained convex regression problem deals with fitting a convex function to the observed data, where additional constraints are imposed, such as component-wise monotonicity and uniform Lipschitz continuity. This paper provides a unified framework for computing the least squares estimator of a multivariate shape-constrained convex regression function in $${\mathbb {R}}^d$$ . We prove that the least squares estimator is computable via solving an essentially constrained convex quadratic programming (QP) problem with $$(d+1)n$$ variables, $$n(n-1)$$ linear inequality constraints and n possibly non-polyhedral inequality constraints, where n is the number of data points. To efficiently solve the generally very large-scale convex QP, we design a proximal augmented Lagrangian method (proxALM) whose subproblems are solved by the semismooth Newton method. To further accelerate the computation when n is huge, we design a practical implementation of the constraint generation method such that each reduced problem is efficiently solved by our proposed proxALM. Comprehensive numerical experiments, including those in the pricing of basket options and estimation of production functions in economics, demonstrate that our proposed proxALM outperforms the state-of-the-art algorithms, and the proposed acceleration technique further shortens the computation time by a large margin.

An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

We propose an inexact proximal augmented Lagrangian framework with explicit inner problem termination rule for composite convex optimization problems. We consider arbitrary linearly convergent inner solver including in particular stochastic algorithms, making the resulting framework more scalable facing the ever-increasing problem dimension. Each subproblem is solved inexactly with an explicit and self-adaptive stopping criterion, without requiring to set an a priori target accuracy. When the primal and dual domain are bounded, our method achieves \(O(1/\sqrt{\epsilon })\) and \(O(1/{\epsilon })\) complexity bound in terms of number of inner solver iterations, respectively for the strongly convex and non-strongly convex case. Without the boundedness assumption, only logarithm terms need to be added and the above two complexity bounds increase respectively to \({\tilde{O}}(1/\sqrt{\epsilon })\) and \({\tilde{O}}(1/{\epsilon })\), which hold both for obtaining \(\epsilon \)-optimal and \(\epsilon \)-KKT solution. Within the general framework that we propose, we also obtain \({\tilde{O}}(1/{\epsilon })\) and \({\tilde{O}}(1/{\epsilon ^2})\) complexity bounds under relative smoothness assumption on the differentiable component of the objective function. We show through theoretical analysis as well as numerical experiments the computational speedup possibly achieved by the use of randomized inner solvers for large-scale problems.

Complexity of Proximal Augmented Lagrangian for Nonconvex Optimization with Nonlinear Equality Constraints

We analyze worst-case complexity of a Proximal augmented Lagrangian (Proximal AL) framework for nonconvex optimization with nonlinear equality constraints. When an approximate first-order (second-order) optimal point is obtained in the subproblem, an $$\epsilon $$ first-order (second-order) optimal point for the original problem can be guaranteed within $${\mathcal {O}}(1/ \epsilon ^{2 - \eta })$$ outer iterations (where $$\eta $$ is a user-defined parameter with $$\eta \in [0,2]$$ for the first-order result and $$\eta \in [1,2]$$ for the second-order result) when the proximal term coefficient $$\beta $$ and penalty parameter $$\rho $$ satisfy $$\beta = {\mathcal {O}}(\epsilon ^\eta )$$ and $$\rho = \varOmega (1/\epsilon ^\eta )$$ , respectively. We also investigate the total iteration complexity and operation complexity when a Newton-conjugate-gradient algorithm is used to solve the subproblems. Finally, we discuss an adaptive scheme for determining a value of the parameter $$\rho $$ that satisfies the requirements of the analysis.

Distributed Composite Optimization Over Relay-Assisted Networks

This article studies the distributed composite optimization problem over relay-assisted networks (DCOP-RNs). By combining the local primal–dual updates with a data fusion strategy, an effective distributed algorithm is developed to solve the DCOP-RN. Compared with the existing linearized alternating direction method of multipliers (ADMMs), this article provides a novel interpretation of the proposed algorithm. More specifically, it is shown that the proposed algorithm can be interpreted as a simplified proximal augmented Lagrangian method . Within this framework, a simplified convergence analysis of the algorithm is conducted, based on which the convergence to an optimal solution to the DCOP-RN is proved. Further, taking into account the limited bandwidth in real networks, this article proposes a quantized distributed algorithm to solve the DCOP-RN. Especially, a relationship between the quantization resolution and the convergence accuracy of the algorithm is established. Finally, simulation examples verify the theoretical results.

On the equivalence of inexact proximal ALM and ADMM for a class of convex composite programming

In this paper, we show that for a class of linearly constrained convex composite optimization problems, an (inexact) symmetric Gauss–Seidel based majorized multi-block proximal alternating direction method of multipliers (ADMM) is equivalent to an inexact proximal augmented Lagrangian method. This equivalence not only provides new perspectives for understanding some ADMM-type algorithms but also supplies meaningful guidelines on implementing them to achieve better computational efficiency. Even for the two-block case, a by-product of this equivalence is the convergence of the whole sequence generated by the classic ADMM with a step-length that exceeds the conventional upper bound of $$(1+\sqrt{5})/2$$ , if one part of the objective is linear. This is exactly the problem setting in which the very first convergence analysis of ADMM was conducted by Gabay and Mercier (Comput Math Appl 2(1):17–40, 1976), but, even under notably stronger assumptions, only the convergence of the primal sequence was known. A collection of illustrative examples are provided to demonstrate the breadth of applications for which our results can be used. Numerical experiments on solving a large number of linear and convex quadratic semidefinite programming problems are conducted to illustrate how the theoretical results established here can lead to improvements on the corresponding practical implementations.

Online Optimization as a Feedback Controller: Stability and Tracking

This paper develops and analyzes feedback-based online optimization methods to regulate the output of a linear time invariant (LTI) dynamical system to the optimal solution of a time-varying convex optimization problem. The design of the algorithm is based on continuous-time primal-dual dynamics, properly modified to incorporate feedback from the LTI dynamical system, applied to a proximal augmented Lagrangian function. The resultant closed-loop algorithm tracks the solution of the time-varying optimization problem without requiring knowledge of (time varying) disturbances in the dynamical system. The analysis leverages integral quadratic constraints to provide linear matrix inequality (LMI) conditions that guarantee global exponential stability and bounded tracking error. Analytical results show that under a sufficient time-scale separation between the dynamics of the LTI dynamical system and the algorithm, the LMI conditions can be always satisfied. This paper further proposes a modified algorithm that can track an approximate solution trajectory of the constrained optimization problem under less restrictive assumptions. As an illustrative example, the proposed algorithms are showcased for power transmission systems, to compress the time scales between secondary and tertiary control, and allow to simultaneously power rebalancing and tracking of the DC optimal power flow points.

The Proximal Augmented Lagrangian Method for Nonsmooth Composite Optimization

We study a class of optimization problems in which the objective function is given by the sum of a differentiable but possibly nonconvex component and a nondifferentiable convex regularization term. We introduce an auxiliary variable to separate the objective function components and utilize the Moreau envelope of the regularization term to derive the proximal augmented Lagrangian $-$ a continuously differentiable function obtained by constraining the augmented Lagrangian to the manifold that corresponds to the explicit minimization over the variable in the nonsmooth term. The continuous differentiability of this function with respect to both primal and dual variables allows us to leverage the method of multipliers (MM) to compute optimal primal-dual pairs by solving a sequence of differentiable problems. The MM algorithm is applicable to a broader class of problems than proximal gradient methods and it has stronger convergence guarantees and a more refined step-size update rules than the alternating direction method of multipliers. These features make it an attractive option for solving structured optimal control problems. We also develop an algorithm based on the primal-descent dual-ascent gradient method and prove global (exponential) asymptotic stability when the differentiable component of the objective function is (strongly) convex and the regularization term is convex. Finally, we identify classes of problems for which the primal-dual gradient flow dynamics are convenient for distributed implementation and compare/contrast our framework to the existing approaches.

PAL-Hom method for QP and an application to LP

In this paper, a proximal augmented Lagrangian homotopy (PAL-Hom) method for solving convex quadratic programming problems is proposed. This method takes the proximal augmented Lagrangian method as the outer iteration. To solve the proximal augmented Lagrangian subproblems, a homotopy method is presented as the inner iteration. The homotopy method tracks the piecewise-linear solution path of a parametric quadratic programming problem whose start problem takes an approximate solution as its solution and the target problem is the subproblem to be solved. To improve the performance of the homotopy method, the accelerated proximal gradient method is used to obtain a fairly good approximate solution that implies a good prediction of the optimal active set. Moreover, a sorting technique for the Cholesky factor update as well as an $$\varepsilon $$ -relaxation technique for checking primal-dual feasibility and correcting the active sets are presented to improve the efficiency and robustness of the homotopy method. Simultaneously, a proximal-point-based AL-Hom method which is shown to converge in finite number of steps, is applied to linear programming. Numerical experiments on randomly generated problems and the problems from the CUTEr and Netlib test collections, support vector machines (SVMs) and contact problems of elasticity demonstrate that PAL-Hom is faster than the active-set methods and the parametric active set methods and is competitive to the interior-point methods and the specialized algorithms designed for specific models (e.g., sequential minimal optimization method for SVMs).