Augmented Lagrangian Framework Research Articles

Total Fractional-Order Variation-Based Constraint Image Deblurring Problem

When deblurring an image, ensuring that the restored intensities are strictly non-negative is crucial. However, current numerical techniques often fail to consistently produce favorable results, leading to negative intensities that contribute to significant dark regions in the restored images. To address this, our study proposes a mathematical model for non-blind image deblurring based on total fractional-order variational principles. Our proposed model not only guarantees strictly positive intensity values but also imposes limits on the intensities within a specified range. By removing negative intensities or constraining them within the prescribed range, we can significantly enhance the quality of deblurred images. The key concept in this paper involves converting the constrained total fractional-order variational-based image deblurring problem into an unconstrained one through the introduction of the augmented Lagrangian method. To facilitate this conversion and improve convergence, we describe new numerical algorithms and introduce a novel circulant preconditioned matrix. This matrix effectively overcomes the slow convergence typically encountered when using the conjugate gradient method within the augmented Lagrangian framework. Our proposed approach is validated through computational tests, demonstrating its effectiveness and viability in practical applications.

ℓ0 Trend Filtering

The [Formula: see text] trend filtering ([Formula: see text]-TF) is a new effective tool for nonparametric regression with the power of automatic knot detection in function values or derivatives. It overcomes the drawback of [Formula: see text]-TF that is known to have bias issues. To solve the [Formula: see text]-TF problem, we propose an alternating minimization induced active set (AMIAS) search method based on the necessary optimality conditions derived from an augmented Lagrangian framework. The proposed method takes full advantage of the primal and dual variables with complementary supports, and decouples the high-dimensional problem into two subsystems on the active and inactive sets, respectively. A sequential AMIAS algorithm with warm start initialization is developed for efficient determination of the cardinality parameter, along with the output of solution paths. Theoretically, the oracle estimator of [Formula: see text]-TF is justified to behave like regression splines under the continuous time setting with mild conditions. Our numerical experiments include simulation studies for comparing [Formula: see text]-TF to [Formula: see text]-TF and free-knot splines on several synthetic examples, and a real data application of time series segmentation on Hong Kong PM2.5 indexes. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms – Continuous. Funding: This work was supported in part by Hong Kong General Research Fund [No. 17306519]. C. Wen’s research is partially supported by National Science Foundation of China [12171449] and Fundamental Research Funds for the Central Universities [WK3470000027, YD2040002019]. X. Wang’s research is partially supported by National Natural Science Foundation of China [Grants 72171216, 12231017, 71921001, and 71991474], and the National Key R&D Program of China [No. 2022YFA1003803]. Supplemental Material: The e-companion is available at https://doi.org/10.1287/ijoc.2021.0313 . The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2021.0313 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0313 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Wind turbine blade bearing fault detection with Bayesian and Adaptive Kalman Augmented Lagrangian Algorithm

As a critically supporting and rotational component for wind turbines, blade bearings need special health monitoring for safe operation in actual industrial conditions. One of the main difficulties of the wind turbine blade bearing condition monitoring is noisy signals generated under fluctuating slow speed with heavy loads. This is because blade bearing rotation speed is influenced by blade flipping and external disturbances, and this influence is time-varying. This paper proposes a new method, Bayesian and Adapted Kalman Augmented Lagrangian (BAKAL), to filter the signal under this time-varying condition. The new method uses a two-step search (coarse and fine search) to deal with the filtering process based on Bayesian Augmented Lagrangian (BAL) framework. In addition, both linear and nonlinear effects and their sparsity are considered for model construction. Finally, the smearing problem in the frequency spectrum is dealt with through signal resample in the order domain for superior performance of fault diagnosis. The proposed BAKAL algorithm is strictly validated in several experiments under approximately fixed speed and variable speed within the condition of heavy loadings. The experiments use an industrial and rotational wind turbine blade bearing with natural defects, which has been served in an actual wind power plant for over 15 years. The experimental results demonstrate the effectiveness of the proposed method.

Augmented Lagrangian approach to deriving discontinuous Galerkin methods for nonlinear elasticity problems

AbstractWe use the augmented Lagrangian formalism to derive discontinuous Galerkin (DG) formulations for problems in nonlinear elasticity. In elasticity, stress is typically a symmetric function of strain, leading to symmetric tangent stiffness matrices in Newton's method when conforming finite elements are used for discretization. By use of the augmented Lagrangian framework, we can also obtain symmetric tangent stiffness matrices in DG methods. We suggest two different approaches and give examples from plasticity and from large deformation hyperelasticity.

A manifold inexact augmented Lagrangian method for nonsmooth optimization on Riemannian submanifolds in Euclidean space

Abstract We develop a manifold inexact augmented Lagrangian framework to solve a family of nonsmooth optimization problem on Riemannian submanifold embedding in Euclidean space, whose objective function is the sum of a smooth function (but possibly nonconvex) and a nonsmooth convex function in Euclidean space. By utilizing the Moreau envelope, we get a smoothing Riemannian minimization subproblem at each iteration of the proposed method. Consequentially, each iteration subproblem is solved by a Riemannian Barzilai–Borwein gradient method. Theoretically, the convergence to critical point of the proposed method is established under some mild assumptions. Numerical experiments on compressed modes problems in physic and sparse principal component analysis demonstrate that the proposed method is a competitive method compared with some state-of-the-art methods.

An Augmented Lagrangian Method Exploiting an Active-Set Strategy and Second-Order Information

In this paper, we consider nonlinear optimization problems with nonlinear equality constraints and bound constraints on the variables. For the solution of such problems, many augmented Lagrangian methods have been defined in the literature. Here, we propose to modify one of these algorithms, namely ALGENCAN by Andreani et al., in such a way to incorporate second-order information into the augmented Lagrangian framework, using an active-set strategy. We show that the overall algorithm has the same convergence properties as ALGENCAN and an asymptotic quadratic convergence rate under suitable assumptions. The numerical results confirm that the proposed algorithm is a viable alternative to ALGENCAN with greater robustness.

Distributed Primal-Dual Method for Convex Optimization With Coupled Constraints

Distributed primal-dual methods have been widely used for solving large-scale constrained optimization problems. The majority of existing results focus on the problems with decoupled constraints. Some recent works have studied the problems subject to separable globally coupled constraints. This paper considers the distributed optimization problems with globally coupled constraints over networks without requiring the separability of the globally coupled constraints. This is made possible by the local estimates of the constraint violations. For solving such a problem, we propose a primal-dual algorithm in the augmented Lagrangian framework, combining the average consensus technique. We first establish a non-ergodic convergence rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1/k)$</tex-math></inline-formula> in terms of the objective residual for solving a distributed constrained convex optimization problem, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> is the iteration counter. Specifically, the global objective function is the aggregate of the local convex and possibly non-smooth costs, and the coupled constraint is the sum of the local linear equality constraints. The numerical results illustrate the performance of the proposed method.

An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

We propose an inexact proximal augmented Lagrangian framework with explicit inner problem termination rule for composite convex optimization problems. We consider arbitrary linearly convergent inner solver including in particular stochastic algorithms, making the resulting framework more scalable facing the ever-increasing problem dimension. Each subproblem is solved inexactly with an explicit and self-adaptive stopping criterion, without requiring to set an a priori target accuracy. When the primal and dual domain are bounded, our method achieves \(O(1/\sqrt{\epsilon })\) and \(O(1/{\epsilon })\) complexity bound in terms of number of inner solver iterations, respectively for the strongly convex and non-strongly convex case. Without the boundedness assumption, only logarithm terms need to be added and the above two complexity bounds increase respectively to \({\tilde{O}}(1/\sqrt{\epsilon })\) and \({\tilde{O}}(1/{\epsilon })\), which hold both for obtaining \(\epsilon \)-optimal and \(\epsilon \)-KKT solution. Within the general framework that we propose, we also obtain \({\tilde{O}}(1/{\epsilon })\) and \({\tilde{O}}(1/{\epsilon ^2})\) complexity bounds under relative smoothness assumption on the differentiable component of the objective function. We show through theoretical analysis as well as numerical experiments the computational speedup possibly achieved by the use of randomized inner solvers for large-scale problems.

Complexity of Proximal Augmented Lagrangian for Nonconvex Optimization with Nonlinear Equality Constraints

We analyze worst-case complexity of a Proximal augmented Lagrangian (Proximal AL) framework for nonconvex optimization with nonlinear equality constraints. When an approximate first-order (second-order) optimal point is obtained in the subproblem, an $$\epsilon $$ first-order (second-order) optimal point for the original problem can be guaranteed within $${\mathcal {O}}(1/ \epsilon ^{2 - \eta })$$ outer iterations (where $$\eta $$ is a user-defined parameter with $$\eta \in [0,2]$$ for the first-order result and $$\eta \in [1,2]$$ for the second-order result) when the proximal term coefficient $$\beta $$ and penalty parameter $$\rho $$ satisfy $$\beta = {\mathcal {O}}(\epsilon ^\eta )$$ and $$\rho = \varOmega (1/\epsilon ^\eta )$$ , respectively. We also investigate the total iteration complexity and operation complexity when a Newton-conjugate-gradient algorithm is used to solve the subproblems. Finally, we discuss an adaptive scheme for determining a value of the parameter $$\rho $$ that satisfies the requirements of the analysis.

On the Nonergodic Convergence Rate of an Inexact Augmented Lagrangian Framework for Composite Convex Programming

In this paper, we consider the linearly constrained composite convex optimization problem, whose objective is a sum of a smooth function and a possibly nonsmooth function. We propose an inexact augmented Lagrangian (IAL) framework for solving the problem. The stopping criterion used in solving the augmented Lagrangian subproblem in the proposed IAL framework is weaker and potentially much easier to check than the one used in most of the existing IAL frameworks/methods. We analyze the global convergence and the nonergodic convergence rate of the proposed IAL framework. Preliminary numerical results are presented to show the efficiency of the proposed IAL framework and the importance of the nonergodic convergence and convergence rate analysis.

An Alternating Augmented Lagrangian method for constrained nonconvex optimization

ABSTRACTWe consider the problem of minimizing a smooth nonconvex function over a structured convex feasible set, that is, defined by two sets of constraints that are easy to treat when considered separately. In order to exploit the structure of the problem, we define an equivalent formulation by duplicating the variables and we consider the augmented Lagrangian of this latter formulation. Following the idea of the Alternating Direction Method of Multipliers (ADMM), we propose an algorithm where a two-blocks decomposition method is embedded within an augmented Lagrangian framework. The peculiarities of the proposed algorithm are the following: (1) the computation of the exact solution of a possibly nonconvex subproblem is not required; (2) the penalty parameter is iteratively updated once an approximated stationary point of the augmented Lagrangian is determined. Global convergence results are stated under mild assumptions and without requiring convexity of the objective function. Although the primary aim of the paper is theoretical, we perform numerical experiments on a nonconvex problem arising in machine learning, and the obtained results show the practical advantages of the proposed approach with respect to classical ADMM.

Research on Optimization of Charge Batch Planning Based on Augmented Lagrangian Relaxation Algorithm

Abstract The steelmaking-continuous casting batch planning plays an important role in the relationship between the Enterprise Resource Planning (ERP) layer and the Manufacturing Execution System (MES) layer. The charge batch planning is one of the important links in the steelmaking-continuous casting batch planning. Efficient optimization of charge batch planning can reduce energy consumption and improve the profit of the enterprise. As the production scale of iron and steel enterprises increases, the number of orders increases exponentially, and the optimization goal becomes more and more complicated, which makes it difficult to ensure the optimization efficiency while ensuring the quality of optimization results. Aiming at this problem, this paper proposes a multi-performance index mixed integer programming model considering actual production, and solves it based on the order decomposition strategy under the linearized Augmented Lagrangian framework. Finally, experiments show that the proposed algorithm can effectively improve the compilation efficiency and quality of the charge batch planning compared with the traditional Lagrangian Relaxation algorithm.

Efficient Dynamic Parallel MRI Reconstruction for the Low-Rank Plus Sparse Model.

The low-rank plus sparse (L+S) decomposition model enables the reconstruction of under-sampled dynamic parallel magnetic resonance imaging (MRI) data. Solving for the low-rank and the sparse components involves non-smooth composite convex optimization, and algorithms for this problem can be categorized into proximal gradient methods and variable splitting methods. This paper investigates new efficient algorithms for both schemes. While current proximal gradient techniques for the L+S model involve the classical iterative soft thresholding algorithm (ISTA), this paper considers two accelerated alternatives, one based on the fast iterative shrinkage-thresholding algorithm (FISTA), and the other with the recent proximal optimized gradient method (POGM). In the augmented Lagrangian (AL) framework, we propose an efficient variable splitting scheme based on the form of the data acquisition operator, leading to simpler computation than the conjugate gradient (CG) approach required by existing AL methods. Numerical results suggest faster convergence of the efficient implementations for both frameworks, with POGM providing the fastest convergence overall and the practical benefit of being free of algorithm tuning parameters.

Augmented Lagrangian Methods for Solving Optimization Problems with Stochastic-Order Constraints

We investigate risk-averse stochastic optimization problems with a risk-shaping constraint in the form of a stochastic-order relation. Both univariate and multivariate orders are considered. We extend the notion of a linear multivariate order, adding flexibility with respect to the controlled portion of the distributions. We propose several methods for the numerical solution of these problems based on augmented Lagrangian framework and analyze their convergence. The methods construct finite-dimensional risk-neutral approximations of the optimization problem whose solutions converge to the solution of the original problem. In the case of univariate stochastic dominance, we explore augmented Lagrangian functionals based on inverse formulations of the stochastic-order constraint. The performance of the methods is compared to other extant numerical methods and shows the numerical advantage of the augmented Lagrangian framework. The proposed numerical approach is particularly successful when applied to problems with multivariate stochastic dominance constraints.

A Distributed Algorithm for Convex Constrained Optimization Under Noise

We present a novel distributed algorithm for convex constrained optimization problems that are subject to noise corruption and uncertainties. The proposed scheme can be classified as a distributed stochastic approximation method, where a unique feature here is that we allow for multiple noise terms to appear in both the computation and communication stages of the distributed iterative process. Specifically, we consider problems that involve multiple agents optimizing a separable convex objective function subject to convex local constraints and linear coupling constraints. This is a richer class of problems compared to those that can be handled by existing distributed stochastic approximation methods which consider only consensus constraints and fewer sources of noise. The proposed algorithm utilizes the augmented Lagrangian (AL) framework, which has been widely used recently to solve deterministic optimization problems in a distributed way. We show that the proposed method generates sequences of primal and dual variables that converge to their respective optimal sets almost surely.

Computational Comparison of Surface Metrics for PDE Constrained Shape Optimization

Abstract We compare surface metrics for shape optimization problems with constraints, consisting mainly of partial differential equations (PDE), from a computational point of view. In particular, classical Laplace–Beltrami type metrics are compared with Steklov–Poincaré type metrics. The test problem is the minimization of energy dissipation of a body in a Stokes flow. We therefore set up a quasi-Newton method on appropriate shape manifolds together with an augmented Lagrangian framework, in order to enable a straightforward integration of geometric constraints for the shape. The comparison is focussed towards convergence behavior as well as effects on the mesh quality during shape optimization.

An augmented Lagrangian contact formulation for frictional discontinuities with the extended finite element method

In this paper, an Uzawa-type augmented Lagrangian contact formulation is presented for modeling frictional discontinuities in the framework of the X-FEM technique. The kinematically nonlinear contact problem is resolved based on an active set strategy to fulfill the Kuhn–Tucker inequalities in the normal direction of contact. The Coulomb’s friction rule is employed to address the stick–slip behavior on the contact interface through a return mapping algorithm in conjunction with a symmetrized (nested) augmented Lagrangian approach. A stabilization algorithm is proposed for the robust imposition of the frictional contact constraints within the proposed augmented Lagrangian framework. Several numerical examples are presented to demonstrate various aspects of the proposed computational algorithm in simulation of the straight, curved and wave-shaped discontinuities.

Sparse Non‐rigid Registration of 3D Shapes

AbstractNon‐rigid registration of 3D shapes is an essential task of increasing importance as commodity depth sensors become more widely available for scanning dynamic scenes. Non‐rigid registration is much more challenging than rigid registration as it estimates a set of local transformations instead of a single global transformation, and hence is prone to the overfitting issue due to underdetermination. The common wisdom in previous methods is to impose an ℓ2‐norm regularization on the local transformation differences. However, the ℓ2‐norm regularization tends to bias the solution towards outliers and noise with heavy‐tailed distribution, which is verified by the poor goodness‐of‐fit of the Gaussian distribution over transformation differences. On the contrary, Laplacian distribution fits well with the transformation differences, suggesting the use of a sparsity prior. We propose a sparse non‐rigid registration (SNR) method with an ℓ1‐norm regularized model for transformation estimation, which is effectively solved by an alternate direction method (ADM) under the augmented Lagrangian framework. We also devise a multi‐resolution scheme for robust and progressive registration. Results on both public datasets and our scanned datasets show the superiority of our method, particularly in handling large‐scale deformations as well as outliers and noise.

Parallel Solution of DDDAS Variational Inference Problems

Abstract Inference problems in dynamically data-driven application systems use physical measurements along with a physical model to estimate the parameters or state of a physical system. Developing parallel algorithms to solve inference problems can improve the process of estimating and predicting the physical state of a system. Solution to inference problems using the variational approach require multiple evaluations of the associated cost function and gradient, where the gradient is defined as the increase/decrease inflection point of the variable between two points. In this paper we present a scalable algorithm based on augmented Lagrangian approach to solve the variational inference problem. The augmented Lagrangian framework facilitates parallel cost function and gradient computations. We show that the methodology is highly scalable with increasing problem size by applying it for the Lorenz-96 model.

Image reconstruction under multiplicative speckle noise using total variation

In this paper, we present a method for reconstructing images or volumes from a partial set of observations, under the Rayleigh distributed multiplicative noise model, which is the appropriate algebraic model in ultrasound (US) imaging. The proposed method performs a variable splitting to introduce an auxiliary variable to serve as the argument of the total variation (TV) regularizer term. Applying the Augmented Lagrangian framework and using an iterative alternating minimization method lead to simpler problems involving TV minimization with a least squares term. The resulting Gauss Seidel scheme is an instance of the Alternating Direction Method of Multipliers (ADMM) method for which convergence is guaranteed. Experimental results show that the proposed method achieves a lower reconstruction error than existing methods.