Convergence Of Alternating Direction Method Of Multipliers Research Articles

A graph decomposition-based approach for the graph-fused lasso

We propose a new algorithm for solving the graph-fused lasso (GFL), a regularized model that operates under the assumption that the signal tends to be locally constant over a predefined graph structure. The proposed method applies a novel decomposition of the objective function for the alternating direction method of multipliers (ADMM) algorithm. While ADMM has been widely used in fused lasso problems, existing works such as the network lasso decompose the objective function into the loss function component and the total variation penalty component. In contrast, based on the graph matching technique in graph theory, we propose a new method of decomposition that separates the objective function into two components, where one component is the loss function plus part of the total variation penalty, and the other component is the remaining total variation penalty. We develop an exact convergence rate of the proposed algorithm by developing a general theory on the local convergence of ADMM. Compared with the network lasso algorithm, our algorithm has a faster exact linear convergence rate (although in the same order as for the network lasso). It also enjoys a smaller computational cost per iteration, thus converges overall faster in most numerical examples.

Distributed restoration for integrated electricity-gas-heating energy systems with an iterative loop scheme

Electricity, gas and heating networks are increasingly coupled to form an integrated energy system (IES), which is beneficial in improving the operational performance of the system during power outages. Generally, IES involves multiple energy operators, making the centralized framework impracticable. To this end, a novel distributed restoration method for IES is proposed in this paper. The IES restoration problem is formulated as a mixed-integer second-order cone programming (MISOCP) model to minimize the total load losses, which considers the coupling effects among the electricity, gas and heating networks. Then, considering the independent management and information privacy of different energy networks, the distributed restoration model is formulated by decomposing shared boundary information using the alternating direction method of multipliers (ADMM). For the sake of the non-convexity caused by binary load state variables, an iterative loop scheme is introduced in ADMM procedure. This allows us to solve a relaxed distributed model in the first stage, and perform iterative loops in the second stage, which can improve the convergence of ADMM and make the distributed solution feasible. Finally, numerical studies are conducted on 33-7-6-node IES and 123-24-32-node IES to validate the effectiveness and feasibility of the proposed method.

The exact worst-case convergence rate of the alternating direction method of multipliers

AbstractRecently, semidefinite programming performance estimation has been employed as a strong tool for the worst-case performance analysis of first order methods. In this paper, we derive new non-ergodic convergence rates for the alternating direction method of multipliers (ADMM) by using performance estimation. We give some examples which show the exactness of the given bounds. We also study the linear and R-linear convergence of ADMM in terms of dual objective. We establish that ADMM enjoys a global linear convergence rate if and only if the dual objective satisfies the Polyak–Łojasiewicz (PŁ) inequality in the presence of strong convexity. In addition, we give an explicit formula for the linear convergence rate factor. Moreover, we study the R-linear convergence of ADMM under two scenarios.

Learning Regionally Decentralized AC Optimal Power Flows With ADMM

One potential future for the next generation of smart grids is the use of decentralized optimization algorithms and secured communications for coordinating renewable generation (e.g., wind/solar), dispatchable devices (e.g., coal/gas/nuclear generations), demand response, battery & storage facilities, and topology optimization. The Alternating Direction Method of Multipliers (ADMM) has been widely used in the community to address such decentralized optimization problems and, in particular, the AC Optimal Power Flow (AC-OPF). This paper studies how machine learning may help in speeding up the convergence of ADMM for solving AC-OPF. It proposes a novel decentralized machine-learning approach, namely ML-ADMM, where each agent uses deep learning to learn the consensus parameters on the coupling branches. The paper also explores the idea of learning only from ADMM runs that exhibit high-quality convergence properties, and proposes filtering mechanisms to select these runs. Experimental results on test cases based on the French system demonstrate the potential of the approach in speeding up the convergence of ADMM significantly.

Linear Convergence of ADMM Under Metric Subregularity for Distributed Optimization

Alternating Direction Method of Multipliers (ADMM) has seen much progress in the literature in recent years. Usually, linear convergence of distributed ADMM is proved under either second-order conditions or strong convexity. When both conditions fail, an alternative is expected to play the role. In this paper, it is shown that distributed ADMM can achieve a linear convergence rate by imposing metric subregularity on a defined mapping. Furthermore, it is proved that both second-order conditions and strong convexity imply metric subregularity under reasonable conditions, e.g. the cost functions being twice continuously differentiable in a neighborhood. In addition, nonergodic convergence rates are presented as well for problems under consideration. Finally, simulation results are carried out to illustrate the efficiency of the proposed algorithm.

Over-relaxed multi-block ADMM algorithms for doubly regularized support vector machines

As a classical machine learning model, support vector machine (SVM) has attracted much attention due to its rigorous theoretical foundation and powerful discriminative performance. The doubly regularized SVM (DRSVM) is an important variant of SVM based on elastic-net regularization, which considers both the sparsity and stability of the model. To tackle the problems of explosive increases in data dimensions and data volume, the alternating direction method of multipliers (ADMM) algorithm can be used to train the DRSVM model. ADMM is an effective iterative algorithm for solving convex optimization problems by decomposing a large issue into a series of solvable subproblems, which is also well suited for distributed computing. However, lack of guaranteed convergence and slow convergence rate are two critical limitations of ADMM. In this paper, a 3-block ADMM algorithm based on the over-relaxation technique is proposed to accelerate DRSVM training, namely, the over-relaxed DRSVM (O-RDRSVM). The main strategy of the over-relaxation technique is to further append the information from the previous iteration to the next iteration to improve the convergence of ADMM. We also propose a distributed version of O-RDRSVM to handle parallel and distributed computing faster, termed DO-RDRSVM. Moreover, we develop a fast O-RDRSVM algorithm (FO-RDRSVM) and a fast DO-RDRSVM algorithm (FDO-RDRSVM), which further reduce the computational cost of O-RDRSVM and DO-RDRSVM by employing the matrix inversion lemma. The convergence analyses ensure the effectiveness of our algorithms for DRSVM training. Finally, extensive experiments on public datasets demonstrate the advantages of our algorithms in terms of convergence rate and training time while maintaining accuracy and sparsity comparable to those of previous works.

Passivity-based Analysis of the ADMM Algorithm for Constraint-Coupled Optimization

We propose a novel, system theoretic analysis of the Alternating Direction Method of Multipliers (ADMM) applied to a convex constraint-coupled optimization problem. The resulting algorithm can be interpreted as a linear, discrete-time dynamical system (modeling the multiplier ascent update) in closed loop with a static nonlinearity (representing the minimization of the augmented Lagrangian). When expressed in suitable coordinates, we prove that the discrete-time linear dynamical system has a discrete positive-real transfer function and is interconnected in closed loop with a static, passive nonlinearity. This readily shows that the origin is a stable equilibrium for the feedback interconnection. Finally, we also show global asymptotic stability of the origin for the closed-loop system and, thus, global asymptotic convergence of ADMM to the optimal solution of the optimization problem.

Effective phase retrieval of sparse signals with convergence guarantee

Phase retrieval, as a representative nonlinear inverse problem, is of increasing interest recently for its broad applications in imaging and engineering. Prediction of the signal amounts to solving a nonconvex optimization problem, which is generally NP-hard to solve. Collecting more measurements or providing prior information of the unknown are two often-seen strategies to facilitate the problem. In this paper, we propose an efficient and effective algorithm to solve phase retrieval with certain prior of the signal, in particular the signal itself is sparse in the natural basis. By formulating phase retrieval (PR) problem in the splitting form, we propose ADMM (alternating direction method of multipliers) with convergence guarantee to tackle the resulting nonconvex problem. Even though the algorithm is not new and there also exist several works on the convergence of ADMM, we clarify that our formulation for phase retrieval is not a specific application example of their investigated models. Hence, we investigate the convergence of our algorithm and show that ADMM converges a stationary point when penalty parameter ρ is large enough. It is observed that the recovery performance degrades when the penalty parameter increases. For better performance, we propose a practical scheme of tuning the penalty parameter ρ. Demonstration of the superior recovery performance on sparse phase retrieval (SPR) is conducted and it shows that our method numerically infers near-exact solution without providing good initialization. Our proposed method distinguishes itself from other existing competitive algorithms in two aspects: (a) the initialization is much less crucial for the algorithmic success than other compared methods; (b) the sampling complexity for the phase transition of recovery is much markedly reduced than other existing methods.

Online Decentralized Decision Making With Inequality Constraints: An ADMM approach

We discuss an online decentralized decision making problem where the agents are coupled with affine inequality constraints. Alternating Direction Method of Multipliers (ADMM) is used as the computation engine and we discuss the convergence of the algorithm in an online setting. To be specific, when decisions have to be made sequentially with a fixed time step, there might not be enough time for the ADMM to converge before the scenario changes and the decision needs to be updated. In this case, a suboptimal solution is employed and we analyze the optimality gap given the convergence condition. Moreover, in many cases, the decision making problem changes gradually over time. We propose a warm-start scheme to accelerate the convergence of ADMM and analyze the benefit of the warm-start. The proposed method is demonstrated in a decentralized multiagent control barrier function problem with simulation.

Linearized symmetric multi-block ADMM with indefinite proximal regularization and optimal proximal parameter

The proximal term plays a significant role in the literature of proximal Alternating Direction Method of Multipliers (ADMM), since (positive-definite or indefinite) proximal terms can promote convergence of ADMM and further simplify the involved subproblems. However, an overlarge proximal parameter decelerates the convergence numerically, though the convergence can be established with it. In this paper, we thus focus on a Linearized Symmetric ADMM (LSADMM) with proper proximal terms for solving a family of multi-block separable convex minimization models, and we determine an optimal (smallest) value of the proximal parameter while convergence of this LSADMM can be still ensured. Our LSADMM partitions the data into two group variables and updates the Lagrange multiplier twice in different forms with suitable step sizes. The region of the proximal parameter, involved in the second group subproblems, is partitioned into the union of three different sets. We show the global convergence and sublinear ergodic convergence rate of LSADMM for the two cases, while a counter-example is given to illustrate that convergence of LSADMM can not be guaranteed for the remaining case. Theoretically, we obtain the optimal lower bound of the proximal parameter. Numerical experiments on the so-called latent variable Gaussian graphical model selection problems are presented to demonstrate performance of the proposed algorithm and the significant advantage of the optimal lower bound of the proximal parameter.

Accelerating ADMM for efficient simulation and optimization

The alternating direction method of multipliers (ADMM) is a popular approach for solving optimization problems that are potentially non-smooth and with hard constraints. It has been applied to various computer graphics applications, including physical simulation, geometry processing, and image processing. However, ADMM can take a long time to converge to a solution of high accuracy. Moreover, many computer graphics tasks involve non-convex optimization, and there is often no convergence guarantee for ADMM on such problems since it was originally designed for convex optimization. In this paper, we propose a method to speed up ADMM using Anderson acceleration, an established technique for accelerating fixed-point iterations. We show that in the general case, ADMM is a fixed-point iteration of the second primal variable and the dual variable, and Anderson acceleration can be directly applied. Additionally, when the problem has a separable target function and satisfies certain conditions, ADMM becomes a fixed-point iteration of only one variable, which further reduces the computational overhead of Anderson acceleration. Moreover, we analyze a particular non-convex problem structure that is common in computer graphics, and prove the convergence of ADMM on such problems under mild assumptions. We apply our acceleration technique on a variety of optimization problems in computer graphics, with notable improvement on their convergence speed.

New Results on the Local Linear Convergence of ADMM: A Joint Approach

Thanks to its versatility, its simplicity, and its fast convergence, alternating direction method of multipliers (ADMM) is among the most widely used approaches for solving a convex problem in distributed form. However, making it running efficiently is an art that requires a fine tuning of system parameters according to the specific application scenario, and which ultimately calls for a thorough understanding of the hidden mechanisms that control the convergence behavior. In this framework, we aim at providing new theoretical insights on the convergence process and specifically on some constituent matrices of ADMM whose eigenstructure provides a close link with the algorithm's convergence speed. One of the key techniques that we develop allows to effectively locate the eigenvalues of a (symmetric) matrix product, thus being able to estimate the contraction properties of ADMM. In the comparison with the results available from the literature, we are able to strengthen the precision of our speed estimate thanks to the fact that we are solving a joint problem (i.e., we are identifying the spectral radius of the product of two matrices) in place of two separate problems (the product of two matrix norms).

Cloud/Edge Computing Service Management in Blockchain Networks: Multi-leader Multi-follower Game-based ADMM for Pricing

The mining process in public blockchains with the Nakamoto consensus protocol requires solving a computational puzzle, i.e., proof-of-work, which is resource expensive to implement in lightweight devices with limited computing resources and energy. Thus, renting mining service from cloud providers becomes a reasonable solution, which is called cloud mining. This enables users who want to mine, i.e., miners, to purchase and lease an amount of hashing power from the cloud/edge providers without any hassle of managing the infrastructure. In this paper, we study the interactions among the cloud/edge providers and miners in blockchain using a multi-leader multi-follower game-theoretic approach, in order to support proof-of-work based blockchains application. Due to the inherent complexity of the formulated game, we employ the Alternating Direction Method of Multipliers (ADMM) algorithm to investigate the optimum solution. Utilizing the decomposition characteristics and fast convergence of ADMM, we obtain the optimum results in a distributed manner. Simulation results demonstrate that with the proposed solutions, the optimization of the utilities of miners and the profits of providers can be jointly achieved.

Improved Susceptibility Artifact Correction of Echo-Planar MRI using the Alternating Direction Method of Multipliers

We present an improved technique for susceptibility artifact correction in echo-planar imaging (EPI), a widely used ultra-fast magnetic resonance imaging (MRI) technique. Our method corrects geometric deformations and intensity modulations present in EPI images. We consider a tailored variational image registration problem incorporating a physical distortion model and aiming at minimizing the distance of two oppositely distorted images subject to invertibility constraints. We derive a novel face-staggered discretization of the variational problem that renders the discretized distance function and constraints separable. Motivated by the presence of a smoothness regularizer, which leads to global coupling, we apply the alternating direction method of multipliers (ADMM) to split the problem into simpler subproblems. We prove the convergence of ADMM for this non-convex optimization problem. We show the superiority of our scheme compared to two state-of-the-art methods both in terms of correction quality and time-to-solution for 13 high-resolution 3D imaging datasets.

Convergence of ADMM for multi-block nonconvex separable optimization models

For solving minimization problems whose objective function is the sum of two functions without coupled variables and the constrained function is linear, the alternating direction method of multipliers (ADMM) has exhibited its efficiency and its convergence is well understood. When either the involved number of separable functions is more than two, or there is a nonconvex function, ADMM or its direct extended version may not converge. In this paper, we consider the multi-block separable optimization problems with linear constraints and absence of convexity of the involved component functions. Under the assumption that the associated function satisfies the Kurdyka- Lojasiewicz inequality, we prove that any cluster point of the iterative sequence generated by ADMM is a critical point, under the mild condition that the penalty parameter is sufficiently large. We also present some sufficient conditions guaranteeing the sublinear and linear rate of convergence of the algorithm.

A note on the convergence of ADMM for linearly constrained convex optimization problems

This note serves two purposes. Firstly, we construct a counterexample to show that the statement on the convergence of the alternating direction method of multipliers (ADMM) for solving linearly constrained convex optimization problems in a highly influential paper by Boyd et al. [Found. Trends Mach. Learn. 3(1) 1-122 (2011)] can be false if no prior condition on the existence of solutions to all the subproblems involved is assumed to hold. Secondly, we present fairly mild conditions to guarantee the existence of solutions to all the subproblems and provide a rigorous convergence analysis on the ADMM, under a more general and useful semi-proximal ADMM (sPADMM) setting considered by Fazel et al. [SIAM J. Matrix Anal. Appl. 34(3) 946-977 (2013)], with a computationally more attractive large step-length that can even exceed the practically much preferred golden ratio of $(1+\sqrt{5})/2$.

Modified alternating direction method of multipliers for convex quadratic semidefinite programming

The direct extension of alternating direction method of multipliers (ADMM) is not necessarily convergent to the dual form of convex quadratic semidefinite programming (CQSDP) problems, though the convergence of ADMM is proved when two blocks of variables are alternatively updated. In this paper, we present a modified ADMM for solving the dual form of the CQSDP problems. At each iteration, the proposed method did not have to work out the sub-problem with the primal variable, compared with the existing ADMM. Assuming that the penalty parameter satisfies the condition related to the quadratic term of the primal objective function, the convergence is proved by using a fixed-point argument. Numerical results for the nearest correlation matrix problems and the random CQSDP problems demonstrate the efficiency of our proposed algorithm.

Local Linear Convergence of the Alternating Direction Method of Multipliers for Quadratic Programs

The Douglas--Rachford alternating direction method of multipliers (ADMM) has been widely used in various areas. The global convergence of ADMM is well known, while research on its convergence rate ...

A Splitting-Based Iterative Algorithm for Accelerated Statistical X-Ray CT Reconstruction

Statistical image reconstruction using penalized weighted least-squares (PWLS) criteria can improve image-quality in X-ray computed tomography (CT). However, the huge dynamic range of the statistical weights leads to a highly shift-variant inverse problem making it difficult to precondition and accelerate existing iterative algorithms that attack the statistical model directly. We propose to alleviate the problem by using a variable-splitting scheme that separates the shift-variant and ("nearly") invariant components of the statistical data model and also decouples the regularization term. This leads to an equivalent constrained problem that we tackle using the classical method-of-multipliers framework with alternating minimization. The specific form of our splitting yields an alternating direction method of multipliers (ADMM) algorithm with an inner-step involving a "nearly" shift-invariant linear system that is suitable for FFT-based preconditioning using cone-type filters. The proposed method can efficiently handle a variety of convex regularization criteria including smooth edge-preserving regularizers and nonsmooth sparsity-promoting ones based on the l(1)-norm and total variation. Numerical experiments with synthetic and real in vivo human data illustrate that cone-filter preconditioners accelerate the proposed ADMM resulting in fast convergence of ADMM compared to conventional (nonlinear conjugate gradient, ordered subsets) and state-of-the-art (MFISTA, split-Bregman) algorithms that are applicable for CT.