Classic Alternating Direction Method Of Multipliers Research Articles

Fast alternating direction algorithms for sparse portfolio model via sorted ℓ1-norm

In this paper, we propose two novel Alternating Direction Method of Multipliers (ADMM) algorithms for the sparse portfolio problem via sorted ℓ1-norm penalization (SLOPE). The first algorithm (FADMM) is presented by adding a prediction-correction step to the classic ADMM framework. Since the problem is not strongly convex, the second fast ADMM (FADMMR) is proposed by utilizing both prediction-correction step and restarting rules. Numerical experiments show that the FADMMR algorithm converges faster than the FADMM algorithm and ADMM algorithm when tuning parameters are relatively small. On the other hand, when tuning parameters are relative large, the FADMM algorithm performs better than the FADMMR algorithm and ADMM algorithm. The FADMM algorithm and FADMMR algorithm converge faster than the ADMM algorithm in terms of convergence time for different sizes of tuning parameters. For large-scale portfolio problem, the proposed algorithms have highly performance as well. Finally, empirical analysis on five datasets of stocks index show that the proposed algorithms are efficient and superior for solving sparse portfolio problems via SLOPE.

An Adaptive Alternating Direction Method of Multipliers

The alternating direction method of multipliers (ADMM) is a powerful splitting algorithm for linearly constrained convex optimization problems. In view of its popularity and applicability, a growing attention is drawn toward the ADMM in nonconvex settings. Recent studies of minimization problems for nonconvex functions include various combinations of assumptions on the objective function including, in particular, a Lipschitz gradient assumption. We consider the case where the objective is the sum of a strongly convex function and a weakly convex function. To this end, we present and study an adaptive version of the ADMM which incorporates generalized notions of convexity and penalty parameters adapted to the convexity constants of the functions. We prove convergence of the scheme under natural assumptions. To this end, we employ the recent adaptive Douglas–Rachford algorithm by revisiting the well-known duality relation between the classical ADMM and the Douglas–Rachford splitting algorithm, generalizing this connection to our setting. We illustrate our approach by relating and comparing to alternatives, and by numerical experiments on a signal denoising problem.

Locating Parking Hubs in Free-Floating Ride Share Systems via Data-Driven Optimization

This paper presents a data-driven study on locating parking hubs in free-floating ride share systems. Recently, there has been an increase in free-floating ride share systems, where users are allowed to pick up and drop off shared vehicles anywhere in the service area. However, these systems can suffer from significant demand and supply imbalance, while certain parking habits may disturb the desired city layout. A potential solution is to allocate parking hubs in an optimal manner to regulate the behaviour of the users. This paper develops a scenario optimization model for finding the optimal locations of parking hubs. The model determines the capacities and locations of the parking hubs, while considering the uncertainty of parking demand and points of interest in the area. We design an algorithm that combines the idea of Constraint-and-Column Generation and the Alternating Direction Method of Multipliers (ADMM) algorithm to solve the optimization problem in a decentralized manner, and accompany the computed solution with a probabilistic performance certificate. We also compare the adopted approach with respect to a worst case paradigm both in terms of computational cost and in terms of conservatism of the resulting solution. Numerical results show that the proposed method leads to a less conservative performance compared to the worst case method, and reduces the computational cost compared to the classical ADMM.

An alternating direction method of multipliers with the BFGS update for structured convex quadratic optimization

The alternating direction method of multipliers (ADMM) is an effective method for solving convex problems from a wide range of fields. At each iteration, the classical ADMM solves two subproblems exactly. However, in many applications, it is expensive or impossible to obtain the exact solutions of the subproblems. To overcome the difficulty, some proximal terms are added to the subproblems. This class of methods typically solves the original subproblem approximately and hence requires more iterations. This fact urges us to consider that a special proximal term can yield better results than the classical ADMM. In this paper, we propose a proximal ADMM whose regularization matrix in the proximal term is generated by the BFGS update (or limited memory BFGS) at every iteration. These types of matrices use second-order information of the objective function. The convergence of the proposed method is proved under certain assumptions. Numerical results are presented to demonstrate the effectiveness of the proposed proximal ADMM.

A two-step iterative algorithm for sparse hyperspectral unmixing via total variation

Sparse hyperspectral unmixing is a hot topic in the field of remote sensing. Its goal is to find an optimal spectral subset, from a large spectral library, to properly model the mixed pixels in hyperspectral images. Sparse unmixing via variable splitting augmented Lagrangian and total variation (SUnSAL-TV) incorporates a TV regularizer into sparse unmixing, achieving a promising unmixing performance. Note that, SUnSAL-TV is solved by the framework of the alternating direction method of multipliers (ADMM). In this paper, we first propose a weighted collaborative sparse unmixing via TV model, named as WCSU-TV, for hyperspectral unmixing. Then a two-step iterative strategy, based on ADMM, is designed to solve the proposed model. Its key idea is to compute the current solution by a linear combination of the results of two previous iterates, instead of only using current solution in classic ADMM. Experiments on simulated and real hyperspectral data illustrate the effectiveness of the proposed approach.

Optimal Risk Operation for a Coupled Electricity and Heat System Considering Different Operation Modes

The coupled electricity and heat system (CEHS) is considered one of the most efficient energy utilization schemes for the energy transition to solve energy crises and environmental problems. However, the individual data between the power system (PS) and heat system (HS) probably limit the high share and optimization operation. Moreover, the uncertain renewables and complex coupled network create challenges regarding the operation risk of CEHS. Given that the CEHS may be affiliated with different operation entities, this paper proposes two operation modes to achieve the solution of the optimal risk operation model (OROM), including the distributionally robust chance constraint (DRCC)-based centralized risk operation mode (CROM) and the DRCC-based alternating direction method of multipliers (ADMM) distributed risk operation mode (DROM). By formulating the moment-based ambiguity set estimated from historical data and introducing the operation risk constraints, a tractable reformulation of two-stage DRCC OROM problems is presented under CROM. Moreover, the DRCC-based ADMM distributed algorithm with the guarantee of convergence is developed to optimize the PS and HS independently under DROM considering the operation risk. The two operation modes achieve the minimum amount of information shared between the two systems regarding the underlying distribution. In the numerical case, the approximate OROM solution is obtained between CROM and DROM. The two proposed approaches based on different operation modes outperform the existing methods according to the risk level depicted by different forms of the violation probability and the risk cost compared with Gaussian chance constraint (GCC) under CROM, while the safety coefficient $\epsilon $ is set as 0.25. Then, the impact of the iteration number on the convergence is also discussed with comparison of the classical ADMM under DROM.

Convergence study of indefinite proximal ADMM with a relaxation factor

The alternating direction method of multipliers (ADMM) is widely used to solve separable convex programming problems. At each iteration, the classical ADMM solves the subproblems exactly. For many problems arising from practical applications, it is usually impossible or too expensive to obtain the exact solution of a subproblem. To overcome this, a special proximal term is added to ease the solvability of a subproblem. In the literature, the proximal term can be relaxed to be indefinite while still with a convergence guarantee; this relaxation permits the adoption of larger step sizes to solve the subproblem, which particularly accelerates its performance. A large value of the relaxation factor introduced in the dual step of ADMM also plays a vital role in accelerating its performance. However, it is still not clear whether these two acceleration strategies can be used simultaneously with no restriction on the penalty parameter. In this paper, we answer this question affirmatively and conduct a rigorous convergence analysis for indefinite proximal ADMM with a relaxation factor (IP-ADMM $$_{r}$$ ), reveal the relationships between the parameter in the indefinite proximal term and the relaxation factor to ensure its global convergence, and establish the worst-case convergence rate in the ergodic sense. Finally, some numerical results on basis pursuit and total variation-based denoising with box constraint problems are presented to verify the efficiency of IP-ADMM $$_{r}$$ .

Large Scale Resource Allocation for the Internet of Things Network Based on ADMM

Large scale deployment of Internet of Things (IoT) devices poses challenges in resource allocation. In this paper, alternating direction method of multipliers (ADMM) is adopted to solve such large scale resource allocation problems. Based on this, three optimization problems are investigated in a hierarchical IoT network. Considering ADMM could not solve a non-convex optimization problem directly, a non-convex fractional programming problem i.e., energy efficiency maximization problem for IoT region server, is formulated. Faced with this problem, we introduce the Dinkelbach algorithm to transfer the energy efficiency maximization problem into an equivalent convex optimization problem. Then the classic ADMM with two blocks is employed to solve the equivalent convex optimization problem. On the other hand, the classic ADMM with two blocks could not satisfy the convergence speed demands of the high-dimensional convex optimization problems any more. Thus, the network latency minimization problem for controller is designed and then solved by the Jacobian-ADMM algorithm in parallel. It is hard to satisfy controller and IoT region servers’ objectives at the same time. Given this, an incentive mechanism on the basis of Stackelberg game is designed. Thus a game-based resource allocation problem is proposed to deal with the contradiction between the centralized objective of the controller and the individual objectives from the IoT region servers. Based on the Dinkelbach algorithm and Jacobian-ADMM algorithm, a two-layer iterative resource allocation algorithm is posed to solve the game-based resource allocation problem. Last but not least, the convergence of the proposed algorithms are analyzed with numerous simulation results.

On the equivalence of inexact proximal ALM and ADMM for a class of convex composite programming

In this paper, we show that for a class of linearly constrained convex composite optimization problems, an (inexact) symmetric Gauss–Seidel based majorized multi-block proximal alternating direction method of multipliers (ADMM) is equivalent to an inexact proximal augmented Lagrangian method. This equivalence not only provides new perspectives for understanding some ADMM-type algorithms but also supplies meaningful guidelines on implementing them to achieve better computational efficiency. Even for the two-block case, a by-product of this equivalence is the convergence of the whole sequence generated by the classic ADMM with a step-length that exceeds the conventional upper bound of $$(1+\sqrt{5})/2$$ , if one part of the objective is linear. This is exactly the problem setting in which the very first convergence analysis of ADMM was conducted by Gabay and Mercier (Comput Math Appl 2(1):17–40, 1976), but, even under notably stronger assumptions, only the convergence of the primal sequence was known. A collection of illustrative examples are provided to demonstrate the breadth of applications for which our results can be used. Numerical experiments on solving a large number of linear and convex quadratic semidefinite programming problems are conducted to illustrate how the theoretical results established here can lead to improvements on the corresponding practical implementations.

An Alternating Augmented Lagrangian method for constrained nonconvex optimization

ABSTRACTWe consider the problem of minimizing a smooth nonconvex function over a structured convex feasible set, that is, defined by two sets of constraints that are easy to treat when considered separately. In order to exploit the structure of the problem, we define an equivalent formulation by duplicating the variables and we consider the augmented Lagrangian of this latter formulation. Following the idea of the Alternating Direction Method of Multipliers (ADMM), we propose an algorithm where a two-blocks decomposition method is embedded within an augmented Lagrangian framework. The peculiarities of the proposed algorithm are the following: (1) the computation of the exact solution of a possibly nonconvex subproblem is not required; (2) the penalty parameter is iteratively updated once an approximated stationary point of the augmented Lagrangian is determined. Global convergence results are stated under mild assumptions and without requiring convexity of the objective function. Although the primary aim of the paper is theoretical, we perform numerical experiments on a nonconvex problem arising in machine learning, and the obtained results show the practical advantages of the proposed approach with respect to classical ADMM.

An FE-Inexact Heterogeneous ADMM for Elliptic Optimal Control Problems with L1-Control Cost

Elliptic PDE-constrained optimal control problems with L1-control cost (L1-EOCP) are considered. To solve L1-EOCP, the primal-dual active set (PDAS) method, which is a special semismooth Newton (SSN) method, used to be a priority. However, in general solving Newton equations is expensive. Motivated by the success of alternating direction method of multipliers (ADMM), we consider extending the ADMM to L1-EOCP. To discretize L1-EOCP, the piecewise linear finite element (FE) is considered. However, different from the finite dimensional l1-norm, the discretized L1-norm does not have a decoupled form. To overcome this difficulty, an effective approach is utilizing nodal quadrature formulas to approximately discretize the L1-norm and L2-norm. It is proved that these approximation steps will not change the order of error estimates. To solve the discretized problem, an inexact heterogeneous ADMM (ihADMM) is proposed. Different from the classical ADMM, the ihADMM adopts two different weighted inner products to define the augmented Lagrangian function in two subproblems, respectively. Benefiting from such different weighted techniques, two subproblems of ihADMM can be efficiently implemented. Furthermore, theoretical results on the global convergence as well as the iteration complexity results o(1/k) for ihADMM are given. In order to obtain more accurate solution, a two-phase strategy is also presented, in which the primal-dual active set (PDAS) method is used as a postprocessor of the ihADMM. Numerical results not only confirm error estimates, but also show that the ihADMM and the two-phase strategy are highly efficient.

An efficient algorithm for batch images alignment with adaptive rank-correction term

With the appearance of approach named “robust alignment by sparse and low-rank decomposition” (RASL), a number of linearly correlated images can be accurately and robustly aligned despite significant corruptions and occlusions. It has been discovered that this aligning task can be characterized as a sequence of 3-block convex minimization problems which can be solved efficiently by the accelerated proximal gradient method (APG), or alternatively, by the directly extended alternating direction method of multipliers (ADMM). However, the directly extended ADMM may diverge although it often performs well in numerical computations. Ideally, one should find an algorithm which can have both theoretical guarantee and superior numerical efficiency over the directly extended ADMM. We achieve this goal by using the intelligent symmetric Gauss–Seidel iteration based ADMM (sGS-ADMM) which only needs to update one of the variables twice, but surprisingly, it leads to the desired convergence to be guaranteed. The convergence of sGS-ADMM can be followed directly by relating it to the classical 2-block ADMM and with a couple of specially designed semi-proximal terms. Beyond this, we also add a rank-correction term to the model with the purpose of deriving the alignment results with higher accuracy. The numerical experiments over a wide range of realistic misalignments demonstrate that sGS-ADMM is at least two times faster than RASL and APG for the vast majority of the tested problems.

Convergence analysis on a modified generalized alternating direction method of multipliers

The alternating direction method of multipliers (ADMM) is one of the most powerful and successful methods for solving convex composite minimization problem. The generalized ADMM relaxes both the variables and the multipliers with a common relaxation factor in (0,2), which has the potential of enhancing the performance of the classic ADMM. Very recently, two different variants of semi-proximal generalized ADMM have been proposed. They allow the weighting matrix in the proximal terms to be positive semidefinite, which makes the subproblems relatively easy to evaluate. One of the variants of semi-proximal generalized ADMMs has been analyzed theoretically, but the convergence result of the other is not known so far. This paper aims to remedy this deficiency and establish its convergence result under some mild conditions in the sense that the relaxation factor is also restricted into (0,2).

Symmetric Gauss–Seidel Technique-Based Alternating Direction Methods of Multipliers for Transform Invariant Low-Rank Textures Problem

Transform Invariant Low-Rank Textures, referred to as TILT, can accurately and robustly extract textural or geometric information in a 3D from user-specified windows in 2D in spite of significant corruptions and warping. It was discovered that the task can be characterized, both theoretically and numerically, by solving a sequence of matrix nuclear-norm and $\ell_1$-norm involved convex minimization problems. For solving this problem, the direct extension of Alternating Direction Method of Multipliers (ADMM) in an usual Gauss-Seidel manner often performs numerically well in practice but there is no theoretical guarantee on its convergence. In this paper, we resolve this dilemma by using the novel symmetric Gauss-Seidel (sGS) based ADMM developed by Li, Sun \& Toh (Math. Prog. 2016). The sGS-ADMM is guaranteed to converge and we shall demonstrate in this paper that it is also practically efficient than the directly extended ADMM. When the sGS technique is applied to this particular problem, we show that only one variable needs to be re-updated, and this updating hardly imposes any excessive computational cost. The sGS decomposition theorem of Li, Sun \& Toh (arXiv: 1703.06629) establishes the equivalent between sGS-ADMM and the classical ADMM with an additional semi-proximal term, so the convergence result is followed directly. Extensive experiments illustrate that the sGS-ADMM and its generalized variant have superior numerical efficiency over the directly extended ADMM.

A generalized alternating direction method of multipliers with semi-proximal terms for convex composite conic programming

In this paper, we propose a generalized alternating direction method of multipliers (ADMM) with semi-proximal terms for solving a class of convex composite conic optimization problems, of which some are high-dimensional, to moderate accuracy. Our primary motivation is that this method, together with properly chosen semi-proximal terms, such as those generated by the recent advance of block symmetric Gauss–Seidel technique, is capable of tackling these problems. Moreover, the proposed method, which relaxes both the primal and the dual variables in a natural way with a common relaxation factor in the interval of (0, 2), has the potential of enhancing the performance of the classic ADMM. Extensive numerical experiments on various doubly non-negative semidefinite programming problems, with or without inequality constraints, are conducted. The corresponding results showed that all these multi-block problems can be successively solved, and the advantage of using the relaxation step is apparent.

Linear Rate Convergence of the Alternating Direction Method of Multipliers for Convex Composite Programming

In this paper, we aim to prove the linear rate convergence of the alternating direction method of multipliers (ADMM) for solving linearly constrained convex composite optimization problems. Under a mild calmness condition, which holds automatically for convex composite piecewise linear-quadratic programming, we establish the global Q-linear rate of convergence for a general semi-proximal ADMM with the dual step-length being taken in (0, (1+51/2)/2). This semi-proximal ADMM, which covers the classic one, has the advantage to resolve the potentially nonsolvability issue of the subproblems in the classic ADMM and possesses the abilities of handling the multi-block cases efficiently. We demonstrate the usefulness of the obtained results when applied to two- and multi-block convex quadratic (semidefinite) programming.

Linearized block-wise alternating direction method of multipliers for multiple-block convex programming

The alternating direction method of multipliers (ADMM) is an efficient approach for two-block separable convex programming, while it is not necessarily convergent when extending this method to multiple-block case directly. One appealing method is that converts the multiple-block variables into two groups firstly and then adopts the classic ADMM with inexact solving to the resulting model, which is so-called block-wise ADMM. However, solving the subproblems in block-wise ADMM are usually difficult when the linear mappings in the constraints are not diagonal or the proximal operator of the objective function is not easy to evaluate. Therefore, in this paper, we adopt the linearization technique to different terms presented in the block-wise ADMM subproblems, and obtain three kinds of linearized block-wise ADMM which make the subproblems easy to solve in general case. Moreover, under some mild conditions, we prove the global convergence of the three new methods and report some preliminary numerical results to indicate the feasibility and effectiveness of the linearization strategy.

An inexact alternating direction method of multipliers with relative error criteria

In this paper, we study an inexact version of the alternating direction method of multipliers (ADMM) for solving two-block separable linearly constrained convex optimization problems. Specifically, the two subproblems in the classic ADMM are allowed to be solved inexactly by certain relative error criteria, in the sense that only two parameters are needed to control the inexactness. Related convergence analysis are established under the assumption that the solution set to the KKT system of the problem is not empty. Numerical results on solving a class of sparse signal recovery problems are also provided to demonstrate the efficiency of the proposed algorithm.

Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems

The alternating direction method of multipliers (ADMM) is widely used to solve large-scale linearly constrained optimization problems, convex or nonconvex, in many engineering fields. However there is a general lack of theoretical understanding of the algorithm when the objective function is nonconvex. In this paper we analyze the convergence of the ADMM for solving certain nonconvex consensus and sharing problems, and show that the classical ADMM converges to the set of stationary solutions, provided that the penalty parameter in the augmented Lagrangian is chosen to be sufficiently large. For the sharing problems, we show that the ADMM is convergent regardless of the number of variable blocks. Our analysis does not impose any assumptions on the iterates generated by the algorithm, and is broadly applicable to many ADMM variants involving proximal update rules and various flexible block selection rules.

ADMM-EM Method for L1-Norm Regularized Weighted Least Squares PET Reconstruction.

The L 1-norm regularization is usually used in positron emission tomography (PET) reconstruction to suppress noise artifacts while preserving edges. The alternating direction method of multipliers (ADMM) is proven to be effective for solving this problem. It sequentially updates the additional variables, image pixels, and Lagrangian multipliers. Difficulties lie in obtaining a nonnegative update of the image. And classic ADMM requires updating the image by greedy iteration to minimize the cost function, which is computationally expensive. In this paper, we consider a specific application of ADMM to the L 1-norm regularized weighted least squares PET reconstruction problem. Main contribution is derivation of a new approach to iteratively and monotonically update the image while self-constraining in the nonnegativity region and the absence of a predetermined step size. We give a rigorous convergence proof on the quadratic subproblem of the ADMM algorithm considered in the paper. A simplified version is also developed by replacing the minima of the image-related cost function by one iteration that only decreases it. The experimental results show that the proposed algorithm with greedy iterations provides a faster convergence than other commonly used methods. Furthermore, the simplified version gives a comparable reconstructed result with far lower computational costs.