Multi-block Alternating Direction Method Of Multipliers Research Articles

Adversarial Decision Making Against Intelligent Targets in Cooperative Multiagent Systems

In this paper we propose a distributed adversarial decision making approach for multi-agent system (MAS), which is supposed to detect a team of intelligent targets. The MAS is demanded to achieve the best detection benefit against the intelligent targets that can shelter part of their features and prevent the detection according to an expert defense policy. One of the key challenges is how to achieve higher detection benefit that is both determined by the target assignment action of the MAS and the non-predictable defense policy of the targets. To handle this problem, we first formulate the multi-agent decision making problem as a max-min problem of detection benefit and break it into separate parts that are easier to be optimized. Then we introduce a new variant of distributed alternating direction method of multipliers (ADMM) to search the optimal solutions under the worst defense policy that the targets choose. To overcome the lack of access to global convergence of multi-block ADMM, we add local additional variables to formulate a penalty for non-convex parts of the local objective function. The convergence to an equilibrium and the optimality of the detection benefit are empirically validated by numerical simulations. The influence of the parameter setting is also presented and can be regarded as a prior suggestion for real applications.

A linear algebra perspective on the random multi-block ADMM: the QP case

Embedding randomization procedures in the Alternating Direction Method of Multipliers (ADMM) has recently attracted an increasing amount of interest as a remedy to the fact that the direct multi-block generalization of ADMM is not necessarily convergent. Even if, in practice, the introduction of such techniques could mitigate the diverging behaviour of the multi-block extension of ADMM, from the theoretical point of view, it can ensure just the convergence in expectation, which may not be a good indicator of its robustness and efficiency. In this work, analysing the strongly convex quadratic programming case from a linear algebra perspective, we interpret the block Gauss–Seidel sweep performed by the multi-block ADMM in the context of the inexact Augmented Lagrangian Method. Using the proposed analysis, we are able to outline an alternative technique to those present in the literature which, supported from stronger theoretical guarantees, is able to ensure the convergence of the multi-block generalization of the ADMM method.

Extended ADMM for general penalized quantile regression with linear constraints in big data

Quantile regression offers a powerful means of understanding the comprehensive relationship between response variables and predictors. By formulating prior domain knowledge and assumptions as constraints on parameters, the estimation efficiency can be enhanced. This paper studies some methods based on multi-block ADMM (Alternating Direction Method of Multipliers) to fit general penalized quantile regression models with linear constraints on regression coefficients. Different formulations for handling linear constraints and general penalties are explored and compared. Among these formulations, the most efficient one is identified, which provides an explicit expression for each parameter during iterations and eliminates the nested-loop in existing algorithms. Furthermore, this work addresses the challenges posed by big data by developing a parallel ADMM algorithm suitable for distributed data storage. The algorithm’s convergence and a robust stopping criterion are established. To demonstrate the excellent performance of the proposed algorithms, extensive numerical experiments and a real data example are presented. These empirical validations showcase the effectiveness of the methods in handling complex datasets. The details of theoretical proofs and different algorithm variations are provided in the Appendix.

Cloud-Based Computational Model Predictive Control Using a Parallel Multiblock ADMM Approach

Heavy computational load for solving nonconvex problems for large-scale systems or systems with real-time demands at each sample step has been recognized as one of the reasons for preventing a wider application of nonlinear model predictive control (NMPC). To improve the real-time feasibility of NMPC with input nonlinearity, we devise an innovative scheme called cloud-based computational model predictive control (MPC) by using an elaborately designed parallel multi-block alternating direction method of multipliers (ADMM) algorithm. This novel parallel multi-block ADMM algorithm is tailored to tackle the computational issue of solving a nonconvex problem with nonlinear constraints. It is ensured that the designed algorithm converges to a locally optimal solution of the optimization problem under reasonable assumptions by using the Kurdyka–Łojasiewicz property. With the help of this distributed optimization algorithm, a computational MPC scheme is developed, which can transform the NMPC optimization problem into a set of subproblems only associated with the decision variables at one prediction step. Through the parallel computing algorithm, the computational MPC can deal with large computational loads caused by high dimensional optimization problems, and improve computational efficiency. Furthermore, to allow for a more efficient implementation of the developed computational MPC and alleviate local calculation loads, a cloud-based computational MPC architecture is devised, which makes significantly better use of computational resources provided by a cloud server. An important advantage of this architecture with Docker (https://www.docker.com/) container to implement parallelization is that it does not lead to large increases in the solution time regardless of how long the prediction horizon is set. Finally, the developed cloud-based computational MPC architecture is trialed on a group of plug-in hybrid electric vehicles (PHEVs).

Over-relaxed multi-block ADMM algorithms for doubly regularized support vector machines

As a classical machine learning model, support vector machine (SVM) has attracted much attention due to its rigorous theoretical foundation and powerful discriminative performance. The doubly regularized SVM (DRSVM) is an important variant of SVM based on elastic-net regularization, which considers both the sparsity and stability of the model. To tackle the problems of explosive increases in data dimensions and data volume, the alternating direction method of multipliers (ADMM) algorithm can be used to train the DRSVM model. ADMM is an effective iterative algorithm for solving convex optimization problems by decomposing a large issue into a series of solvable subproblems, which is also well suited for distributed computing. However, lack of guaranteed convergence and slow convergence rate are two critical limitations of ADMM. In this paper, a 3-block ADMM algorithm based on the over-relaxation technique is proposed to accelerate DRSVM training, namely, the over-relaxed DRSVM (O-RDRSVM). The main strategy of the over-relaxation technique is to further append the information from the previous iteration to the next iteration to improve the convergence of ADMM. We also propose a distributed version of O-RDRSVM to handle parallel and distributed computing faster, termed DO-RDRSVM. Moreover, we develop a fast O-RDRSVM algorithm (FO-RDRSVM) and a fast DO-RDRSVM algorithm (FDO-RDRSVM), which further reduce the computational cost of O-RDRSVM and DO-RDRSVM by employing the matrix inversion lemma. The convergence analyses ensure the effectiveness of our algorithms for DRSVM training. Finally, extensive experiments on public datasets demonstrate the advantages of our algorithms in terms of convergence rate and training time while maintaining accuracy and sparsity comparable to those of previous works.

A multi-parameter parallel ADMM for multi-block linearly constrained separable convex optimization

The alternating direction method of multipliers (ADMM) has been proved to be effective for solving two-block convex minimization model subject to linear constraints. However, the convergence of multiple-block convex minimization model with linear constraints may not be guaranteed without additional assumptions. Recently, some parallel multi-block ADMM algorithms which solve the subproblems in a parallel way have been proposed. This paper is a further study on this method with the purpose of improving the parallel multi-block ADMM algorithm by introducing more parameters. We propose two multi-parameter parallel ADMM algorithms with proximal point terms attached to all subproblems. Comparing with some popular parallel ADMM-based algorithms, the parameter conditions of the new algorithms are relaxed. Experiments on both real and synthetic problems are conducted to justify the effectiveness of the proposed algorithms compared to several efficient ADMM-based algorithms for multi-block problems.

A parallel multi‐block alternating direction method of multipliers for tensor completion

This paper proposes an algorithm for the tensor completion problem of estimating multi-linear data under the limitation of observation rate. Many tensor completion methods are based on nuclear norm minimization, they may fail to achieve the global solution for solving nuclear norm minimization in tensor completion problem with high missing ratio. To tackle this issue, an adaptive tensor completion method based on parallel multi-block alternating direction method of multipliers (ADMM) algorithm is proposed, it can derive the model from the initial estimate and compute the next estimate from the current solution. The parallel multi-block ADMM with global convergence is adopted to solve the dual problem, which greatly improves the processing power and reliability of the algorithm.

A multi-block ADMM based approach for distribution market clearing with distribution locational marginal price

This work investigates a distribution locational marginal price (DLMP) scheme for promoting the market penetration of small-scale prosumers/consumers connected at distribution level and boosting their potential of demand response. In the distribution market, the distribution market operator (DMO) negotiates with each participant a node-specific DLMP and the respective demand profile of customer, so as to maximize the social welfare while respecting the operational constraints of the distribution system and load devices. First, different customers at demand side are modeled, including prosumers with photovoltaic (PV) generation, distributed storages and electric vehicles. All of them communicate with DMO and realize automated demand response by intelligent trading agent. Then, the distribution market clearing model for a massive number of customers in an ex-ante trade scheme is formulated as a market-based multi-period alternating current optimal power flow (ACOPF). As this clearing model is separable, the augmented Lagrangian relaxation (ALR) method is adopted to solve the market clearing model in a parallel manner. In the ALR, a second-order penalty term at each node is introduced to ensure the convergence, whereas it is not separable for customers. Hence, there are a multi-block demand variable coupled within the second-order penalty term at each node. A standard alternating direction method of multipliers (ADMM) algorithm cannot converge in this situation. To tackle this multi-block problem, we exploit the multi-block ADMM and distributed Dauglas-Rarchford Splitting method (d-DRSM), both of which proximally regularize the customer-specific subproblem to ensure global convergence. Furthermore, the prediction-correction mechanism in d-DRSM algorithm can achieve better rate of convergence in comparison with the multi-block ADMM. Finally, case studies are conducted and numerical results have shown the effectiveness of the proposed methodology, and the performance of different algorithms are analyzed for improving the clearing method’s efficiency.

Managing randomization in the multi-block alternating direction method of multipliers for quadratic optimization

The Alternating Direction Method of Multipliers (ADMM) has gained a lot of attention for solving large-scale and objective-separable constrained optimization. However, the two-block variable structure of the ADMM still limits the practical computational efficiency of the method, because one big matrix factorization is needed at least once even for linear and convex quadratic programming. This drawback may be overcome by enforcing a multi-block structure of the decision variables in the original optimization problem. Unfortunately, the multi-block ADMM, with more than two blocks, is not guaranteed to be convergent. On the other hand, two positive developments have been made: first, if in each cyclic loop one randomly permutes the updating order of the multiple blocks, then the method converges in expectation for solving any system of linear equations with any number of blocks. Secondly, such a randomly permuted ADMM also works for equality-constrained convex quadratic programming even when the objective function is not separable. The goal of this paper is twofold. First, we add more randomness into the ADMM by developing a randomly assembled cyclic ADMM (RAC-ADMM) where the decision variables in each block are randomly assembled. We discuss the theoretical properties of RAC-ADMM and show when random assembling helps and when it hurts, and develop a criterion to guarantee that it converges almost surely. Secondly, using the theoretical guidance on RAC-ADMM, we conduct multiple numerical tests on solving both randomly generated and large-scale benchmark quadratic optimization problems, which include continuous, and binary graph-partition and quadratic assignment, and selected machine learning problems. Our numerical tests show that the RAC-ADMM, with a variable-grouping strategy, could significantly improve the computation efficiency on solving most quadratic optimization problems.

Randomized Neural Networks Based Decentralized Multi-Task Learning via Hybrid Multi-Block ADMM

In multi-task learning (MTL), related tasks learn jointly to improve generalization performance. To exploit the high learning speed of feed-forward neural networks (FNN), we apply the randomized single-hidden layer FNN (RSF) to the MTL problem, where the output weights of RSFs for all the tasks are learned collaboratively. We first present the RSF based MTL problem in the centralized setting, which is solved by the proposed MTL-RSF algorithm. Due to the fact that many data sets of different tasks are geo-distributed, decentralized machine learning is studied. We formulate the decentralized MTL problem based on RSF as majorized multi-block optimization with coupled bi-convex objective functions. To solve the problem, we propose the DMTL-RSF algorithm, which is a hybrid Jacobian and Gauss-Seidel Proximal multi-block alternating direction method of multipliers (ADMM). Further, to reduce the computation load of DMTL-RSF, DMTL-RSF with first-order approximation (FO-DMTL-RSF) is presented. Theoretical analysis shows that the convergence to the stationary point of proposed decentralized algorithms can be guaranteed conditionally. Through simulations, we demonstrate the convergence of presented algorithms, and also show that they can outperform existing MTL methods. Moreover, by adjusting the dimension of hidden feature space, there exists a trade-off between communication load and learning accuracy for DMTL-RSF.

Multiblock ADMM Heuristics for Mixed-Binary Optimization on Classical and Quantum Computers

Solving combinatorial optimization problems on current noisy quantum devices is currently being advocated for (and restricted to) binary polynomial optimization with equality constraints via quantum heuristic approaches. This is achieved using, e.g., the variational quantum eigensolver (VQE) and the quantum approximate optimization algorithm (QAOA). We present a decomposition-based approach to extend the applicability of current approaches to "quadratic plus convex" mixed binary optimization (MBO) problems, so as to solve a broad class of real-world optimization problems. In the MBO framework, we show that the alternating direction method of multipliers (ADMM) can split the MBO into a binary unconstrained problem (that can be solved with quantum algorithms), and continuous constrained convex subproblems (that can be solved cheaply with classical optimization solvers). The validity of the approach is then showcased by numerical results obtained on several optimization problems via simulations with VQE and QAOA on the quantum circuits implemented in Qiskit, an open-source quantum computing software development framework.

On the Efficiency of Random Permutation for ADMM and Coordinate Descent

Random permutation is observed to be powerful for optimization algorithms: for multiblock ADMM (alternating direction method of multipliers), whereas the classical cyclic version diverges, the randomly permuted version converges in practice; for BCD (block coordinate descent), the randomly permuted version is typically faster than other versions. In this paper we provide strong theoretical evidence that random permutation has positive effects on ADMM and BCD, by analyzing randomly permuted ADMM (RP-ADMM) for solving linear systems of equations, and randomly permuted BCD (RP-BCD) for solving unconstrained quadratic problems. First, we prove that RP-ADMM converges in expectation for solving systems of linear equations. The key technical result is that the spectrum of the expected update matrix of RP-BCD lies in (−1/3, 1), instead of the typical range (−1, 1). Second, we establish expected convergence rates of RP-ADMM for solving linear systems and RP-BCD for solving unconstrained quadratic problems. This expected rate of RP-BCD is O(n) times better than the worst-case rate of cyclic BCD, thus establishing a gap of at least O(n) between RP-BCD and cyclic BCD. To analyze RP-BCD, we propose a conjecture of a new matrix algebraic mean-geometric mean inequality and prove a weaker version of it.

Parallel Multi-Block ADMM with o(1 / k) Convergence

This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem: minimize $\sum_{i=1}^N f_i(x_i)$ subject to $\sum_{i=1}^N A_i x_i=c, x_i\in \mathcal{X}_i$. The algorithm decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This Jacobian-type algorithm is well suited for distributed computing and is particularly attractive for solving certain large-scale problems. This paper introduces a few novel results. Firstly, it shows that extending ADMM straightforwardly from the classic Gauss-Seidel setting to the Jacobian setting, from 2 blocks to N blocks, will preserve convergence if matrices $A_i$ are mutually near-orthogonal and have full column-rank. Secondly, for general matrices $A_i$, this paper proposes to add proximal terms of different kinds to the N subproblems so that the subproblems can be solved in flexible and efficient ways and the algorithm converges globally at a rate of o(1/k). Thirdly, a simple technique is introduced to improve some existing convergence rates from O(1/k) to o(1/k). In practice, some conditions in our convergence theorems are conservative. Therefore, we introduce a strategy for dynamically tuning the parameters in the algorithm, leading to substantial acceleration of the convergence in practice. Numerical results are presented to demonstrate the efficiency of the proposed method in comparison with several existing parallel algorithms. We implemented our algorithm on Amazon EC2, an on-demand public computing cloud, and report its performance on very large-scale basis pursuit problems with distributed data.

Iteration Complexity Analysis of Multi-block ADMM for a Family of Convex Minimization Without Strong Convexity

The alternating direction method of multipliers (ADMM) is widely used in solving structured convex optimization problems due to its superior practical performance. On the theoretical side however, a counterexample was shown in Chen et al. (Math Program 155(1):57---79, 2016.) indicating that the multi-block ADMM for minimizing the sum of N$$(N\ge 3)$$(Nź3) convex functions with N block variables linked by linear constraints may diverge. It is therefore of great interest to investigate further sufficient conditions on the input side which can guarantee convergence for the multi-block ADMM. The existing results typically require the strong convexity on parts of the objective. In this paper, we provide two different ways related to multi-block ADMM that can find an $$\epsilon $$∈-optimal solution and do not require strong convexity of the objective function. Specifically, we prove the following two results: (1) the multi-block ADMM returns an $$\epsilon $$∈-optimal solution within $$O(1/\epsilon ^2)$$O(1/∈2) iterations by solving an associated perturbation to the original problem; this case can be seen as using multi-block ADMM to solve a modified problem; (2) the multi-block ADMM returns an $$\epsilon $$∈-optimal solution within $$O(1/\epsilon )$$O(1/∈) iterations when it is applied to solve a certain sharing problem, under the condition that the augmented Lagrangian function satisfies the Kurdyka---źojasiewicz property, which essentially covers most convex optimization models except for some pathological cases; this case can be seen as applying multi-block ADMM to solving a special class of problems.

On the Sublinear Convergence Rate of Multi-block ADMM

The alternating direction method of multipliers (ADMM) is widely used in solving structured convex optimization problems. Despite of its success in practice, the convergence properties of the standard ADMM for minimizing the sum of $N$ $(N\geq 3)$ convex functions with $N$ block variables linked by linear constraints, have remained unclear for a very long time. In this paper, we present convergence and convergence rate results for the standard ADMM applied to solve $N$-block $(N\geq 3)$ convex minimization problem, under the condition that one of these functions is convex (not necessarily strongly convex) and the other $N-1$ functions are strongly convex. Specifically, in that case the ADMM is proven to converge with rate $O(1/t)$ in a certain ergodic sense, and $o(1/t)$ in non-ergodic sense, where $t$ denotes the number of iterations. As a by-product, we also provide a simple proof for the $O(1/t)$ convergence rate of two-block ADMM in terms of both objective error and constraint violation, without assuming any condition on the penalty parameter and strong convexity on the functions.

Multi-Block ADMM for Big Data Optimization in Modern Communication Networks

In this paper, we review the parallel and distributed optimization algorithms based on the Alternating Direction Method of Multipliers (ADMM) for solving optimization problems in modern communication networks. We first introduce the canonical formulation of the large-scale optimization problem. Next, we describe the general form of ADMM and then focus on several direct extensions and sophisticated modifications of ADMM from 2-block to N-block settings to deal with the optimization problem. The iterative schemes and convergence properties of each extension/modification are given, and the implementation on large-scale computing facilities is also illustrated. Finally, we numerate several applications in communication networks, such as the security constrained optimal power flow problem in smart grid networks and mobile data offloading problem in Software Defined Networks (SDNs). Index Terms—Alternating Direction Method of Multipliers (ADMM), modern communication networks, big data

A Schur complement based semi-proximal ADMM for convex quadratic conic programming and extensions

This paper is devoted to the design of an efficient and convergent {semi-proximal} alternating direction method of multipliers (ADMM) for finding a solution of low to medium accuracy to convex quadratic conic programming and related problems. For this class of problems, the convergent two block semi-proximal ADMM can be employed to solve their primal form in a straightforward way. However, it is known that it is more efficient to apply the directly extended multi-block semi-proximal ADMM, though its convergence is not guaranteed, to the dual form of these problems. Naturally, one may ask the following question: can one construct a convergent multi-block semi-proximal ADMM that is more efficient than the directly extended semi-proximal ADMM? Indeed, for linear conic programming with 4-block constraints this has been shown to be achievable in a recent paper by Sun, Toh and Yang [arXiv preprint arXiv:1404.5378, (2014)]. Inspired by the aforementioned work and with the convex quadratic conic programming in mind, we propose a Schur complement based convergent semi-proximal ADMM for solving convex programming problems, with a coupling linear equality constraint, whose objective function is the sum of two proper closed convex functions plus an arbitrary number of convex quadratic or linear functions. Our convergent semi-proximal ADMM is particularly suitable for solving convex quadratic semidefinite programming (QSDP) with constraints consisting of linear equalities, a positive semidefinite cone and a simple convex polyhedral set. The efficiency of our proposed algorithm is demonstrated by numerical experiments on various examples including QSDP.