Alternating Direction Method Of Multipliers Algorithm Research Articles

Mechanism and a new algorithm for nonconvex clustering

ABSTRACTNonconvex clustering is based on over-parameterization and uses a shrinkage term to achieve sparsity in the differences among individual centroids. In this paper, we elaborate the mechanism of nonconvex clustering from a new perspective by reformulating the basic alternating direction method of multipliers (ADMM) algorithm, and explain for the first time how nonconvex clustering can identify outliers that are far from the nearest cluster centre [Knorr EM, Ng RT. Algorithms for mining distance-based outliers in large datasets. Proceedings of the 24th VLDB Conference; New York, USA: 1998. p. 392–403]. To overcome the computational burden owing to complexity, inspired by the mechanism of nonconvex clustering, we develop another novel and accelerating algorithm called aBlock_ADMM, which combines adaptive block clustering and the ADMM algorithm. We evaluate the clustering performance of the new algorithm on both simulated and real data examples.

Decentralized Consensus Optimization Based on Parallel Random Walk

The alternating direction method of multipliers (ADMM) has recently been recognized as a promising approach for large-scale machine learning models. However, very few results study ADMM from the aspect of communication costs, especially jointly with running time. In this letter, we investigate the communication efficiency and running time of ADMM in solving the consensus optimization problem over decentralized networks. We first review the effort of random walk ADMM (W-ADMM), which reduces communication costs at the expense of running time. To accelerate the convergence speed of W-ADMM, we propose the parallel random walk ADMM (PW-ADMM) algorithm, where multiple random walks are active at the same time. Moreover, to further reduce the running time of PW-ADMM, the intelligent parallel random walk ADMM (IPW-ADMM) algorithm is proposed through integrating the Random Walk with Choice with PW-ADMM. By numerical results from simulations, we demonstrate that the proposed algorithms can be both communication efficient and fast in running speed compared with state-of-the-art methods.

Decentralized Consensus Optimization Based on Parallel Random Walk

The alternating direction method of multipliers (ADMM) has recently been recognized as a promising approach for large-scale machine learning models. However, very few results study ADMM from the aspect of communication costs, especially jointly with running time. In this letter, we investigate the communication efficiency and running time of ADMM in solving the consensus optimization problem over decentralized networks. We first review the effort of random walk ADMM (W-ADMM), which reduces communication costs at the expense of running time. To accelerate the convergence speed of W-ADMM, we propose the parallel random walk ADMM (PW-ADMM) algorithm, where multiple random walks are active at the same time. Moreover, to further reduce the running time of PW-ADMM, the intelligent parallel random walk ADMM (IPW-ADMM) algorithm is proposed through integrating the \textit{Random Walk with Choice} with PW-ADMM. By numerical results from simulations, we demonstrate that the proposed algorithms can be both communication efficient and fast in running speed compared with state-of-the-art methods.

Efficient QP-ADMM Decoder for Binary LDPC Codes and Its Performance Analysis

This paper presents an efficient quadratic programming (QP) decoder via the alternating direction method of multipliers (ADMM) technique, called QP-ADMM, for binary low-density parity-check (LDPC) codes. Its main contents are as follows: first, we relax maximum likelihood (ML) decoding problem to a non-convex quadratic program. Then, we develop an ADMM solving algorithm for the formulated non-convex QP decoding model. In the proposed QP-ADMM decoder, complex Euclidean projections onto the check polytope are eliminated and variables in each updated step can be solved analytically in parallel. Moreover, it is proved that the proposed ADMM algorithm converges to a stationary point of the non-convex QP problem under the assumption of sequence convergence. We also verify that the proposed decoder satisfies the favorable property of the all-zeros assumption. Furthermore, by exploiting the inside structures of the QP model, the complexity of the proposed algorithm in each iteration is shown to be linear in terms of LDPC code length. Simulation results demonstrate the effectiveness of the proposed QP-ADMM decoder.

Efficient Algorithms for Maximum Consensus Robust Fitting

Maximum consensus robust fitting is a fundamental problem in many computer vision applications, such as vision-based robotic navigation and mapping. While exact search algorithms are computationally demanding, randomized algorithms are cheap but the solution quality is not guaranteed. Deterministic algorithms fill the gap between these two kinds of algorithms, which have better solution quality than randomized algorithms while being much faster than exact algorithms. In this article, we develop two highly efficient deterministic algorithms based on the alternating direction method of multipliers (ADMM) and proximal block coordinate descent (BCD) frameworks. Particularly, the proposed BCD algorithm is guaranteed convergent. Furthermore, on the slack variable in the BCD algorithm, which indicates the inliers and outliers, we establish some meaningful properties, such as support convergence within finite iterations and convergence to restricted strictly local minimizer. Compared with state-of-the-art algorithms, the new algorithms with initialization from a randomized or convex relaxed algorithm can achieve improved solution quality while being much more efficient (e.g., more than an order of magnitude faster). An application of the new ADMM algorithm in simultaneous localization and mapping (SLAM) has also been provided to demonstrate its effectiveness. Code for reproducing the results is available online.1

DSTPCA: Double-Sparse Constrained Tensor Principal Component Analysis Method for Feature Selection.

The identification of differentially expressed genes plays an increasingly important role biologically. Therefore, the feature selection approach has attracted much attention in the field of bioinformatics. The most popular method of principal component analysis studies two-dimensional data without considering the spatial geometric structure of the data. The recently proposed tensor robust principal component analysis method performs sparse and low-rank decomposition on three-dimensional tensors and effectively preserves the spatial structure. Based on this approach, the L2,1- norm regularization term is introduced into the DSTPCA (Double-Sparse Constrained Tensor Principal Component Analysis) method. The DSTPCA method removes the redundant noise by double sparse constraints on the objective function to obtain sufficiently sparse results. After the regularization norm is introduced into the model, the ADMM (alternating direction method of multipliers) algorithm is used to solve the optimal problem. In the experiment of feature selection, while the more redundant genes were filtered out, the more genes closely associated with disease were screened. Experimental results using different datasets indicate that our method outperforms other methods.

Distributed inexact dual consensus ADMM for network resource allocation

SummaryThis paper investigates two novel distributed algorithms based on alternating direction method of multipliers (ADMM) for network resource allocation of N agents. The main objective is to derive an optimal allocation that minimizes a global objective expressed as a sum of locally known separable convex objective functions. Based on a communication matrix, the dual resource allocation problem is changed into a consensus optimization problem, in which each agent broadcasts the outcome of its local processing to all his neighbors. In this paper, we first propose a new distributed dual consensus ADMM (DC‐ADMM) algorithm to address this consensus problem. Moreover, by applying an inexact step for each ADMM update, a distributed inexact DC‐ADMM (IDC‐ADMM) is also developed, which enables agents to perform cheap computation at each iteration. Finally, numerical simulations are delivered to illustrate and validate the proposed algorithm.

Energy Management in Plug-In Hybrid Electric Vehicles: Convex Optimization Algorithms for Model Predictive Control

This article details an investigation into the computational performance of algorithms used for solving a convex formulation of the optimization problem associated with model predictive control for energy management in hybrid electric vehicles with nonlinear losses. A projected interior-point method is proposed, where the size and complexity of the Newton step matrix inversion is reduced by applying inequality constraints on the control input as a projection, and its properties are demonstrated through simulation in comparison with an alternating direction method of multipliers (ADMM) algorithm and a general purpose convex optimization software CVX. It is found that the ADMM algorithm has favorable properties when a solution with modest accuracy is required, whereas the projected interior-point method is favorable when high accuracy is required, and that both are significantly faster than CVX.

A pan-sharpening method based on the ADMM algorithm

Pan-sharpening is a method of integrating low-resolution multispectral images with corresponding high-resolution panchromatic images to obtain multispectral images with high spectral and spatial resolution. A novel variational model for pan-sharpening is proposed in this paper. The model is mainly based on three hypotheses: 1) the pan-sharpened image can be linearly represented by the corresponding panchromatic image; 2) the low-resolution multispectral image is down-sampled from the highresolution multispectral image through the down-sampling operator; and 3) the satellite image has the low-rank property. Three energy components corresponding to these assumptions are integrated into a variational framework to obtain a total energy function. We adopt the alternating direction method of multipliers (ADMM) to optimize the total energy function. The experimental results show that the proposed method performs better than other mainstream methods in spectral and spatial information preserving aspect.

Fast ADMM for Sum-of-Squares Programs Using Partial Orthogonality

When sum-of-squares (SOS) programs are recast as semidefinite programs (SDPs) using the standard monomial basis, the constraint matrices in the SDP possess a structural property that we call partial orthogonality. In this paper, we leverage partial orthogonality to develop a fast first-order method, based on the alternating direction method of multipliers (ADMM), for the solution of the homogeneous self-dual embedding of SDPs describing SOS programs. Precisely, we show how a “diagonal plus low rank” structure implied by partial orthogonality can be exploited to project efficiently the iterates of a recent ADMM algorithm for generic conic programs onto the set defined by the affine constraints of the SDP. The resulting algorithm, implemented as a new package in the solver CDCS, is tested on a range of large-scale SOS programs arising from constrained polynomial optimization problems and from Lyapunov stability analysis of polynomial dynamical systems. These numerical experiments demonstrate the effectiveness of our approach compared to common state-of-the-art solvers.

On the Design of Constant Modulus Probing Waveforms With Good Correlation Properties for MIMO Radar via Consensus-ADMM Approach

In this paper, we design constant modulus probing waveforms with good correlation properties for collocated multi-input multi-output (MIMO) radar systems. The main content is as follows: first, we formulate the design problem as a fourth order polynomial minimization problem with constant modulus constraints. Then, by exploiting introduced auxiliary variables and their inherent structures, the polynomial optimization model is equivalent to a non-convex consensus minimization problem. Second, a customized alternating direction method of multipliers (ADMM) algorithm is proposed to solve the non-convex problem approximately. In the algorithm, all the subproblems can be solved analytically. Moreover, all subproblems except one subproblem can be performed in parallel. Third, we prove that the customized ADMM algorithm is theoretically-guaranteed convergent if proper parameters are chosen. Fourth, two variant ADMM algorithms, based on stochastic block coordinate descent and accelerated gradient descent, are proposed to reduce computational complexity and speed up the convergence rate. Numerical examples show the effectiveness of the proposed consensus-ADMM algorithm and its variants.

Improved ADMM-based algorithms for multi-group multicasting in large-scale antenna systems with extension to hybrid beamforming

In this paper, multi-group multicast beamforming is considered for the full digital and hybrid beamforming. The wireless system comprises of a multiple-antenna base station and single-antenna users. Quality-of-service (QoS)-aware design is investigated where the optimization objective is to minimize the total transmitted power subject to the signal-to-interference-plus-noise ratio (SINR) constraint at each user. In addition to the SINR constraints, per-antenna power constraint is included for each antenna of the base station. The original optimization problem for full digital beamforming is transformed into an equivalent form such that the alternating direction method of multipliers (ADMM) can be applied in an effective and computationally inexpensive manner for the large-scale antenna systems. In this new formulation, the beamformer weight vectors are decomposed into two subspaces in order to decrease the number of dual variables and multiplications. The optimum update equations are obtained for the proposed ADMM algorithm. This new reformulation is used for two different hybrid beamforming structures employing phase shifters and vector modulators. Optimum updates are derived for each system. The proposed algorithms decrease computational complexity of the existing ADMM algorithms due to the effective reformulation as well as the direct solution of the nonconvex problem. In the simulation results, it is shown that the proposed methods have better convergence behavior and less computational time than the benchmark algorithms. Furthermore, the proposed method for hybrid beamforming with vector modulators performs better than its counterpart in the literature in terms of transmitted power.

High-dimensional integrative analysis with homogeneity and sparsity recovery

This paper studies integrative analysis of multiple units in the context of high-dimensional linear regression. We consider the case where a fraction of the covariates have different effects on the responses across various units, e.g., some coefficients are the same for all the units, while others have grouping structures. We propose a least squares approach, combined with a difference penalty term to penalize the difference between any two units’ coefficients of the same covariate for identifying latent grouping structure, as well as a common sparsity penalty to detect important covariates. Without the need to know the grouping structure of every variable across the data units and the sparsity construction within the variables, the proposed double penalized procedure can automatically identify the covariates with heterogeneous effects, covariates with homogeneous effects, and recover the sparsity, the grouping structures of the heterogeneous covariates, and provide estimates of all regression coefficients simultaneously. We proceed the alternating direction method of multipliers algorithm (ADMM) through effectively utilizing the storage and reading of the datasets, and demonstrate the convergence of the proposed procedure. We show that the proposed estimator enjoys the oracle property. Simulation studies demonstrate the good performance of the new method with finite samples, and a real data example is provided for illustration.

ADMM-based deep reconstruction for limited-angle CT

In the real applications of computed tomography (CT) imaging, the projection data of the scanned objects are usually acquired within a limited-angle range because of the limitation of the scanning condition. Under these circumstances, conventional analytical algorithms, such as filtered back-projection (FBP), do not work because the projection data are incomplete. The regularization method has proven to be effective for tomographic reconstruction from under-sampled measurements. To deal with the limited-angle CT reconstruction problem, the regularization method is commonly used, but it is difficult to find a generic regularization term and choose the regularization parameters. Moreover, in some cases, the quality of reconstructed images is less than satisfactory. To solve this problem, we developed an alternating direction method of multipliers (ADMM)-based deep reconstruction (ADMMBDR) algorithm for limited-angle CT. First, we used the ADMM algorithm to decompose a regularization reconstruction model. Then, we utilized a deep convolutional neural network (CNN) to replace a part of the ADMM algorithm to reduce artifacts and avoid the choice of the regularization term and the regularization parameter. Furthermore, we conducted some numerical experiments to evaluate the feasibility and the advantages of the proposed algorithm. The results showed that the proposed algorithm had a better performance than several state-of-the-art algorithms; with respect to structure preservation and artifact reduction.

New auxiliary variable-based ADMM for nonconvex AC OPF

The main challenge in implementing alternating direction method of multipliers (ADMM) for nonconvex alternating current optimal power flow (AC OPF) is that ADMM method does not guarantee convergence for nonconvex problems. Using auxiliary variables for information exchange among subareas play a critical role in convergence improvement. This paper proposes a new auxiliary variable-based ADMM for nonconvex AC OPF. The proposed approach can improve convergence with less iterations compared with the existing method. The proposed ADMM algorithm is tested for power grids with sizes ranges from 30 buses to 1354 buses. Subareas are generated using spectral clustering based on a graph Laplacian representing network connectivity. The numerical results are compared with those based on the existing auxiliary variables-based method in the literature. Case studies demonstrate improvement in convergence due to the new auxiliary variables.

A fundamental proof of convergence of alternating direction method of multipliers for weakly convex optimization

The convergence of the alternating direction method of multipliers (ADMMs) algorithm to convex/nonconvex combinational optimization has been well established in the literature. Due to the extensive applications of a weakly convex function in signal processing and machine learning, in this paper, we investigate the convergence of an ADMM algorithm to the strongly and weakly convex combinational optimization (SWCCO) problem. Specifically, we firstly show the convergence of the iterative sequences of the SWCCO-ADMM under a mild regularity condition; then we establish the o(1/k) sublinear convergence rate of the SWCCO-ADMM algorithm using the same conditions and the linear convergence rate by imposing the gradient Lipschitz continuity condition on the objective function. The techniques used for the convergence analysis in this paper are fundamental, and we accomplish the global convergence without using the Kurdyka–Łojasiewicz (KL) inequality, which is common but complex in the proof of nonconvex ADMM.

Robust PCA Using Generalized Nonconvex Regularization

Recently, the robustification of principal component analysis (PCA) has attracted much research attention in numerous areas of science and engineering. The most popular and successful approach is to model the robust PCA problem as a low-rank matrix recovery problem in the presence of sparse corruption. With this model, the nuclear norm and $\ell _{1}$ -norm penalties are usually used for low-rank and sparsity promotion. Although the nuclear norm and $\ell _{1}$ -norm are favorable due to their convexity, they have a bias problem. In comparison, nonconvex penalties can be expected to yield better recovery performance. In this paper, we consider a formulation for robust PCA using generalized nonconvex penalties for low-rank and sparsity inducing. This nonconvex formulation is efficiently solved by a multi-block alternative direction method of multipliers (ADMM) algorithm. A sufficient condition for the convergence of this new ADMM algorithm has been derived. Furthermore, to address the important issue of nonconvex penalty selection, we have evaluated the new algorithm via numerical experiments in various low-rank and sparsity conditions. The results indicate that, “exact” recovery of the low-rank principle component can be achieved only by nonconvex regularization. MATLAB code is available at https://github.com/FWen/RPCA.git .

Distributed Stochastic Reserve Scheduling in AC Power Systems With Uncertain Generation

This paper presents a framework to carry out multi-area stochastic reserve scheduling (RS) based on an ac optimal power flow (OPF) model with high penetration of wind power using distributed consensus and the alternating direction method of multipliers (ADMM). We first formulate the OPF-RS problem using semidefinite programming (SDP) in infinite-dimensional spaces that is in general computationally intractable. Using a novel affine policy, we develop an approximation of the infinite-dimensional SDP as a tractable finite dimensional SDP, and explicitly quantify the performance of the approximation. To this end, we adopt the recent developments in randomized optimization that allow a priori probabilistic feasibility guarantees to optimally schedule power generating units while simultaneously determining the geographical allocation of the required reserve. We then use the geographical pattern of the power system to decompose the large-scale system into a multi-area power network, and provide a consensus ADMM algorithm to find a feasible solution for both local and overall multi-area network. Using our distributed stochastic framework, each area can use its own wind information to achieve local feasibility certificates, while ensuring overall feasibility of the multi-area power network under mild conditions. We provide numerical comparisons with a new benchmark formulation, the so-called converted dc (CDC) power flow model, using Monte Carlo simulations for two different IEEE case studies.

Distributed Voltage Control Based on ADMM for Large-Scale Wind Farm Cluster Connected to VSC-HVDC

A distributed voltage control (DVC) scheme based on the alternating direction method of multipliers (ADMM) is proposed for the large-scale wind farm cluster (WFC) connected to a voltage source converter-high voltage direct current system. The proposed scheme coordinates the reactive power output among several wind farms, and wind turbines (WTs) within each wind farm. This keeps the collector bus voltage of each wind farm and WT terminal voltages within the feasible range, and minimizes active power losses. First, a distributed reactive power control scheme based on the consensus ADMM is designed for the WFC, which can achieve fair reactive power sharing among wind farms with the optimal reactive power utilization ratio, to regulate the collector bus voltage close to the rated voltage. Second, a decentralized reactive power control scheme based on the standard ADMM is designed for each wind farm, which can effectively regulate reactive power outputs of all WTs within the wind farm. The ADMM algorithm is used to reduce the calculation burden of the wind farm controller. Moreover, the proposed decentralized control scheme coordinates the voltage regulation and economical operation. The DVC is able to effectively reduce the computation burden of the WFC and wind farm controllers, and the communication cost is reduced. A WFC with eight wind farms was used to validate the proposed DVC scheme.

Chordal decomposition in operator-splitting methods for sparse semidefinite programs

We employ chordal decomposition to reformulate a large and sparse semidefinite program (SDP), either in primal or dual standard form, into an equivalent SDP with smaller positive semidefinite (PSD) constraints. In contrast to previous approaches, the decomposed SDP is suitable for the application of first-order operator-splitting methods, enabling the development of efficient and scalable algorithms. In particular, we apply the alternating direction method of multipliers (ADMM) to solve decomposed primal- and dual-standard-form SDPs. Each iteration of such ADMM algorithms requires a projection onto an affine subspace, and a set of projections onto small PSD cones that can be computed in parallel. We also formulate the homogeneous self-dual embedding (HSDE) of a primal-dual pair of decomposed SDPs, and extend a recent ADMM-based algorithm to exploit the structure of our HSDE. The resulting HSDE algorithm has the same leading-order computational cost as those for the primal or dual problems only, with the advantage of being able to identify infeasible problems and produce an infeasibility certificate. All algorithms are implemented in the open-source MATLAB solver CDCS. Numerical experiments on a range of large-scale SDPs demonstrate the computational advantages of the proposed methods compared to common state-of-the-art solvers.