Alternating Direction Method Of Multipliers Algorithm Research Articles

A flexible and efficient algorithm for high dimensional support vector regression

In high dimensional statistical learning, variable selection and handling highly correlated phenomena are two crucial topics. Elastic-net regularization can automatically perform variable selection and tends to either simultaneously select or remove highly correlated variables. Consequently, it has been widely applied in machine learning. In this paper, we incorporate elastic-net regularization into the support vector regression model, introducing the Elastic-net Support Vector Regression (En-SVR) model. Due to the inclusion of elastic-net regularization, the En-SVR model possesses the capability of variable selection, addressing high dimensional and highly correlated statistical learning problems. However, the optimization problem for the En-SVR model is rather complex, and common methods for solving the En-SVR model are challenging. Nevertheless, we observe that the optimization problem for the En-SVR model can be reformulated as a convex optimization problem where the objective function is separable into multiple blocks and connected by an inequality constraint. Therefore, we employ a novel and efficient Alternating Direction Method of Multipliers (ADMM) algorithm to solve the En-SVR model, and provide a complexity analysis as well as convergence analysis for the algorithm. Furthermore, extensive numerical experiments validate the outstanding performance of the En-SVR model in high dimensional statistical learning and the efficiency of this novel ADMM algorithm.

VSHARP: Variable Splitting Half-quadratic ADMM algorithm for reconstruction of inverse-problems

Medical Imaging (MI) tasks, such as accelerated parallel Magnetic Resonance Imaging (MRI), often involve reconstructing an image from noisy or incomplete measurements. This amounts to solving ill-posed inverse problems, where a satisfactory closed-form analytical solution is not available. Traditional methods such as Compressed Sensing (CS) in MRI reconstruction can be time-consuming or prone to obtaining low-fidelity images. Recently, a plethora of Deep Learning (DL) approaches have demonstrated superior performance in inverse-problem solving, surpassing conventional methods. In this study, we propose vSHARP (variable Splitting Half-quadratic ADMM algorithm for Reconstruction of inverse Problems), a novel DL-based method for solving ill-posed inverse problems arising in MI. vSHARP utilizes the Half-Quadratic Variable Splitting method and employs the Alternating Direction Method of Multipliers (ADMM) to unroll the optimization process. For data consistency, vSHARP unrolls a differentiable gradient descent process in the image domain, while a DL-based denoiser, such as a U-Net architecture, is applied to enhance image quality. vSHARP also employs a dilated-convolution DL-based model to predict the Lagrange multipliers for the ADMM initialization. We evaluate vSHARP on tasks of accelerated parallel MRI Reconstruction using two distinct datasets and on accelerated parallel dynamic MRI Reconstruction using another dataset. Our comparative analysis with state-of-the-art methods demonstrates the superior performance of vSHARP in these applications.

Robust Support Vector Data Description with Truncated Loss Function for Outliers Depression.

Support vector data description (SVDD) is widely regarded as an effective technique for addressing anomaly detection problems. However, its performance can significantly deteriorate when the training data are affected by outliers or mislabeled observations. This study introduces a universal truncated loss function framework into the SVDD model to enhance its robustness and employs the fast alternating direction method of multipliers (ADMM) algorithm to solve various truncated loss functions. Moreover, the convergence of the fast ADMM algorithm is analyzed theoretically. Within this framework, we developed the truncated generalized ramp, truncated binary cross entropy, and truncated linear exponential loss functions for SVDD. We conducted extensive experiments on synthetic and real-world datasets to validate the effectiveness of these three SVDD models in handling data with different noise levels, demonstrating their superior robustness and generalization capabilities compared to other SVDD models.

Distributed Cooperative Optimal Operation of Multiple Virtual Power Plants Based on Multi-Stage Robust Optimization

This paper develops a distributed cooperative optimization model for multiple virtual power plant (VPP) operations based on multi-stage robust optimization and proposes a distributed solution methodology based on the combination of the alternating direction method of multipliers (ADMMs) and column-and-constraint generation (CCG) algorithm to solve the corresponding optimization problem. Firstly, considering the peer-to-peer (P2P) electricity transactions among multiple VPPs, a deterministic cooperative optimal operation model of multiple VPPs based on Nash bargaining is constructed. Secondly, considering the uncertainties of photovoltaic generation and load demand, as well as the non-anticipativity of real-time scheduling of VPPs in engineering, a cooperative optimal operation model of multiple VPPs based on multi-stage robust optimization is then constructed. Thirdly, the constructed model is solved using a distributed solution methodology based on the combination of the ADMM and CCG algorithms. Finally, a case study is solved. The case study results show that the proposed method can realize the optimal scheduling of renewable energy in a more extensive range, which contributes to the promotion of the local consumption of renewable energy and the improvement of the renewable energy utilization efficiency of VPPs. Compared with the traditional deterministic cooperative optimal operation method of multiple VPPs, the proposed method is more resistant to the risk of the uncertainties of renewable energy and load demand and conforms to the non-anticipativity of real-time scheduling of VPPs in engineering. In summary, the presented works strike a balance between the operational robustness and operational economy of VPPs. In addition, under the presented works, there is no need for each VPP to divulge personal private data such as photovoltaic generation and load demand to other VPPs, so the security privacy protection of each VPP can be achieved.

Sequential inertial linear ADMM algorithm for nonconvex and nonsmooth multiblock problems with nonseparable structure

The alternating direction method of multipliers (ADMM) has been widely used to solve linear constrained problems in signal processing, matrix decomposition, machine learning, and many other fields. This paper introduces two linearized ADMM algorithms, namely sequential partial linear inertial ADMM (SPLI-ADMM) and sequential complete linear inertial ADMM (SCLI-ADMM), which integrate linearized ADMM approach with inertial technique in the full nonconvex framework with nonseparable structure. Iterative schemes are formulated using either partial or full linearization while also incorporating the sequential gradient of the composite term in each subproblem’s update. This adaptation ensures that each iteration utilizes the latest information to improve the efficiency of the algorithms. Under some mild conditions, we prove that the sequences generated by two proposed algorithms converge to the critical points of the problem with the help of KŁ property. Finally, some numerical results are reported to show the effectiveness of the proposed algorithms.

A decentralized HC-ADMM approach for large antenna arrays

This work addresses the evolving landscape of internet of things (IoT) applications and large antenna array systems, where optimizing spectral efficiency and simplifying design complexities are crucial. Focusing on two key challenges, the study introduces a novel hybrid analog-digital transceiver strategy tailored for frequency-selective channels. By integrating Shannon and Hartley theorems, the approach enhances data transfer rates, thereby optimizing radio frequency (RF) chain utilization in large-scale antennas. To achieve a balance between transceiver performance and hardware complexity, the study employs a decentralized alternating direction method of multipliers (ADMM) framework. The proposed hybrid consensus ADMM algorithm (HC-ADMM) ensures efficient convergence in decentralized optimization scenarios. Comparative analyses with ADMM and existing system transceiver optimization (ESTO) models highlight HC-ADMM's superior performance across key metrics such as spectral efficiency, efficiency per cell, total efficiency, and optimal scheduling of user equipment (UEs). Particularly notable is HC-ADMM's advanced optimization capabilities as the number of transmit antennas increases, positioning it as a promising approach for enhancing overall communication network performance.

Efficient decentralized optimization for edge-enabled smart manufacturing: A federated learning-based framework

The volume of industrial data of smart manufacturing is growing rapidly. Edge computing has emerged as an advanced technique that provides scalable resources for Industrial Internet of Things (IIoT) devices to analyze industrial data and enhance productivity. However, real-time and privacy-protecting data services are crucial for comprehensive decision-making, particularly utilizing machine learning in edge-enabled smart manufacturing. Therefore, how to optimize efficiently the data processing has become a critical concern for manufacturers seeking to upgrade the overall manufacturing operations. In this paper, we propose a decentralized federated learning-based framework to ensure the processing of industrial data in a low-latency and secure manner. Our solution designs the combination between edge nodes and IIoT devices as a global consensus problem with equilibrium constraints. Moreover, we adopt the distributed Alternating Direction Method of Multipliers (ADMM) algorithm to optimize the proposed data processing model, considering its advantages of decomposability and parallelism. To flexibly handle various types of industrial data, such as low-rank and high-dimensional data, we present two types of inexact ADMM algorithms to provide efficient model training services, respectively. Furthermore, an integrated optimization flow is designed for data processing in edge-enabled smart manufacturing. Finally, using industrial datasets from a thermal power plant for steam prediction case, we show that the presented inexact algorithms can decrease the response time by up to 17.2% and 58% respectively compared to other existing algorithms, respectively, while achieving the comparable level of statistical accuracy.

Optimal querying for communication-efficient ADMM using Gaussian process regression

In distributed optimization schemes consisting of a group of agents connected to a central coordinator, the optimization algorithm often involves the agents solving private local sub-problems and exchanging data frequently with the coordinator to solve the global distributed problem. In those cases, the query-response mechanism usually causes excessive communication costs to the system, necessitating communication reduction in scenarios where communication is costly. Integrating Gaussian processes (GP) as a learning component to the Alternating Direction Method of Multipliers (ADMM) has proven effective in learning each agent’s local proximal operator to reduce the required communication exchange. A key element for integrating GP into the ADMM algorithm is the querying mechanism upon which the coordinator decides when communication with an agent is required. In this paper, we formulate a general querying decision framework as an optimization problem that balances reducing the communication cost and decreasing the prediction error. Under this framework, we propose a joint query strategy that takes into account the joint statistics of the query and ADMM variables and the total communication cost of all agents in the presence of uncertainty caused by the GP regression. In addition, we derive three different decision mechanisms that simplify the general framework by making the communication decision for each agent individually. We integrate multiple measures to quantify the trade-off between the communication cost reduction and the optimization solution’s accuracy/optimality. The proposed methods can achieve significant communication reduction and good optimization solution accuracy for distributed optimization, as demonstrated by extensive simulations of a distributed sharing problem.

A Machine Learning-Assisted Distributed Optimization Method for Inverter-Based Volt-VAR Control in Active Distribution Networks

The number of smart inverters in active distribution networks is growing rapidly, making it challenging to realize a fast, distributed Volt/Var control (VVC). This work proposes a machine learning-assisted distributed algorithm to accelerate the solution of the VVC strategy. We first observe the convergence process of the Alternating Direction Method of Multipliers (ADMM)-based VVC problem and explore the potential relationships between the convergence and time-series regression. Then, the long short-term memory (LSTM) technique is applied to learn the convergence process and regress the converged values of the dual and global variables with previous ADMM observations. After that, the LSTM-assisted ADMM algorithm is proposed, where the regressions are used for ADMM parameter updates. In this algorithm, the inputs of the LSTM model are carefully designed since the complementary conditions implied in the conventional ADMM should be considered. Unlike existing methods, the proposed method does not use the LSTM to determine the VVC strategy directly, indicating that it is non-intrusive and can satisfy all safety constraints during operations. The proof of its optimality and convergence is also given. The numerical simulations on the 33-bus distribution system demonstrate the effectiveness and efficiency of the proposed method.

A distributed robust state estimation method based on alternating direction method of multipliers for integrated electricity‐heat system

AbstractIntegrated electricity‐heat system (IEHS) has been paid more and more attention in recent years for its advantage in improving energy efficiency, reducing carbon emissions and increasing renewable energy penetration. To ensure the safety, reliability and economic operation of IEHS, several centralised state estimation (SE) methods for IEHS have been proposed. However, power systems and heat systems often belong to different management entities, and there are industrial barriers such as information privacy, operational differences, and target differences between them, which leads to less applicability of centralised SE methods. In addition, the robustness of existing distributed SE methods for IEHS is not satisfactory. To this end, a distributed robust state estimation (DRSE) model for IEHS based on the alternating direction method of multipliers (ADMM) is proposed. Firstly, by introducing auxiliary state variables and measurements, a robust linear SE model based on weighted least absolute values (WLAV) is proposed. Then, second‐order cone constraints composed of auxiliary state variables are added to the SE model, leading a SOCP‐based robust SE model. Finally, the ADMM algorithm is used to solve the proposed SOCP‐based robust SE model. Simulations demonstrate that the proposed method has higher estimation accuracy in both general and strongly correlated adverse data tests and also can ensure data privacy, good robustness and high estimation accuracy. This indicates that the method proposed has good robustness and solves the problem of weak robustness of existing distributed static state estimation methods.

Asynchronous ADMM for nonlinear continuous‐time systems

AbstractThis paper presents synchronous as well as asynchronous formulations of the alternating direction method of multipliers (ADMM) for solving continuous‐time nonlinear distributed model predictive control (DMPC) problems. It is shown that the optimal control problems of certain system classes can be transformed to fit the consensus‐based ADMM variant problem formulation. The arising subproblems are solved locally on the agent level while the consensus step is solved centrally by a coordinator. Furthermore, the convergence of the synchronous and asynchronous ADMM algorithms to their respective first‐order optimality conditions is presented in a continuous‐time setting. The algorithm is applied to different example systems for which the convergence behavior and influence of the individual algorithmic parameters are investigated. The computation time of the agents remains unaffected by the system size and thus demonstrates the applicability to high‐scaled systems. Moreover, results show that the asynchronous algorithm performs better in terms of execution time when compared to its synchronous counterpart.

Distributed ADMM power optimal control for standalone hybrid generation systems

With the rapid development and increased demand for renewable energy sources, standalone hybrid generation systems have become an essential energy solution. Power optimization control is thus critical to achieving the efficient operation and stability of this system. The distributed ADMM (alternating direction method of multipliers)-based approach has the full potential to deal with the power optimization problem of standalone hybrid generation systems. This study uses an optimization algorithm with a Gaussian penalty function, ADMM-ρ, to alternately optimize the power reference values of wind, light, and battery-containing power generation subsystems. The local controller regulates the output power of the converter according to this reference value. This ensures that the wind and photovoltaic power generation subsystem work in load-tracking or maximum power-tracking modes so that the optimal operation of hybrid power generation meets the balance of supply and demand while prolonging the service life of the batteries. Simulation experiments show that the distributed ADMM algorithm can reliably address the power optimization challenge of hybrid power generation systems.

Privacy-Preserving Distributed ADMM With Event-Triggered Communication.

This article addresses distributed optimization problems, in which a group of agents cooperatively minimize the sum of their private objective functions via information exchanging. Building on alternating direction method of multipliers (ADMM), we propose a privacy-preserving and communication-efficient decentralized quadratically approximated ADMM algorithm, termed PC-DQM, for solving such type of problems under the scenario of limited communication. In PC-DQM, an event-triggered mechanism is designed to schedule the communication instants for reducing communication cost. Simultaneously, for privacy preservation, a Hessian matrix with perturbed noise is introduced to quadratically approximate the objective function, which results in a closed form of primal vector update and then avoids solving a subproblem at each iteration with possible high computation cost. In addition, the triggered scheme is also utilized to schedule the update of Hessian, which can also reduce computation cost. We theoretically show that PC-DQM can protect privacy but without losing accuracy. In addition, we rigorously prove that PC-DQM converges linearly to the exact optimal solution for strongly convex and smooth objective functions. Finally, numerical simulation is presented to illustrate the effectiveness and efficiency of our algorithm.

Imputed quantile vector autoregressive model for multivariate spatial–temporal data

AbstractImputing missing values in multivariate spatial–temporal data is important in many fields. Existing low rank tensor learning methods are popular for handling this task but are sensitive to high level of skewness. The aim of this paper is to develop an alternative method with robustness and high imputation accuracy for multivariate spatial–temporal data. In view of the fact that quantile regression is robust to noises and outliers, we propose an imputed quantile vector autoregressive (IQVAR) model. IQVAR can simultaneously impute missing values and estimate parameters of quantile vector autoregressive model. The objective function includes check loss and nuclear norm penalization. We develop an ADMM (Alternating Direction Method of Multipliers) algorithm to solve the resulting optimization problem. Simulation studies and real data analysis are conducted to verify the efficiency of IQVAR. Compared with other approaches, IQVAR is more robust and accurate.

Optimizing Logistic Regression with Enhanced Convergence: A Modified ADMM Approach

In this study, we present a revised version of the Alternating Direction Method of Multipliers (ADMM) algorithm, specifically developed to enhance the convergence and stability of logistic regression models. Our updated technique achieves substantial enhancements in classification accuracy and model interpretability by integrating supplementary regularization and stability components into the ADMM update equations. The experimental findings on synthetic datasets demonstrate that our improved ADMM algorithm surpasses the conventional ADMM, offering a resilient and efficient solution for logistic regression. In this paper, we emphasize the potential of our adjustments to be a valuable improvement in optimizing logistic regression algorithms.

The ADMM algorithm for audio signal recovery and performance modification with the dual Douglas-Rachford dynamical system

<abstract><p>Practitioners employ operator splitting methods—such as alternating direction method of multipliers (ADMM) and its "dual" Douglas-Rachford method (DR)—to solve many kinds of optimization problems. We provide a gentle introduction to these algorithms, and illustrations of their duality-like relationship in the context of solving basis pursuit problems for audio signal recovery. Recently, researchers have used the dynamical systems associated with the iterates of splitting methods to motivate the development of schemes to improve performance. These developments include a class of methods that act by iteratively minimizing surrogates for a Lyapunov function for the dynamical system. An exemplar of this class is currently state-of-the-art for the feasibility problem of finding wavelets with special structure. Early experimental evidence has also suggested that, when implemented in a primal-dual (ADMM and DR) framework, this exemplar may provide improved performance for basis pursuit problems. We provide a reasonable way to compute the updates for this exemplar, and we study the application of this method to the aforementioned basis pursuit audio problems. We provide experimental results and visualizations of the dynamical system for the dual DR sequence. We observe that for highly structured problems with real data, the algorithmic behavior is noticeably different than for randomly generated problems.</p></abstract>

Research on power sharing optimization strategy based on hybrid multi-microgrid

To alleviate the increasing distributed energy penetration problem in the distribution network under the “double carbon”, a P2P multi-microgrid power sharing and trading method based on the alternating direction method of multipliers (ADMM) are proposed in the hybrid multi-microgrid. The paper constructs the optimization model of each sub-microgrid and a multi-microgrid model for P2P power sharing transactions based on the ADMM algorithm. Finally, the proposed model is simulated and verified using a multi-microgrid composed of three microgrids. The results show that the electric energy trading tariff between microgrids obtained from the model in this paper is reasonable, which can improve the utilization rate of new energy and reduce the penetration rate of distributed energy in the distribution network.

Adaptive Trend Filtering for ECG Denoising and Delineation.

Standard recordings of electrocardiograhic signals are contaminated by a large variety of noises and interferences, which impair their analysis and the further related diagnosis. In this article, we propose a method, based on compressive sensing techniques, to remove the main noise artifacts and to locate the main features of the pulses in the electrocardiogram (ECG). The motivation is to use trend filtering with a varying proximal parameter, in order to sequentially capture the peaks of the ECG, which have different functional regularities. The practical implementation is based on an adaptive version of the alternating direction method of multiplier (ADMM) algorithm. We present results obtained on simulated signals and on real data illustrating the validity of this approach, showing that results in peak localization are very good in both cases and comparable to state of the art approaches.

Multi-block alternating direction method of multipliers for ultrahigh dimensional quantile fused regression

In this paper, we consider a quantile fused LASSO regression model that combines quantile regression loss with the fused LASSO penalty. Intuitively, this model offers robustness to outliers, thanks to the quantile regression, while also effectively recovering sparse and block coefficients through the fused LASSO penalty. To adapt our proposed method for ultrahigh dimensional datasets, we introduce an iterative algorithm based on the multi-block alternating direction method of multipliers (ADMM). Moreover, we demonstrate the global convergence of the algorithm and derive comparable convergence rates. Importantly, our ADMM algorithm can be easily applied to solve various existing fused LASSO models. In terms of theoretical analysis, we establish that the quantile fused LASSO can achieve near oracle properties with a practical penalty parameter, and additionally, it possesses a sure screening property under a wide class of error distributions. The numerical experimental results support our claims, showing that the quantile fused LASSO outperforms existing fused regression models in robustness, particularly under heavy-tailed distributions.

Activating demand side flexibility market in a fully decentralized P2P transactive energy trading framework using ADMM algorithm

The communication network development and rising penetration of prosumers in distribution networks pave the way to emerge a fully decentralized energy transactions framework. Prosumers can share their excess energy production in a new market called the peer-to-peer (P2P) market. This paper proposes a decentralized P2P energy trading market that facilitates electrical energy trading between customers. In this new market, customers act as individual players and decide on their own willingness to maximize their social welfare by selling the extra energy production in the P2P market. Other participants can compensate for their energy deficiency in the P2P market at lower prices. In the decentralized framework, the participant's privacy is protected by sharing a limited amount of data between peers. The P2P market paved the way for another technology named the DSF market. The DSF market is activated by the distribution system operator (DSO) in the local P2P market framework. When congestion is detected in the following day, the DSO requests flexibility from the demand-side flexibility (DSF) and publishes a flexibility requirement table (FRT). Such market structure can incentivize customers to participate in DSF programs and help the main grid by mitigating congestion. This paper employed the alternating direction method of multipliers (ADMM) algorithm for solving the decentralized market. Numerical studies are carried out for several peers in a local community. Simulation results demonstrate that the customers participating in the P2P market supply 15.9% of their demand within the P2P market. Additionally, the findings indicate that customers exhibit a willingness to engage in the DSF program. Moreover, it is observed that 99% of the demand changes requested from the DSO in the FRT are adhered to by the peers.