A Novel Approach to Large-Scale Dynamically Weighted Directed Network Representation.

A dynamically weighted directed network (DWDN) is frequently encountered in various big data-related applications like a terminal interaction pattern analysis system (TIPAS) concerned in this study. It consists of large-scale dynamic interactions among numerous entities. Moreover, as the involved entities increase drastically, it becomes impossible to observe their full interactions at each time span, making a corresponding DWDN high-dimensional and incomplete. However, it contains vital knowledge regarding involved entities' behavior patterns. To extract such knowledge from DWDN, this paper proposes a novel Alternating direction method of multipliers (ADMM)-based Nonnegative Latent-factorization of Tensors (ANLT) model. It adopts two novel ideas: a) building a data density-oriented augmented Lagrangian function for efficiently handling a tensor's incompleteness and nonnegativity; and b) splitting an optimization task in each iteration into an elaborately designed subtask series where each one is solved based on the previously solved ones following the ADMM principle to achieve fast model convergence. Experimental results on two large-scale DWDNs from a real TIPAS demonstrate that the proposed ANLT model outperforms state-of-the-art models significantly in both computational efficiency and prediction accuracy when addressing missing link prediction on DWDW.

Purpose: In radiation therapy optimization the constraints can be either hard constraints which must be satisfied or soft constraints which are included but do not need to be satisfied exactly. Currently the voxel dose constraints are viewed as soft constraints and included as a part of the objective function and approximated as an unconstrained problem. However in some treatment planning cases the constraints should be specified as hard constraints and solved by constrained optimization. The goal of this work is to present a computation efficiency graph form alternating direction method of multipliers (ADMM) algorithm for constrained quadratic treatment planning optimization and compare it with several commonly used algorithms/toolbox. Method: ADMM can be viewed as an attempt to blend the benefits of dual decomposition and augmented Lagrangian methods for constrained optimization. Various proximal operators were first constructed as applicable to quadratic IMRT constrained optimization and the problem was formulated in a graph form of ADMM. A pre-iteration operation for the projection of a point to a graph was also proposed to further accelerate the computation. Result: The graph form ADMM algorithm was tested by the Common Optimization for Radiation Therapy (CORT) dataset including TG119, prostate, liver, and head & neck cases. Both unconstrained and constrained optimization problems were formulated for comparison purposes. All optimizations were solved by LBFGS, IPOPT, Matlab built-in toolbox, CVX (implementing SeDuMi) and Mosek solvers. For unconstrained optimization, it was found that LBFGS performs the best, and it was 3–5 times faster than graph form ADMM. However, for constrained optimization, graph form ADMM was 8 – 100 times faster than the other solvers. Conclusion: A graph form ADMM can be applied to constrained quadratic IMRT optimization. It is more computationally efficient than several other commercial and noncommercial optimizers and it also used significantly less computer memory.

An HDI dynamic network contains a multitude of knowledge regarding involved nodes’ various behavior patterns. How to accurately represent such an HDI dynamic network is of the essence to effectively extract knowledge. Therefore, in order to accomplish precisely represent to an HDI dynamic network, this chapter present a novel Alternating direction method of multipliers (ADMM)-based Nonnegative Latent-factorization of Tensors (ANLT) model. It adopts two-fold ideas: (a) building a data density-oriented augmented Lagrangian function to efficiently handle the incompleteness and non-negativity of an HDI tensor; and (b) dividing the optimization task in each iteration into a skillfully designed subtask series where each one is solved depended on the previously solved ones following the principle of ADMM to achieve fast convergence. Empirical studies on six different size dynamic networks demonstrate that compared with several state-of-the-art models, the proposed ANLT model achieves significant gain in prediction accuracy and computational efficiency for predicting missing links of an HDI dynamic network.

The Alternating Direction Method of Multipliers (ADMM) has gained a lot of attention for solving large-scale and objective-separable constrained optimization. However, the two-block variable structure of the ADMM still limits the practical computational efficiency of the method, because one big matrix factorization is needed at least once even for linear and convex quadratic programming. This drawback may be overcome by enforcing a multi-block structure of the decision variables in the original optimization problem. Unfortunately, the multi-block ADMM, with more than two blocks, is not guaranteed to be convergent. On the other hand, two positive developments have been made: first, if in each cyclic loop one randomly permutes the updating order of the multiple blocks, then the method converges in expectation for solving any system of linear equations with any number of blocks. Secondly, such a randomly permuted ADMM also works for equality-constrained convex quadratic programming even when the objective function is not separable. The goal of this paper is twofold. First, we add more randomness into the ADMM by developing a randomly assembled cyclic ADMM (RAC-ADMM) where the decision variables in each block are randomly assembled. We discuss the theoretical properties of RAC-ADMM and show when random assembling helps and when it hurts, and develop a criterion to guarantee that it converges almost surely. Secondly, using the theoretical guidance on RAC-ADMM, we conduct multiple numerical tests on solving both randomly generated and large-scale benchmark quadratic optimization problems, which include continuous, and binary graph-partition and quadratic assignment, and selected machine learning problems. Our numerical tests show that the RAC-ADMM, with a variable-grouping strategy, could significantly improve the computation efficiency on solving most quadratic optimization problems.

Alternating direction method of multiplier (ADMM) is a widely used algorithm for solving constrained optimization problems in image restoration. Among many useful features, one critical feature of the ADMM algorithm is its modular structure, which allows one to plug in any off-the-shelf image denoising algorithm for a subproblem in the ADMM algorithm. Because of the plug-in nature, this type of ADMM algorithms is coined the name “Plug-and-Play ADMM.” Plug-and-Play ADMM has demonstrated promising empirical results in a number of recent papers. However, it is unclear under what conditions and by using what denoising algorithms would it guarantee convergence. Also, since Plug-and-Play ADMM uses a specific way to split the variables, it is unclear if fast implementation can be made for common Gaussian and Poissonian image restoration problems. In this paper, we propose a Plug-and-Play ADMM algorithm with provable fixed-point convergence. We show that for any denoising algorithm satisfying an asymptotic criteria, called bounded denoisers, Plug-and-Play ADMM converges to a fixed point under a continuation scheme. We also present fast implementations for two image restoration problems on superresolution and single-photon imaging. We compare Plug-and-Play ADMM with state-of-the-art algorithms in each problem type and demonstrate promising experimental results of the algorithm.

Privacy-preserving and communication-efficient stochastic alternating direction method of multipliers for federated learning

In this paper, we show that for a class of linearly constrained convex composite optimization problems, an (inexact) symmetric Gauss–Seidel based majorized multi-block proximal alternating direction method of multipliers (ADMM) is equivalent to an inexact proximal augmented Lagrangian method. This equivalence not only provides new perspectives for understanding some ADMM-type algorithms but also supplies meaningful guidelines on implementing them to achieve better computational efficiency. Even for the two-block case, a by-product of this equivalence is the convergence of the whole sequence generated by the classic ADMM with a step-length that exceeds the conventional upper bound of $$(1+\sqrt{5})/2$$ , if one part of the objective is linear. This is exactly the problem setting in which the very first convergence analysis of ADMM was conducted by Gabay and Mercier (Comput Math Appl 2(1):17–40, 1976), but, even under notably stronger assumptions, only the convergence of the primal sequence was known. A collection of illustrative examples are provided to demonstrate the breadth of applications for which our results can be used. Numerical experiments on solving a large number of linear and convex quadratic semidefinite programming problems are conducted to illustrate how the theoretical results established here can lead to improvements on the corresponding practical implementations.

The Light Transport Matrix (LTM) is a fundamental expression of the light propagation of the projector-camera system. The matrix includes all the characteristics of light rays transferred from the projector to the camera, and it is used for scene relighting, understanding the light path, and 3D measurement. Especially, LTM enables robust 3D measurement even if the scene includes metallic or semi-transparent objects; thus it is already used for robot vision. The LTM is often estimated by l 1 minimization because the LTM has a huge number of elements. l 1 minimization methods, which utilize the Alternating Direction Method of Multipliers (ADMM), can reduce the number of observations. In addition, a powerful extended ADMM l 1 minimization method named Saturation ADMM, which can estimate the LTM under saturated conditions, also exists. In the study presented in this paper, we reduce the computational cost of ADMM l 1 minimization by applying the Sherman-Morrison-Woodbury (SMW) formula. Furthermore, we propose multi-row simultaneous LTM estimation, which is a new method to improve the computational efficiency. The contribution of this paper is to propose the use of these two methods to speed up LTM estimation and demonstrate that our methods reduce the computational cost in theory and the calculation time in practice. Experiments indicate that our method accelerates ADMM l 1 minimization by up to 4.64 times, and Saturation ADMM l 1 minimization by up to 2.54 times compared to the original methods.

This paper develops the application of the alternating direction method of multipliers (ADMM) to optimize a dynamic objective function in a decentralized multi-agent system. At each time slot, agents in the network observe local functions and cooperate to track the optimal time-varying argument of the sum objective. This cooperation is based on maintaining local primal variables that estimate the value of the optimal argument and auxiliary dual variables that encourage proximity with neighboring estimates. Primal and dual variables are updated by an ADMM iteration that can be implemented in a distributed manner whereby local updates require access to local variables and the most recent primal variables from adjacent agents. For objective functions that are strongly convex and have Lipschitz continuous gradients, the distances between the primal and dual iterates to their corresponding time-varying optimal values are shown to converge to a steady state gap. This gap is explicitly characterized in terms of the condition number of the objective function, the condition number of the network that is defined as the ratio between the largest and smallest nonzero Laplacian eigenvalues, and a bound on the drifts of the optimal primal variables and the optimal gradients. Numerical experiments corroborate theoretical findings and show that the results also hold for non-differentiable and non-strongly convex primal objectives.

To embrace the era of big data, there has been growing interest in designing distributed machine learning to exploit the collective computing power of the local computing nodes. Alternating Direction Method of Multipliers (ADMM) is one of the most popular methods. This method applies iterative local computations over local datasets at each agent and computation results exchange between the neighbors. During this iterative process, data privacy leakage arises when performing local computation over sensitive data. Although many differentially private ADMM algorithms have been proposed to deal with such privacy leakage, they still have to face many challenging issues such as low model accuracy over strict privacy constraints and requiring strong assumptions of convexity of the objective function. To address those issues, in this paper, we propose a differentially private robust ADMM algorithm (PR-ADMM) with Gaussian mechanism. We employ two kinds of noise variance decay schemes to carefully adjust the noise addition in the iterative process and utilize a threshold to eliminate the too noisy results from neighbors. We also prove that PR-ADMM satisfies dynamic zero-concentrated differential privacy (dynamic zCDP) and a total privacy loss is given by $ (\epsilon, \delta)$-differential privacy. From a theoretical point of view, we analyze the convergence rate of PR-ADMM for general convex objectives, which is $\mathcal{O}(1 /K)$ with K being the number of iterations. The performance of the proposed algorithm is evaluated on real-world datasets. The experimental results show that the proposed algorithm outperforms other differentially private ADMM based algorithms under the same total privacy loss.

In a class of large-scale distributed optimization, the calculation of RELM based on the Moore–Penrose inverse matrix is prohibitively expensive, which hinders the formulation of a computationally efficient optimization model. Attempting to improve the model’s convergence performance, this paper proposes a low computing cost Alternating Direction Method of Multipliers (ADMM), where the original update in ADMM is solved inexactly with approximate curvature information. Based on quasi-Newton techniques, the ADMM approach allows us to solve convex optimization with reasonable accuracy and computational effort. By introducing this algorithm into the RELM model, the model fitting problem can be decomposed into a set of subproblems that can be executed in parallel to achieve efficient classification performance. To avoid the storage of expensive Hessian for large problems, BFGS with limited memory is proposed with computational efficiency. And the optimal parameter values of the step-size search method are obtained through Wolfe line search strategy. To demonstrate the superiority of our methods, numerical experiments are conducted on eight real-world datasets. Results on problems arising in machine learning suggest that the proposed method is competitive with other similar methods, both in terms of better computational efficiency as well as accuracy.

In this paper, the medical CT image blind restoration is translated into two sub problems, namely, image estimation based on dictionary learning and point spread function estimation. A blind restoration algorithm optimized by the alternating direction method of multipliers for medical CT images was proposed. At present, the existing methods of blind image restoration based on dictionary learning have the problem of low efficiency and precision. This paper aims to improve the effectiveness and accuracy of the algorithm and to improve the robustness of the algorithm. The local CT images are selected as training samples, and the K-SVD algorithm is used to construct the dictionary by iterative optimization, which is beneficial to improve the efficiency of the algorithm. Then, the orthogonal matching pursuit algorithm is employed to implement the dictionary update. Dictionary learning is accomplished by sparse representation of medical CT images. The alternating direction method of multipliers (ADMM) is used to solve the objective function and realize the local image restoration, so as to eliminate the influence of point spread function. Secondly, the local restoration image is used to estimate the point spread function, and the convex quadratic optimization method is used to solve the point spread function sub problems. Finally, the optimal estimation of point spread function is obtained by iterative method, and the global sharp image is obtained by the alternating direction method of multipliers. Experimental results show that, compared with the traditional adaptive dictionary restoration algorithm, the new algorithm improves the objective image quality metrics, such as peak signal to noise ratio, structural similarity, and universal image quality index. The new algorithm optimizes the restoration effect, improves the robustness of noise immunity and improves the computing efficiency.

The Alternating Direction Method of Multipliers (ADMM) and its distributed version have been widely used in machine learning. In the iterations of ADMM, model updates using local private data and model exchanges among agents impose critical privacy concerns. Despite some pioneering works to relieve such concerns, differentially private ADMM still confronts many research challenges. For example, the guarantee of differential privacy (DP) relies on the premise that the optimality of each local problem can be perfectly attained in each ADMM iteration, which may never happen in practice. The model trained by DP ADMM may have low prediction accuracy. In this paper, we address these concerns by proposing a novel (Improved) Plausible differentially Private ADMM algorithm, called PP-ADMM and IPP-ADMM. In PP-ADMM, each agent approximately solves a perturbed optimization problem that is formulated from its local private data in an iteration, and then perturbs the approximate solution with Gaussian noise to provide the DP guarantee. To further improve the model accuracy and convergence, an improved version IPP-ADMM adopts sparse vector technique (SVT) to determine if an agent should update its neighbors with the current perturbed solution. The agent calculates the difference of the current solution from that in the last iteration, and if the difference is larger than a threshold, it passes the solution to neighbors; or otherwise the solution will be discarded. Moreover, we propose to track the total privacy loss under the zero-concentrated DP (zCDP) and provide a generalization performance analysis. Experiments on real-world datasets demonstrate that under the same privacy guarantee, the proposed algorithms are superior to the state of the art in terms of model accuracy and convergence rate.

This paper proposes a customized Alternating Direction Method of Multipliers (ADMM) algorithm to solve the Rank-Constrained Optimization Problems (RCOPs) with approximate formulations. Here RCOP refers to an optimization problem whose objective and constraints are convex except a (nonconvex) matrix rank constraint. We first present an approximate formulation for the RCOP with high accuracy by selecting an appropriate parameter set. Then a general ADMM frame is employed to solve the approximated problem without requiring singular value decomposition in each subproblem. The new formulation and the customized ADMM algorithm greatly enhance the computational efficiency and scalability. While ADMM has been extensively investigated for convex optimization problems, its convergence property is still open for nonconvex problems. Another contribution of this paper is to prove that the proposed ADMM globally converges to a stationary point of the approximate problem of RCOP. Simulation examples are provided to demonstrate the feasibility and efficiency of the proposed method.

Optimization of production networks is key for managing efficient hydrocarbon production as part of closed-loop asset management. Large-scale surface network optimization is a challenging task that involves high nonlinearity with numerous constraints. In existing tools, the computational cost of solving the surface network optimization can exponentially increase with the size and complexities of the network using traditional approaches involving nonlinear programming methods. In this study, we accelerate the large-scale surface network optimization by using a distributed agent optimization algorithm called alternating direction method of multipliers (ADMM). We develop and apply the ADMM algorithm for large-scale network optimization with over 1000 wells and interconnecting pipelines. In the ADMM framework, a large-scale network system is broken down into many small sub-network systems. Then, a smaller optimization problem is formulated for each sub-network. These sub-network optimization problems are solved in parallel using multiple computer cores so that the entire system optimization will be accelerated. A large-scale surface network involves many inequality and equality constraints, which are effectively handled by using augmented Lagrangian method to enhance the robustness of convergence quality. Additionally, proxy or hybrid models can also be used for pipe flow and pressure calculation for every network segment to further speed up the optimization. The proposed ADMM optimization method is validated by several synthetic cases. We first apply the proposed method to surface network simulation problems of various sizes and complexities (configurations, fluid types, pressure regimes, etc.), where the pressure for all nodes and fluxes in all links will be calculated with a specified separator pressure and reservoir pressures. High accuracy was obtained from the ADMM framework compared with a commercial simulator. Next, the ADMM is applied to network optimization problems, where we optimize the pressure drop across a surface choke for every well to maximize oil production. In a large-scale network case with over 1000 wells, we achieve 2X – 3X speedups in computation time with reasonable accuracy from the ADMM framework compared with benchmarks. Finally, we apply the proposed method to a field case, and validate that the ADMM framework properly works for the actual field applications. A novel framework for surface network optimization was developed using the distributed agent optimization algorithm. The proposed framework provides superior computational efficiency for large- scale network optimization problems compared with existing benchmark methods. It enables more efficient and frequent decision-making of large-scale petroleum field management to maximize the hydrocarbon production subject to numerous system constraints.