Stochastic Proximal Linearized ADMM for Sparse Distribution Control Problem Constrained by Random Elliptic Equation

We consider a regularized version of a Jacobi-type alternating direction method of multipliers (ADMM) for the solution of a class of separable convex optimization problems in a Hilbert space. The analysis shows that this method is equivalent to the standard proximal-point method applied in a Hilbert space with a transformed scalar product. The method therefore inherits the known convergence results from the proximal-point method and allows suitable modifications to get a strongly convergent variant. Some additional properties are also shown by exploiting the particular structure of the ADMM-type solution method. Applications and numerical results are provided for the domain decomposition method and potential (generalized) Nash equilibrium problems in a Hilbert space setting.

A wide variety of learning problems can be posed in the framework of convex optimization. Many efficient algorithms have been developed based on solving the induced optimization problems. However, there exists a gap between the theoretically unbeatable convergence rate and the practically efficient learning speed. In this paper, we use the variational inequality (VI) convergence to describe the learning speed. To this end, we avoid the hard concept of regret in online learning and directly discuss the stochastic learning algorithms. We first cast the regularized learning problem as a VI. Then, we present a stochastic version of alternating direction method of multipliers (ADMMs) to solve the induced VI. We define a new VI-criterion to measure the convergence of stochastic algorithms. While the rate of convergence for any iterative algorithms to solve nonsmooth convex optimization problems cannot be better than O(1/√t), the proposed stochastic ADMM (SADMM) is proved to have an O(1/t) VI-convergence rate for the l1-regularized hinge loss problems without strong convexity and smoothness. The derived VI-convergence results also support the viewpoint that the standard online analysis is too loose to analyze the stochastic setting properly. The experiments demonstrate that SADMM has almost the same performance as the state-of-the-art stochastic learning algorithms but its O(1/t) VI-convergence rate is capable of tightly characterizing the real learning speed.

Alternating direction method of multiplier (ADMM) is a widely used algorithm for solving constrained optimization problems in image restoration. Among many useful features, one critical feature of the ADMM algorithm is its modular structure, which allows one to plug in any off-the-shelf image denoising algorithm for a subproblem in the ADMM algorithm. Because of the plug-in nature, this type of ADMM algorithms is coined the name “Plug-and-Play ADMM.” Plug-and-Play ADMM has demonstrated promising empirical results in a number of recent papers. However, it is unclear under what conditions and by using what denoising algorithms would it guarantee convergence. Also, since Plug-and-Play ADMM uses a specific way to split the variables, it is unclear if fast implementation can be made for common Gaussian and Poissonian image restoration problems. In this paper, we propose a Plug-and-Play ADMM algorithm with provable fixed-point convergence. We show that for any denoising algorithm satisfying an asymptotic criteria, called bounded denoisers, Plug-and-Play ADMM converges to a fixed point under a continuation scheme. We also present fast implementations for two image restoration problems on superresolution and single-photon imaging. We compare Plug-and-Play ADMM with state-of-the-art algorithms in each problem type and demonstrate promising experimental results of the algorithm.

In this paper, we reviewed several forms of alternating direction method of multipliers (ADMM) for distributed power system computing. The major focus is on ADMM based distributed parallel optimization algorithm which is feasible to be implemented in power network. Firstly, we introduced the general form of ADMM, and extend the 2-block ADMM to N-block multi-block ADMM. Next, we focus on two distributed parallel ADMM based optimization algorithms: Consensus ADMM (C-ADMM) and Proximal Jacobian ADMM (PJ-ADMM). A three-area DC optimal power flow (OPF) problem and a two-area AC OPF problem are tested for ADMM implementation. Information exchange structure and the numerical convergence results of the ADMM algorithms are given.

In this paper, we propose two novel Alternating Direction Method of Multipliers (ADMM) algorithms for the sparse portfolio problem via sorted ℓ1-norm penalization (SLOPE). The first algorithm (FADMM) is presented by adding a prediction-correction step to the classic ADMM framework. Since the problem is not strongly convex, the second fast ADMM (FADMMR) is proposed by utilizing both prediction-correction step and restarting rules. Numerical experiments show that the FADMMR algorithm converges faster than the FADMM algorithm and ADMM algorithm when tuning parameters are relatively small. On the other hand, when tuning parameters are relative large, the FADMM algorithm performs better than the FADMMR algorithm and ADMM algorithm. The FADMM algorithm and FADMMR algorithm converge faster than the ADMM algorithm in terms of convergence time for different sizes of tuning parameters. For large-scale portfolio problem, the proposed algorithms have highly performance as well. Finally, empirical analysis on five datasets of stocks index show that the proposed algorithms are efficient and superior for solving sparse portfolio problems via SLOPE.

(Communicated by Xinwei Liu) The alternating direction method of multipliers (ADMM) is one of the effective methods for solving two-block separable optimization problems with linear constraints, and has been widely applied in image processing, power systems, sparse learning, and other fields. Its essence is the application of the Douglas-Rachford splitting method to the dual problem. Classical ADMM updates the Lagrange multipliers only once per iteration, while symmetric ADMM achieves dual updates of the Lagrange multipliers in each iteration by introducing an additional multiplier update step, thereby significantly improving the convergence performance and numerical stability of the algorithm. In this paper, we propose a novel symmetric ADMM algorithmic framework for two-block separable nonconvex optimization problems with linear constraints, which introduces two distinct relaxation factors to enhance the flexibility and convergence efficiency of the algorithm. This method has been comprehensively studied for convex problems. However, for nonconvex problems, in the convergence analysis of symmetric alternating direction method of multipliers with two different relaxation factors without introducing Bregman distances or regularization terms, proving the monotonicity of the Lagrangian function remains a challenging problem. In terms of theoretical analysis, we establish the convergence theory of the proposed algorithm in this paper. Notably, our convergence proof does not rely on common technical assumptions such as Bregman distances or regularization terms. Specifically, by constructing a novel auxiliary function and under the mild condition that the Kurdyka–Łojasiewicz (KŁ) inequality is satisfied, we prove that the iterative sequence generated by the algorithm converges to a stationary point of the problem. The main contributions of this paper can be summarized as follows: First, we design a symmetric ADMM scheme with dual relaxation factors to solve two-block separable nonconvex and nonsmooth optimization problems with linear constraints, and the proposed method allows for a wider range of parameters, which can better adapt to the structural characteristics of different problems through flexible adjustment of relaxation parameters. Second, we establish a concise convergence analysis framework that does not depend on Bregman distances and regularization terms, reducing the complexity of theoretical analysis; moreover, it can degenerate into the classical ADMM. Finally, we validate the practical application effectiveness of the proposed algorithm through numerical experiments, and the experimental results demonstrate that the algorithm outperforms traditional ADMM and its variants in terms of convergence speed and solution accuracy.

One of the main challenges for optimal designs of two-dimensional (2D) finite impulse response (FIR) filters is their heavy computational load due to the large number of filter coefficients and the high dimensions of the data for model fitting. The alternating direction method of multipliers (ADMM) is a powerful technique appropriate for optimization for big data. In this paper, a relaxed ADMM is presented and then applied in the weighted least-squares (WLS) design of linear-phase 2D FIR filters. It is shown that the relaxed ADMM algorithm converges much faster than the standard ADMM algorithm. In addition, a salient feature of the relaxed ADMM is its highly parallel structure which makes it very efficient if implemented in parallel. Simulation examples and comparisons demonstrate the fast convergence and high efficiency of the relaxed ADMM algorithm.

Numerous applied models used in the study of optimal control problems, inverse problems, shape optimization, machine learning, fractional programming, neural networks, image registration and so on lead to stochastic optimization problems in Hilbert spaces. Under a suitable convexity assumption on the objective function, a necessary and sufficient optimality condition for stochastic optimization problems is a stochastic variational inequality. This article presents a new stochastic regularized second-order iterative scheme for solving a variational inequality in a stochastic environment where the primary operator is accessed by employing sampling techniques. The proposed iterative scheme, which fits within the general framework of the stochastic approximation approach, has its almost-sure convergence analysis given in a Hilbert space. We test the feasibility and the efficacy of the proposed stochastic approximation approach for a stochastic optimal control problem and a stochastic inverse problem, both associated with a second-order stochastic partial differential equation. This article is part of the theme issue 'Non-smooth variational problems and applications'.

Alternating direction method of multipliers (ADMM) is a popular first-order method owing to its simplicity and efficiency. However, similar to other proximal splitting methods, the performance of ADMM degrades significantly when the scale of optimization problems to solve becomes large. In this paper, we consider combining ADMM with a class of variance-reduced stochastic gradient estimators for solving large-scale non-convex and non-smooth optimization problems. Global convergence of the generated sequence is established under the additional assumption that the object function satisfies Kurdyka-Łojasiewicz property. Numerical experiments on graph-guided fused lasso and computed tomography are presented to demonstrate the performance of the proposed methods.

In this paper, we study a general optimization model, which covers a large class of existing models for many applications in imaging sciences. To solve the resulting possibly nonconvex, nonsmooth, and non-Lipschitz optimization problem, we adapt the alternating direction method of multipliers (ADMM) with a general dual step-size to solve a reformulation that contains three blocks of variables and analyze its convergence. We show that for any dual step-size less than the golden ratio, there exists a computable threshold such that if the penalty parameter is chosen above such a threshold and the sequence thus generated by our ADMM is bounded, then the cluster point of the sequence gives a stationary point of the nonconvex optimization problem. We achieve this via a potential function specifically constructed for our ADMM. Moreover, we establish the global convergence of the whole sequence if, in addition, this special potential function is a Kurdyka--Łojasiewicz function. Furthermore, we present a simple st...

Purpose: In radiation therapy optimization the constraints can be either hard constraints which must be satisfied or soft constraints which are included but do not need to be satisfied exactly. Currently the voxel dose constraints are viewed as soft constraints and included as a part of the objective function and approximated as an unconstrained problem. However in some treatment planning cases the constraints should be specified as hard constraints and solved by constrained optimization. The goal of this work is to present a computation efficiency graph form alternating direction method of multipliers (ADMM) algorithm for constrained quadratic treatment planning optimization and compare it with several commonly used algorithms/toolbox. Method: ADMM can be viewed as an attempt to blend the benefits of dual decomposition and augmented Lagrangian methods for constrained optimization. Various proximal operators were first constructed as applicable to quadratic IMRT constrained optimization and the problem was formulated in a graph form of ADMM. A pre-iteration operation for the projection of a point to a graph was also proposed to further accelerate the computation. Result: The graph form ADMM algorithm was tested by the Common Optimization for Radiation Therapy (CORT) dataset including TG119, prostate, liver, and head & neck cases. Both unconstrained and constrained optimization problems were formulated for comparison purposes. All optimizations were solved by LBFGS, IPOPT, Matlab built-in toolbox, CVX (implementing SeDuMi) and Mosek solvers. For unconstrained optimization, it was found that LBFGS performs the best, and it was 3–5 times faster than graph form ADMM. However, for constrained optimization, graph form ADMM was 8 – 100 times faster than the other solvers. Conclusion: A graph form ADMM can be applied to constrained quadratic IMRT optimization. It is more computationally efficient than several other commercial and noncommercial optimizers and it also used significantly less computer memory.

In this two-part study, we develop a general theory of the so-called exact augmented Lagrangians for constrained optimization problems in Hilbert spaces. In contrast to traditional nonsmooth exact penalty functions, these augmented Lagrangians are continuously differentiable for smooth problems and do not suffer from the Maratos effect, which makes them especially appealing for applications in numerical optimization. Our aim is to present a detailed study of various theoretical properties of exact augmented Lagrangians and discuss several applications of these functions to constrained variational problems, problems with PDE constraints, and optimal control problems. The first paper is devoted to a theoretical analysis of an exact augmented Lagrangian for optimization problems in Hilbert spaces. We obtain several useful estimates of this augmented Lagrangian and its gradient, and present several types of sufficient conditions for KKT-points of a constrained problem corresponding to locally/globally optimal solutions to be local/global minimizers of the exact augmented Lagrangian.

The efficient optimization method for locally Lipschitz continuous multiobjective optimization problems from Gebken and Peitz (J Optim Theory Appl 188:696–723, 2021) is extended from finite-dimensional problems to general Hilbert spaces. The method iteratively computes Pareto critical points, where in each iteration, an approximation of the Clarke subdifferential is computed in an efficient manner and then used to compute a common descent direction for all objective functions. To prove convergence, we present some new optimality results for nonsmooth multiobjective optimization problems in Hilbert spaces. Using these, we can show that every accumulation point of the sequence generated by our algorithm is Pareto critical under common assumptions. Computational efficiency for finding Pareto critical points is numerically demonstrated for multiobjective optimal control of an obstacle problem.

To embrace the era of big data, there has been growing interest in designing distributed machine learning to exploit the collective computing power of the local computing nodes. Alternating Direction Method of Multipliers (ADMM) is one of the most popular methods. This method applies iterative local computations over local datasets at each agent and computation results exchange between the neighbors. During this iterative process, data privacy leakage arises when performing local computation over sensitive data. Although many differentially private ADMM algorithms have been proposed to deal with such privacy leakage, they still have to face many challenging issues such as low model accuracy over strict privacy constraints and requiring strong assumptions of convexity of the objective function. To address those issues, in this paper, we propose a differentially private robust ADMM algorithm (PR-ADMM) with Gaussian mechanism. We employ two kinds of noise variance decay schemes to carefully adjust the noise addition in the iterative process and utilize a threshold to eliminate the too noisy results from neighbors. We also prove that PR-ADMM satisfies dynamic zero-concentrated differential privacy (dynamic zCDP) and a total privacy loss is given by $ (\epsilon, \delta)$-differential privacy. From a theoretical point of view, we analyze the convergence rate of PR-ADMM for general convex objectives, which is $\mathcal{O}(1 /K)$ with K being the number of iterations. The performance of the proposed algorithm is evaluated on real-world datasets. The experimental results show that the proposed algorithm outperforms other differentially private ADMM based algorithms under the same total privacy loss.

The Alternating Direction Method of Multiplier (ADMM) is a simple algorithm to resolve decomposable convex optimization problems, especially effective in solving large-scale problems. However, this algorithm suffers from the straggler problem its updates have to be synchronized. Therefore, the asynchronous ADMM algorithm is proposed. However, the convergence speed of the ADMM algorithm is not very satisfactory. In this paper, we propose a dynamic scheduling strategy for sub-problems-automatically calling different algorithms at different iteration periods of each iteration, and combining this strategy with a hierarchical communication structure. The experiments based on ZiQiang 4000 cluster experimental environment show that the dynamic scheduling strategy based on hierarchical communication structure can solve the ADMM sub-problem and effectively improve the convergence speed and communication efficiency of the algorithm.