Separable Convex Optimization Research Articles

Proximal Alternating Direction Method of Multipliers with Convex Combination Proximal Centers

Proximal alternating direction method of multipliers (PADMM) is a classical primal-dual splitting method for solving separable convex optimization problems with linear equality constraints, which have numerous applications in, e.g., signal and image processing, machine learning, and statistics. In this paper, we propose a new variant of PADMM, called PADMC, whose proximal centers are constructed by convex combinations of the iterates. PADMC is able to take advantage of problem structures and preserves the desirable properties of the classical PADMM. We establish iterate convergence as well as [Formula: see text] ergodic and [Formula: see text] nonergodic sublinear convergence rate results measured by function residual and feasibility violation, where [Formula: see text] denotes the iteration number. Moreover, we propose two fast variants of PADMC, one achieves faster [Formula: see text] ergodic convergence rate when one of the component functions is strongly convex, and the other ensures faster [Formula: see text] nonergodic convergence rate measured by constraint violation. Finally, preliminary numerical results on the LASSO and the elastic-net regularization problems are presented to demonstrate the performance of the proposed methods.

Parallel Alternating Direction Primal-Dual (PADPD) Algorithm for Multi-Block Centralized Optimization

Abstract In this article, a centralized two-block separable convex optimization with equality constraint and its extension to multi-block optimization are considered. The first fully parallel primal-dual discrete-time algorithm called Parallel Alternating Direction Primal-Dual (PADPD) is proposed. In the algorithm, the primal variables are updated in an alternating fashion like Alternating Direction Method of Multipliers (ADMM). The algorithm can handle non-smoothness of objective functions with strong convergence. Unlike existing discrete-time algorithms such as Method of Multipliers (MM), ADMM, Parallel ADMM, Bi-Alternating Direction Method of Multipliers (Bi-ADMM), and Primal-Dual Fixed Point (PDFP) algorithms, all primal and dual variables in the proposed algorithm are updated independently. Therefore, the time complexity of the algorithm can be significantly reduced. It is shown that the rate of convergence of the algorithm for quadratic or linear cost functions is exponential or linear under suitable assumptions. The algorithm can be directly extended to any finite multi-block optimization without further assumptions while preserving its convergence. PADPD algorithm not only can compute more iterations (since it is fully parallel) for the same time-step but it is also possible that PADPD algorithm can have a faster convergence rate than that of ADMM. Finally, two numerical examples are provided in order to show advantage of PADPD algorithm.

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

AbstractAlternating direction method of multipliers (ADMM) receives much attention in the field of optimization and computer science, etc. The generalized ADMM (G-ADMM) proposed by Eckstein and Bertsekas incorporates an acceleration factor and is more efficient than the original ADMM. However, G-ADMM is not applicable in some models where the objective function value (or its gradient) is computationally costly or even impossible to compute. In this paper, we consider the two-block separable convex optimization problem with linear constraints, where only noisy estimations of the gradient of the objective function are accessible. Under this setting, we propose a stochastic linearized generalized ADMM (called SLG-ADMM) where two subproblems are approximated by some linearization strategies. And in theory, we analyze the expected convergence rates and large deviation properties of SLG-ADMM. In particular, we show that the worst-case expected convergence rates of SLG-ADMM are $\mathcal{O}\left( {{N}^{-1/2}}\right)$ and $\mathcal{O}\left({\ln N} \cdot {N}^{-1}\right)$ for solving general convex and strongly convex problems, respectively, where N is the iteration number, similarly hereinafter, and with high probability, SLG-ADMM has $\mathcal{O}\left ( \ln N \cdot N^{-1/2} \right ) $ and $\mathcal{O}\left ( \left ( \ln N \right )^{2} \cdot N^{-1} \right ) $ constraint violation bounds and objective error bounds for general convex and strongly convex problems, respectively.

An inexact version of the symmetric proximal ADMM for solving separable convex optimization

In this paper, we propose and analyze an inexact version of the symmetric proximal alternating direction method of multipliers (ADMM) for solving linearly constrained optimization problems. Basically, the method allows its first subproblem to be solved inexactly in such way that a relative approximate criterion is satisfied. In terms of the iteration number k, we establish global \(\boldsymbol {\mathcal {O}} \mathbf {(1/ \sqrt {k})}\) pointwise and \({\boldsymbol {\mathcal {O}} \mathbf {(1/ {k}})}\) ergodic convergence rates of the method for a domain of the acceleration parameters, which is consistent with the largest known one in the exact case. Since the symmetric proximal ADMM can be seen as a class of ADMM variants, the new algorithm as well as its convergence rates generalize, in particular, many others in the literature. Numerical experiments illustrating the practical advantages of the method are reported. To the best of our knowledge, this work is the first one to study an inexact version of the symmetric proximal ADMM.

A Multi-Step Inertial Proximal Peaceman-Rachford Splitting Method for Separable Convex Programming

In this paper, we propose a multi-step inertial proximal Peaceman-Rachford splitting method (abbreviated as MIP-PRSM) for solving the two-block separable convex optimization problems with linear constraints, which is a unified framework for such Peaceman-Rachford splitting methods (PRSM)-based improved algorithms with inertial step. Furthermore, we establish the global convergence of the MIP-PRSM under some assumptions. Finally, some numerical experimental results on the least squares semidefinite programming (LSSP), LASSO, the convex quadratic programming problem (CQPP), total variation (TV) based denoising and medical image reconstruction problems demonstrate the efficiency of the proposed method.

An inexact ADMM with proximal-indefinite term and larger stepsize

In this paper, an inexact Alternating Direction Method of Multipliers (ADMM) has been proposed for solving the two-block separable convex optimization problem subject to linear equality constraints. The first resulting subproblem is solved inexactly under relative error criterion, while another subproblem called regularization problem is solved inexactly by introducing an indefinite proximal term. Meanwhile, the dual variable is updated twice with relatively larger stepsizes since an indefinite proximal term is exploited. By reformulating the first-order optimality conditions of the involved subproblems as a monotone inclusion problem, the sublinear convergence rate of the proposed algorithm is established in the pointwise and ergodic sense. Numerical experiments on testing two sparse optimization problems from statistical learning and image restoration indicate that our inexact ADMM performs much better than several well-established algorithms.

Approximate Customized Proximal Point Algorithms for Separable Convex Optimization

Approximate Customized Proximal Point Algorithms for Separable Convex Optimization

A golden ratio proximal alternating direction method of multipliers for separable convex optimization

A golden ratio proximal alternating direction method of multipliers for separable convex optimization

An inexact accelerated stochastic ADMM for separable convex optimization

An inexact accelerated stochastic Alternating Direction Method of Multipliers (AS-ADMM) scheme is developed for solving structured separable convex optimization problems with linear constraints. The objective function is the sum of a possibly nonsmooth convex function and a smooth function which is an average of many component convex functions. Problems having this structure often arise in machine learning and data mining applications. AS-ADMM combines the ideas of both ADMM and the stochastic gradient methods using variance reduction techniques. One of the ADMM subproblems employs a linearization technique while a similar linearization could be introduced for the other subproblem. For a specified choice of the algorithm parameters, it is shown that the objective error and the constraint violation are $$\mathcal {O}(1/k)$$ relative to the number of outer iterations k. Under a strong convexity assumption, the expected iterate error converges to zero linearly. A linearized variant of AS-ADMM and incremental sampling strategies are also discussed. Numerical experiments with both stochastic and deterministic ADMM algorithms show that AS-ADMM can be particularly effective for structured optimization arising in big data applications.

Linear Convergence of Prox-SVRG Method for Separable Non-smooth Convex Optimization Problems under Bounded Metric Subregularity

Linear Convergence of Prox-SVRG Method for Separable Non-smooth Convex Optimization Problems under Bounded Metric Subregularity

The Extension of Auxiliary Problem Principle for Solving Dynamic Economic Dispatch Problem in Power Systems

In this paper, we study the convex optimization problem with linear constraint, and the objective function is composed of m separable convex functions. Considering the special case where the objective function is composed of two separable convex functions, the auxiliary problem principle (APP) is an effective parallel distributed algorithm for solving the special case. Inspired by the principle of APP, a natural idea to solve separable convex optimization problem with m ≥ 3 is to extend the method of APP, resulting in the APP‐like algorithm. The convergence of the APP‐like algorithm is not clear yet. In this paper, we give a sufficient condition for the convergence of the APP‐like algorithm. Specifically, the APP algorithm is a special case of the APP‐like algorithm when m = 2. However, simulation results show that the convergence efficiency of the APP‐like algorithm is affected by the selection of penalty parameter. Therefore, we propose an improved APP‐like algorithm in this paper. Simulation results show that the improved APP‐like algorithm is robust to the selection of penalty parameter and that the convergence efficiency of the improved APP‐like algorithm is better when compared with the APP‐like algorithm.

A new accelerated positive-indefinite proximal ADMM for constrained separable convex optimization problems

A new accelerated positive-indefinite proximal ADMM for constrained separable convex optimization problems

A multi-parameter parallel ADMM for multi-block linearly constrained separable convex optimization

The alternating direction method of multipliers (ADMM) has been proved to be effective for solving two-block convex minimization model subject to linear constraints. However, the convergence of multiple-block convex minimization model with linear constraints may not be guaranteed without additional assumptions. Recently, some parallel multi-block ADMM algorithms which solve the subproblems in a parallel way have been proposed. This paper is a further study on this method with the purpose of improving the parallel multi-block ADMM algorithm by introducing more parameters. We propose two multi-parameter parallel ADMM algorithms with proximal point terms attached to all subproblems. Comparing with some popular parallel ADMM-based algorithms, the parameter conditions of the new algorithms are relaxed. Experiments on both real and synthetic problems are conducted to justify the effectiveness of the proposed algorithms compared to several efficient ADMM-based algorithms for multi-block problems.

Modified proximal symmetric ADMMs for multi-block separable convex optimization with linear constraints

We consider the linearly constrained separable convex optimization problem whose objective function is separable with respect to m blocks of variables. A bunch of methods have been proposed and extensively studied in the past decade. Specifically, a modified strictly contractive Peaceman–Rachford splitting method (SC-PRCM) [S. H. Jiang and M. Li, A modified strictly contractive Peaceman–Rachford splitting method for multi-block separable convex programming, J. Ind. Manag. Optim. 14(1) (2018) 397-412] has been well studied in the literature for the special case of m = 3. Based on the modified SC-PRCM, we present modified proximal symmetric ADMMs (MPSADMMs) to solve the multi-block problem. In MPSADMMs, all subproblems but the first one are attached with a simple proximal term, and the multipliers are updated twice. At the end of each iteration, the output is corrected via a simple correction step. Without stringent assumptions, we establish the global convergence result and the O(1/t) convergence rate in the ergodic sense for the new algorithms. Preliminary numerical results show that our proposed algorithms are effective for solving the linearly constrained quadratic programming and the robust principal component analysis problems.

Optimal deterministic and robust selection of electricity contracts

We address the Electricity Contract Selection Problem (ECSP), of finding best parameters of an electricity contract for a client based on his/her past records of electricity consumption over a fixed time period. The objective is to optimize the electricity bill composed by some fixed cost, the cost of the subscription of the electricity contract and penalties due to overpowering when consumption exceeds subscribed power. The ECSP can be formulated as a convex separable optimization problem subject to total order constraints. Due to this special structure, ECSP is a special case of two well known classes of convex separable optimization problems, namely the minimum network flow under convex separable cost and minimizing convex separable functions under chain constraints. Both classes are well treated in the litterature and can be solved in polynomial time (Ahuja and Orlin in Oper Res 49(5):784–789, 2001; Ahuja et al. in Manag Sci 49(7):950–964, 2003; Best et al. in SIAM J Optim 10:658–672, 2000; Karzanov and McCormick in SIAM J Comput 26(4):1245–1275, 1997; Minoux in Eur J Oper Res 18(3):377–387, 1984; Minoux, in Gallo and Sandi (eds) Netflow at pisa, mathematical programming studies, Springer, Berlin, 1986). In particular, the algorithm in Ahuja and Orlin (2001) achieves the best theoretical time complexity assuming that computing the objective function value at one specific point can be done in constant time. However, when we work on a big amount of historic data as in ECSP, the time required for evaluating the objective function cannot be assumed to be O(1) anymore. In this paper, we propose a new algorithm for ECSP which is specially designed to reduce the computational effort over large scale historical data. We present numerical results showing that our algorithm outperforms the algorithm in Ahuja and Orlin (2001) when applied to consumption data of various types of clients. A robust version of ECSP based on a Seasonal and Trend decomposition approach for modelling consumption uncertainty is also investigated. The resulting worst-case cost minimization problem is shown to be efficiently solvable using the same algorithm as for deterministic ECSP.

A customized low-rank prior model for structured cartoon–texture image decomposition

The mathematical characterization of the texture component plays an instrumental role in image decomposition. In this paper, we are concerned with a low-rank texture prior based cartoon–texture image decomposition model, which utilizes a total variation norm and a global nuclear norm to characterize the cartoon and texture components, respectively. It is promising that our decomposition model is not only extremely simple, but also works perfectly for globally well-patterned images in the sense that the model can recover cleaner texture (or details) than the other novel models. Moreover, such a model can be easily reformulated as a separable convex optimization problem, thereby enjoying a splitting nature so that we can employ a partially parallel splitting method (PPSM) to solve it efficiently. A series of numerical experiments on image restoration demonstrate that PPSM can recover slightly higher quality images than some existing algorithms in terms of taking less iterations or computing time in many cases.

A faster generalized ADMM-based algorithm using a sequential updating scheme with relaxed step sizes for multiple-block linearly constrained separable convex programming

The multi-block linearly constrained separable convex optimization is frequently applied in numerous applications, including image/signal processing, statistical learning, and data mining, where the objective function is the sum of multiple individual convex functions, and the key constraints are linear. A classical approach to solving such optimization problem could be the alternating direction method of multipliers (ADMM). It decomposes the subproblem into a series of small-scale ones such that its per-iteration cost may be meager. ADMM, however, is designed initially for the two-block model, and its convergence cannot be guaranteed for a general multi-block model without additional assumptions. Dai et al. (2017) proposed the algorithm SUSLM (for Sequential Updating Scheme of the Lagrange Multiplier) for separable convex programming problems. The Lagrange multipliers are updated several times at each iteration, and a correction step is imposed at the end of each iteration. In order to derive its convergence property, a correction step is imposed at the end of each iteration. In this paper, we improve the SUSLM algorithm by introducing two controlled parameters in the updating expressions for decision variables and Lagrange multipliers. The condition of step sizes is then relaxed. We show experimentally that our SUSLM algorithm converges faster than SUSLM. Moreover, result comparisons on robust principal component analysis (RPCA) show better performances than other ADMM-based algorithms.

Convergence Rates of Inertial Primal-Dual Dynamical Methods for Separable Convex Optimization Problems

In this paper, we propose a second-order continuous primal-dual dynamical system with time-dependent positive damping terms for a separable convex optimization problem with linear equality constrai...

基于广义近似交替方向乘子法求解可分离凸优化问题

基于广义近似交替方向乘子法求解可分离凸优化问题

A Dynamical Approach to Two-Block Separable Convex Optimization Problems with Linear Constraints

The aim of this manuscript is to approach by means of first order differential equations/inclusions convex programming problems with two-block separable linear constraints and objectives, whereby (at least) one of the components of the latter is assumed to be strongly convex. Each block of the objective contains a further smooth convex function. We investigate the dynamical system proposed and prove that its trajectories converge weakly to a saddle point of the Lagrangian of the convex optimization problem. The dynamical system provides through time discretization the alternating minimization algorithm AMA and also its proximal variant recently introduced in the literature.