Separable Convex Optimization Problems Research Articles

Proximal Alternating Direction Method of Multipliers with Convex Combination Proximal Centers

Proximal alternating direction method of multipliers (PADMM) is a classical primal-dual splitting method for solving separable convex optimization problems with linear equality constraints, which have numerous applications in, e.g., signal and image processing, machine learning, and statistics. In this paper, we propose a new variant of PADMM, called PADMC, whose proximal centers are constructed by convex combinations of the iterates. PADMC is able to take advantage of problem structures and preserves the desirable properties of the classical PADMM. We establish iterate convergence as well as [Formula: see text] ergodic and [Formula: see text] nonergodic sublinear convergence rate results measured by function residual and feasibility violation, where [Formula: see text] denotes the iteration number. Moreover, we propose two fast variants of PADMC, one achieves faster [Formula: see text] ergodic convergence rate when one of the component functions is strongly convex, and the other ensures faster [Formula: see text] nonergodic convergence rate measured by constraint violation. Finally, preliminary numerical results on the LASSO and the elastic-net regularization problems are presented to demonstrate the performance of the proposed methods.

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

AbstractAlternating direction method of multipliers (ADMM) receives much attention in the field of optimization and computer science, etc. The generalized ADMM (G-ADMM) proposed by Eckstein and Bertsekas incorporates an acceleration factor and is more efficient than the original ADMM. However, G-ADMM is not applicable in some models where the objective function value (or its gradient) is computationally costly or even impossible to compute. In this paper, we consider the two-block separable convex optimization problem with linear constraints, where only noisy estimations of the gradient of the objective function are accessible. Under this setting, we propose a stochastic linearized generalized ADMM (called SLG-ADMM) where two subproblems are approximated by some linearization strategies. And in theory, we analyze the expected convergence rates and large deviation properties of SLG-ADMM. In particular, we show that the worst-case expected convergence rates of SLG-ADMM are $\mathcal{O}\left( {{N}^{-1/2}}\right)$ and $\mathcal{O}\left({\ln N} \cdot {N}^{-1}\right)$ for solving general convex and strongly convex problems, respectively, where N is the iteration number, similarly hereinafter, and with high probability, SLG-ADMM has $\mathcal{O}\left ( \ln N \cdot N^{-1/2} \right ) $ and $\mathcal{O}\left ( \left ( \ln N \right )^{2} \cdot N^{-1} \right ) $ constraint violation bounds and objective error bounds for general convex and strongly convex problems, respectively.

A Multi-Step Inertial Proximal Peaceman-Rachford Splitting Method for Separable Convex Programming

In this paper, we propose a multi-step inertial proximal Peaceman-Rachford splitting method (abbreviated as MIP-PRSM) for solving the two-block separable convex optimization problems with linear constraints, which is a unified framework for such Peaceman-Rachford splitting methods (PRSM)-based improved algorithms with inertial step. Furthermore, we establish the global convergence of the MIP-PRSM under some assumptions. Finally, some numerical experimental results on the least squares semidefinite programming (LSSP), LASSO, the convex quadratic programming problem (CQPP), total variation (TV) based denoising and medical image reconstruction problems demonstrate the efficiency of the proposed method.

An inexact ADMM with proximal-indefinite term and larger stepsize

In this paper, an inexact Alternating Direction Method of Multipliers (ADMM) has been proposed for solving the two-block separable convex optimization problem subject to linear equality constraints. The first resulting subproblem is solved inexactly under relative error criterion, while another subproblem called regularization problem is solved inexactly by introducing an indefinite proximal term. Meanwhile, the dual variable is updated twice with relatively larger stepsizes since an indefinite proximal term is exploited. By reformulating the first-order optimality conditions of the involved subproblems as a monotone inclusion problem, the sublinear convergence rate of the proposed algorithm is established in the pointwise and ergodic sense. Numerical experiments on testing two sparse optimization problems from statistical learning and image restoration indicate that our inexact ADMM performs much better than several well-established algorithms.

An inexact accelerated stochastic ADMM for separable convex optimization

An inexact accelerated stochastic Alternating Direction Method of Multipliers (AS-ADMM) scheme is developed for solving structured separable convex optimization problems with linear constraints. The objective function is the sum of a possibly nonsmooth convex function and a smooth function which is an average of many component convex functions. Problems having this structure often arise in machine learning and data mining applications. AS-ADMM combines the ideas of both ADMM and the stochastic gradient methods using variance reduction techniques. One of the ADMM subproblems employs a linearization technique while a similar linearization could be introduced for the other subproblem. For a specified choice of the algorithm parameters, it is shown that the objective error and the constraint violation are $$\mathcal {O}(1/k)$$ relative to the number of outer iterations k. Under a strong convexity assumption, the expected iterate error converges to zero linearly. A linearized variant of AS-ADMM and incremental sampling strategies are also discussed. Numerical experiments with both stochastic and deterministic ADMM algorithms show that AS-ADMM can be particularly effective for structured optimization arising in big data applications.

Linear Convergence of Prox-SVRG Method for Separable Non-smooth Convex Optimization Problems under Bounded Metric Subregularity

Linear Convergence of Prox-SVRG Method for Separable Non-smooth Convex Optimization Problems under Bounded Metric Subregularity

The Extension of Auxiliary Problem Principle for Solving Dynamic Economic Dispatch Problem in Power Systems

In this paper, we study the convex optimization problem with linear constraint, and the objective function is composed of m separable convex functions. Considering the special case where the objective function is composed of two separable convex functions, the auxiliary problem principle (APP) is an effective parallel distributed algorithm for solving the special case. Inspired by the principle of APP, a natural idea to solve separable convex optimization problem with m ≥ 3 is to extend the method of APP, resulting in the APP‐like algorithm. The convergence of the APP‐like algorithm is not clear yet. In this paper, we give a sufficient condition for the convergence of the APP‐like algorithm. Specifically, the APP algorithm is a special case of the APP‐like algorithm when m = 2. However, simulation results show that the convergence efficiency of the APP‐like algorithm is affected by the selection of penalty parameter. Therefore, we propose an improved APP‐like algorithm in this paper. Simulation results show that the improved APP‐like algorithm is robust to the selection of penalty parameter and that the convergence efficiency of the improved APP‐like algorithm is better when compared with the APP‐like algorithm.

A new accelerated positive-indefinite proximal ADMM for constrained separable convex optimization problems

A new accelerated positive-indefinite proximal ADMM for constrained separable convex optimization problems

Modified proximal symmetric ADMMs for multi-block separable convex optimization with linear constraints

We consider the linearly constrained separable convex optimization problem whose objective function is separable with respect to m blocks of variables. A bunch of methods have been proposed and extensively studied in the past decade. Specifically, a modified strictly contractive Peaceman–Rachford splitting method (SC-PRCM) [S. H. Jiang and M. Li, A modified strictly contractive Peaceman–Rachford splitting method for multi-block separable convex programming, J. Ind. Manag. Optim. 14(1) (2018) 397-412] has been well studied in the literature for the special case of m = 3. Based on the modified SC-PRCM, we present modified proximal symmetric ADMMs (MPSADMMs) to solve the multi-block problem. In MPSADMMs, all subproblems but the first one are attached with a simple proximal term, and the multipliers are updated twice. At the end of each iteration, the output is corrected via a simple correction step. Without stringent assumptions, we establish the global convergence result and the O(1/t) convergence rate in the ergodic sense for the new algorithms. Preliminary numerical results show that our proposed algorithms are effective for solving the linearly constrained quadratic programming and the robust principal component analysis problems.

A customized low-rank prior model for structured cartoon–texture image decomposition

The mathematical characterization of the texture component plays an instrumental role in image decomposition. In this paper, we are concerned with a low-rank texture prior based cartoon–texture image decomposition model, which utilizes a total variation norm and a global nuclear norm to characterize the cartoon and texture components, respectively. It is promising that our decomposition model is not only extremely simple, but also works perfectly for globally well-patterned images in the sense that the model can recover cleaner texture (or details) than the other novel models. Moreover, such a model can be easily reformulated as a separable convex optimization problem, thereby enjoying a splitting nature so that we can employ a partially parallel splitting method (PPSM) to solve it efficiently. A series of numerical experiments on image restoration demonstrate that PPSM can recover slightly higher quality images than some existing algorithms in terms of taking less iterations or computing time in many cases.

Convergence Rates of Inertial Primal-Dual Dynamical Methods for Separable Convex Optimization Problems

In this paper, we propose a second-order continuous primal-dual dynamical system with time-dependent positive damping terms for a separable convex optimization problem with linear equality constrai...

基于广义近似交替方向乘子法求解可分离凸优化问题

基于广义近似交替方向乘子法求解可分离凸优化问题

A Dynamical Approach to Two-Block Separable Convex Optimization Problems with Linear Constraints

The aim of this manuscript is to approach by means of first order differential equations/inclusions convex programming problems with two-block separable linear constraints and objectives, whereby (at least) one of the components of the latter is assumed to be strongly convex. Each block of the objective contains a further smooth convex function. We investigate the dynamical system proposed and prove that its trajectories converge weakly to a saddle point of the Lagrangian of the convex optimization problem. The dynamical system provides through time discretization the alternating minimization algorithm AMA and also its proximal variant recently introduced in the literature.

Convergence study on strictly contractive Peaceman–Rachford splitting method for nonseparable convex minimization models with quadratic coupling terms

The alternating direction method of multipliers (ADMM) and Peaceman Rachford splitting method (PRSM) are two popular splitting algorithms for solving large-scale separable convex optimization problems. Though problems with nonseparable structure appear frequently in practice, researches on splitting methods for these problems remain to be scarce. Very recently, Chen et al. (Math Program 173(1–2):37–77, 2019) extended the 2-block ADMM to linearly constrained nonseparable models with quadratic coupling terms and established its convergence. However, theoretical researches about nonseparable PRSM or its variants are still lacking. To fill the gap, in this paper we focus on the strictly contractive PRSM (SC-PRSM) applied to 2-block linearly constrained convex minimization problems with quadratic coupling objective functions. Under mild conditions, we prove the convergence of our proposed SC-PRSM and establish its o(1/k) convergence rate. Moreover, we implement the SC-PRSM to solve a problem of calculating the Euclidian distance between two ellipsoids, and compare its performance with three ADMM type algorithms. The results show the nonseparable SC-PRSM outperforms the other three algorithms in terms of both the iteration numbers and CPU time.

A Three-Operator Splitting Perspective of a Three-Block ADMM for Convex Quadratic Semidefinite Programming and Beyond

In recent years, several convergent variants of the multi-block alternating direction method of multipliers (ADMM) have been proposed for solving the convex quadratic semidefinite programming via its dual, which is inherently a [Formula: see text]-block separable convex optimization problem with coupled linear constraints. Among these multi-block ADMM-type algorithms, the modified [Formula: see text]-block ADMM in [Chang, XK, SY Liu and X Li (2016). Modified alternating direction method of multipliers for convex quadratic semidefinite programming. Neurocomputing, 214, 575–586] bears a peculiar feature that the augmented Lagrangian function is not necessarily to be minimized with respect to the block-variable corresponding to the quadratic term in the objective function. In this paper, we lay the theoretical foundation of this phenomenon by interpreting this modified [Formula: see text]-block ADMM as a special implementation of the Davis–Yin [Formula: see text]-operator splitting [Davis, D and WT Yin (2017). A three-operator splitting scheme and its optimization applications. Set-Valued and Variational Analysis, 25, 829–858]. Based on this perspective, we are able to extend this modified [Formula: see text]-block ADMM to a generalized [Formula: see text]-block ADMM, in the sense of [Eckstein, J and DP Bertsekas (1992). On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming, 55, 293–318], which not only applies to the more general convex composite quadratic programming problems but also admits the flexibility of achieving even better numerical performance.

A parallel Gauss-Seidel method for convex problems with separable structure

<p style='text-indent:20px;'>The convergence of direct ADMM is not guaranteed when used to solve multi-block separable convex optimization problems. In this paper, we propose a Gauss-Seidel method which can be calculated in parallel while solving subproblems. First we divide the variables into different groups. In the inner group, we use Gauss-Seidel method solving the subproblem. Among the different groups, Jacobi-like method is used. The effectiveness of the algorithm is proved by some numerical experiments.

Two New Customized Proximal Point Algorithms Without Relaxation for Linearly Constrained Convex Optimization

In this paper, we propose a new customized proximal point algorithm for linearly constrained convex optimization problem, and further extend the proposed method to separable convex optimization problem. Unlike the existing customized proximal point algorithms, the proposed algorithms do not involve relaxation step, but still ensure the convergence. We obtain the particular iteration schemes and the unified variational inequality perspective. The global convergence and $${\mathcal {O}}(1/k)$$-convergence rate of the proposed methods are investigated under some mild assumptions. Numerical experiments show that compared to some state-of-the-art methods, the proposed methods are effective.

An indefinite proximal Peaceman–Rachford splitting method with substitution procedure for convex programming

The strictly contractive Peaceman–Rachford splitting method (SCPRSM) has received a tremendous amount of attention for solving linearly constrained separable convex optimization problems. In this paper, we propose an indefinite proximal SCPRSM with substitution procedure (abbreviated as PPRSM-S) to improve numerical results. The prediction step takes a proximal SCPRSM cycle to update the variable blocks, then the correction step corrects the output slightly by computing a combination of the prediction step and the previous iteration. We derive the global convergence of the proposed method and analyze the convergence rate results under much mild conditions. Some experimental results on LASSO and total variation-based denoising problems demonstrate the efficiency of the substitution step and the indefinite proximal term.

On the Distributed Method of Multipliers for Separable Convex Optimization Problems

In this paper, we present a novel method for convex optimization in distributed networks called the distributed method of multipliers (DMM). The proposed method is based on a combination of a particular dual lifting and classic monotone operator splitting approaches to produce an algorithm with guaranteed asymptotic convergence in undirected networks. The proposed method allows any separable convex problem with linear constraints to be solved in undirected networks. In contrast to typical distributed approaches, the structure of the network does not restrict the types of problems that can be solved. Furthermore, the solver can be applied to general separable problems, those with separable convex objectives and constraints, via the use of an additional primal lifting approach. Finally, we demonstrate the use of DMM in solving a number of classic signal processing problems including beamforming, channel capacity maximization and portfolio optimization.

An LQP-Based Symmetric Alternating Direction Method of Multipliers with Larger Step Sizes

Symmetric alternating direction method of multipliers (ADMM) is an efficient method for solving a class of separable convex optimization problems. This method updates the Lagrange multiplier twice with appropriate step sizes at each iteration. However, such step sizes were conservatively shrunk to guarantee the convergence in recent studies. In this paper, we are devoted to seeking larger step sizes whenever possible. The logarithmic-quadratic proximal (LQP) terms are applied to regularize the symmetric ADMM subproblems, allowing the constrained subproblems to then be converted to easier unconstrained ones. Theoretically, we prove the global convergence of such LQP-based symmetric ADMM by specifying a larger step size domain. Moreover, the numerical results on a traffic equilibrium problem are reported to demonstrate the advantage of the method with larger step sizes.