Separable Convex Optimization Research Articles

Alternating Direction Method of Multipliers for Separable Convex Optimization of Real Functions in Complex Variables

The alternating direction method of multipliers (ADMM) has been widely explored due to its broad applications, and its convergence has been gotten in the real field. In this paper, an ADMM is presented for separable convex optimization of real functions in complex variables. First, the convergence of the proposed method in the complex domain is established by using the Wirtinger Calculus technique. Second, the basis pursuit (BP) algorithm is given in the form of ADMM in which the projection algorithm and the soft thresholding formula are generalized from the real case. The numerical simulations on the reconstruction of electroencephalogram (EEG) signal are provided to show that our new ADMM has better behavior than the classic ADMM for solving separable convex optimization of real functions in complex variables.

Convex Separable Problems With Linear Constraints in Signal Processing and Communications

In this work, we focus on separable convex optimization problems with box constraints and a set of triangular linear constraints. The solution is given in closed-form as a function of some Lagrange multipliers that can be computed through an iterative procedure in a finite number of steps. Graphical interpretations are given casting valuable insights into the proposed algorithm and allowing to retain some of the intuition spelled out by the water-filling policy. It turns out that it is not only general enough to compute the solution to different instances of the problem at hand but also remarkably simple in the way it operates. We also show how some power allocation problems in signal processing and communications can be solved with the proposed algorithm.

The incremental subgradient methods on distributed estimations in-network

A unified framework on distributed estimations in-network using incremental subgradient (IS) methods is introduced. The IS methods meet the requirement of the asynchronously processing, which are efficient solvers developed recently focusing on separable non-differentiable convex optimization problems. The main contribution of this paper is to formalize distributed estimators as equivalent separable convex optimization problems, where general skills and several specific cases on signal estimations are presented. Analytical and simulation results show that the IS framework can solve general nonlinear estimation problems in-network, and achieve comparable performances as the centralized estimators.

A Line-Search-Based Partial Proximal Alternating Directions Method for Separable Convex Optimization

We propose an appealing line-search-based partial proximal alternating directions (LSPPAD) method for solving a class of separable convex optimization problems. These problems under consideration are common in practice. The proposed method solves two subproblems at each iteration: one is solved by a proximal point method, while the proximal term is absent from the other. Both subproblems admit inexact solutions. A line search technique is used to guarantee the convergence. The convergence of the LSPPAD method is established under some suitable conditions. The advantage of the proposed method is that it provides the tractability of the subproblem in which the proximal term is absent. Numerical tests show that the LSPPAD method has better performance compared with the existing alternating projection based prediction-correction (APBPC) method if both are employed to solve the described problem.

Path-following gradient-based decomposition algorithms for separable convex optimization

A new decomposition optimization algorithm, called path-following gradient-based decomposition, is proposed to solve separable convex optimization problems. Unlike path-following Newton methods considered in the literature, this algorithm does not require any smoothness assumption on the objective function. This allows us to handle more general classes of problems arising in many real applications than in the path-following Newton methods. The new algorithm is a combination of three techniques, namely smoothing, Lagrangian decomposition and path-following gradient framework. The algorithm decomposes the original problem into smaller subproblems by using dual decomposition and smoothing via self-concordant barriers, updates the dual variables using a path-following gradient method and allows one to solve the subproblems in parallel. Moreover, compared to augmented Lagrangian approaches, our algorithmic parameters are updated automatically without any tuning strategy. We prove the global convergence of the new algorithm and analyze its convergence rate. Then, we modify the proposed algorithm by applying Nesterov's accelerating scheme to get a new variant which has a better convergence rate than the first algorithm. Finally, we present preliminary numerical tests that confirm the theoretical development.

Graver basis and proximity techniques for block-structured separable convex integer minimization problems

We consider N-fold 4-block decomposable integer programs, which simultaneously generalize N-fold integer programs and two-stage stochastic integer programs with N scenarios. In previous work [R. Hemmecke, M. Koeppe, R. Weismantel, A polynomial-time algorithm for optimizing over N-fold 4-block decomposable integer programs, Proc. IPCO 2010, Lecture Notes in Computer Science, vol. 6080, Springer, 2010, pp. 219--229], it was proved that for fixed blocks but variable N, these integer programs are polynomial-time solvable for any linear objective. We extend this result to the minimization of separable convex objective functions. Our algorithm combines Graver basis techniques with a proximity result [D.S. Hochbaum and J.G. Shanthikumar, Convex separable optimization is not much harder than linear optimization, J. ACM 37 (1990), 843--862], which allows us to use convex continuous optimization as a subroutine.

Well-Posedness and Primal-Dual Analysis of Some Convex Separable Optimization Problems

We focus on some convex separable optimization problems, considered by the author in previous papers, for which problems, necessary and sufficient conditions or sufficient conditions have been proved, and convergent algorithms of polynomial computational complexity have been proposed for solving these problems. The concepts of well-posedness of optimization problems in the sense of Tychonov, Hadamard, and in a generalized sense, as well as calmness in the sense of Clarke, are discussed. It is shown that the convex separable optimization problems under consideration are calm in the sense of Clarke. The concept of stability of the set of saddle points of the Lagrangian in the sense of Gol'shtein is also discussed, and it is shown that this set is not stable for the “classical” Lagrangian. However, it turns out that despite this instability, due to the specificity of the approach, suggested by the author for solving problems under consideration, it is not necessary to use modified Lagrangians but only the “classical” Lagrangians. Also, a primal-dual analysis for problems under consideration in view of methods for solving them is presented.

Combining Lagrangian decomposition and excessive gap smoothing technique for solving large-scale separable convex optimization problems

A new algorithm for solving large-scale convex optimization problems with a separable objective function is proposed. The basic idea is to combine three techniques: Lagrangian dual decomposition, excessive gap and smoothing. The main advantage of this algorithm is that it automatically and simultaneously updates the smoothness parameters which significantly improves its performance. The convergence of the algorithm is proved under weak conditions imposed on the original problem. The rate of convergence is \(O(\frac {1}{k})\), where k is the iteration counter. In the second part of the paper, the proposed algorithm is coupled with a dual scheme to construct a switching variant in a dual decomposition framework. We discuss implementation issues and make a theoretical comparison. Numerical examples confirm the theoretical results.

A block coordinate gradient descent method for regularized convex separable optimization and covariance selection

We consider a class of unconstrained nonsmooth convex optimization problems, in which the objective function is the sum of a convex smooth function on an open subset of matrices and a separable convex function on a set of matrices. This problem includes the covariance selection problem that can be expressed as an l 1-penalized maximum likelihood estimation problem. In this paper, we propose a block coordinate gradient descent method (abbreviated as BCGD) for solving this class of nonsmooth separable problems with the coordinate block chosen by a Gauss-Seidel rule. The method is simple, highly parallelizable, and suited for large-scale problems. We establish global convergence and, under a local Lipschizian error bound assumption, linear rate of convergence for this method. For the covariance selection problem, the method can terminate in $${O(n^3/\epsilon)}$$ iterations with an $${\epsilon}$$ -optimal solution. We compare the performance of the BCGD method with the first-order methods proposed by Lu (SIAM J Optim 19:1807–1827, 2009; SIAM J Matrix Anal Appl 31:2000–2016, 2010) for solving the covariance selection problem on randomly generated instances. Our numerical experience suggests that the BCGD method can be efficient for large-scale covariance selection problems with constraints.

A primal-dual consensus algorithm for convex problems from network control and estimation

AbstractIn this paper we propose a distributed algorithm for solving separable convex optimization problems on graphs arising for example from estimation and control in networks. We derive a primal-dual decomposition algorithm for solving this type of separable optimization problems in a distributed fashion given restrictions on the communication topology. The proposed algorithm is based on consensus principles in combination with local subgradient updates for the primal-dual variables. The main novelty of our algorithm consists in the way the weights in the consensus process are updated using arguments from optimization theory, while in most of the existing distributed algorithms based on consensus principles the weights have to be tuned.

Power minimization for CDMA under colored noise

Rate-constrained power minimization (PMIN) over a code division multiple-access (CDMA) channel with correlated noise is studied. PMIN is shown to be an instance of a separable convex optimization problem subject to linear ascending constraints. PMIN is further reduced to a dual problem of sumrate maximization (RMAX). The results highlight the underlying unity between PMIN, RMAX, and a problem closely related to PMIN but with linear receiver constraints. Subsequently, conceptually simple sequence design algorithms are proposed to explicitly identify an assignment of sequences and powers that solve PMIN. The algorithms yield an upper bound of 2N - 1 on the number of distinct sequences where N is the processing gain. The sequences generated using the proposed algorithms are in general real-valued. If a rate-splitting and multi-dimensional CDMA approach is allowed, the upper bound reduces to N distinct sequences, in which case the sequences can form an orthogonal set and be binary plusmn1-valued.

Optimization driven bandwidth provisioning in service overlay networks

We consider the bandwidth provisioning problem in service overlay networks (SON). We formulate the bandwidth provisioning problem in an optimization framework taking the stochastic processes involved into consideration. First, we provide the optimal solution that maximizes the expected net revenue in a single-link network under stationary traffic demand. Then, we show that the revenue maximization problem in a general topology under stationary traffic can be nicely approximated by a separable convex optimization problem. We derive an approximate optimal solution for a general topology and provide a bandwidth provisioning algorithm via a gradient method, which works in a distributed manner. We also verify the effectiveness of the approximate optimal solution by showing several important characteristics of the solution. The performance of the algorithm is investigated through extensive simulations.

Separable Convex Optimization Problems with Linear Ascending Constraints

Separable convex optimization problems with linear ascending inequality and equality constraints are addressed in this paper. An algorithm that explicitly characterizes the optimum point in a finite number of steps is described. The optimum value is shown to be monotone with respect to a partial order on the constraint parameters. Moreover, the optimum value is convex with respect to these parameters. This work generalizes the existing algorithms of Morton, von Randow, and Ringwald [Math. Programming, 32 (1985), pp. 238–241] and Viswanath and Anantharam [IEEE Trans. Inform. Theory, 48 (2002), pp. 1295–1318] to a wider class of separable convex objective functions. Computational experiments that compare the proposed algorithm with a standard convex optimization tool are also provided.

Iterative bundle-based decomposition for large-scale nonseparable convex optimization

Abstract There has been considerable research in solving large-scale separable convex optimization problems. In this paper we present an algorithm for large-scale nonseparable smooth convex optimization problems with block-angular linear constraints. The solution of the problem is approximated by solving a sequence of structured separable quadratic programs. The Bundle-based decomposition (BBD) method of Robinson (In: Prekopa, A., Szelezsan, J., Strazicky, B. (Eds.), System Modelling and Optimization, Springer, 1986, pp. 751–756; Annals de Institute Henri Poincare: Analyse Non Lineaire 6 (1989) 435–447) is applied to each separable quadratic program. We implement the algorithm and present computational experience.

Polynomial Methods for Separable Convex Optimization in Unimodular Linear Spaces with Applications

We consider the problem of minimizing a separable convex objective function over the linear space given by a system Mx=0 with M a totally unimodular matrix. In particular, this generalizes the usual minimum linear cost circulation and cocirculation problems in a network and the problems of determining the Euclidean distance from a point to the perfect bipartite matching polytope and the feasible flows polyhedron. We first show that the idea of minimum mean cycle canceling originally worked out for linear cost circulations by Goldberg and Tarjan [J. Assoc. Comput. Mach., 36 (1989), pp. 873--886.] and extended to some other problems [T. R. Ervolina and S. T. McCormick, Discrete Appl. Math., 46 (1993), pp. 133--165], [A. Frank and A. V. Karzanov, Technical Report RR 895-M, Laboratoire ARTEMIS IMAG, Université Joseph Fourier, Grenoble, France, 1992], [T. Ibaraki, A. V. Karzanov, and H. Nagamochi, private communication, 1993], [M. Hadjiat, Technical Report, Groupe Intelligence Artificielle, Faculté des Sciences de Luminy, Marseille, France, 1994] can be generalized to give a combinatorial method with geometric convergence for our problem. We also generalize the computationally more efficient cancel-and-tighten method. We then consider objective functions that are piecewise linear, pure and piecewise quadratic, or piecewise mixed linear and quadratic, and we show how both methods can be implemented to find exact solutions in polynomial time (strongly polynomial in the piecewise linear case). These implementations are then further specialized for finding circulations and cocirculations in a network. We finish by showing how to extend our methods to find optimal integer solutions, to linear spaces of larger fractionality, and to the case when the objective functions are given by approximate oracles.

A Min--Max Theorem for Bisubmodular Polyhedra

For a family ${\cal F} \subseteq 3^E$ closed with respect to the reduced union and intersection and for a bisubmodular function $f: {\cal F}\rightarrow \mbox{\bf R}$ with $(\emptyset,\emptyset) \in \cal F$ and $f(\emptyset,\emptyset)=0$, the bisubmodular polyhedron associated with $({\cal F},f)$ is given by \[ {\rm P}_*(f)=\{x\,|\,x\in\mbox{\bf R}^E\quad \forall (X,Y)\in {\cal F}: x(X,Y)\le f(X,Y)\}, \] where $x(X,Y)=\sum_{e\in X}x(e)-\sum_{e\in Y}x(e)$. We show a min--max relation that characterizes the distance between ${\rm P}_*(f)$ and a given point $x^0$ with respect to the $l_1$ norm: for any vector $x^0\in\mbox{\bf R}^E$, \[ \left. \min\left\{\sum_{e\in E}|x(e)-x^0(e)|\, \right| \,x\in{\rm P}_*(f)\right\} =\max\{x^0(X,Y)-f(X,Y)\,|\,(X,Y)\in{\cal F}\}, \] where if f is integer valued and $x^0$ is integral, then the minimum is attained by an integral $x\in{\rm P}_*(f)$. This is in a sense equivalent to but is in a nicer symmetric form than a min--max theorem of Cunningham and Green-Krótki [Combinatorica, 11 (1991), pp. 219--230] shown to be associated with b-matching degree-sequence polyhedra and generalizes the well-known min--max theorem concerning a vector reduction of polymatroids and submodular systems. We also give an application of the theorem to a separable convex optimization problem on bisubmodular polyhedra.

A unified approach to polynomially solvable cases of integer “non-separable” quadratic optimization

A recent paper by Hochbaum and Shanthikumar presented “a general-purpose algorithm for converting procedures that solve linear programming problems with … integer variables, to procedures for solving … separable [non-linear] problems”. 1 1 [18, abstract]. Their work showed that “convex separable optimization is not much harder than linear optimization”. In contrast, polynomial algorithms in the literature for “non-separable” integer quadratic problems use qualitatively different techniques. By linearly transforming these problems so that the objective is separable in the transformed reference frame, we provide alternative algorithms for these problems based on Hochbaum and Shanthikumar's algorithms. Inter alia we introduce a new class of polynomially solvable integer quadratic optimization problems. We also show that a slight generalization of integer linear programming having a non-separable, non-linear objective and totally unimodular constraints in NP-hard.

Convex separable optimization is not much harder than linear optimization

The polynomiality of nonlinear separable convex (concave) optimization problems, on linear constraints with a matrix with “small” subdeterminants, and the polynomiality of such integer problems, provided the inteter linear version of such problems ins polynomial, is proven. This paper presents a general-purpose algorithm for converting procedures that solves linear programming problems. The conversion is polynomial for constraint matrices with polynomially bounded subdeterminants. Among the important corollaries of the algorithm is the extension of the polynomial solvability of integer linear programming problems with totally unimodular constraint matrix, to integer-separable convex programming. An algorithm for finding a ε-accurate optimal continuous solution to the nonlinear problem that is polynomial in log(1/ε) and the input size and the largest subdeterminant of the constraint matrix is also presented. These developments are based on proximity results between the continuous and integral optimal solutions for problems with any nonlinear separable convex objective function. The practical feature of our algorithm is that is does not demand an explicit representation of the nonlinear function, only a polynomial number of function evaluations on a prespecified grid.

A comparative study of primal and dual approaches for solving separable and partially-separable nonlinear optimization problems

In nonlinear optimization, the dual problem is in general not easier to solve than the primal problem. Convex separable optimization problems, frequently arising in electrical and mechanical engineering, constitute a notable exception to the above rule. The dual problem is to optimize the dual objective functionl over a non-negative orthant, and the evaluation ofl reduces to the execution of independentlinear searches only. To generalize the idea, we also consider partially-separable problems with objective and constraint functions such that the Hessian matrix of the Lagrange function is a block-diagonal matrix with 2*2 blocks. The evaluation of the dual objective function is accordingly reduced to a number of independentplanar searches. Obviously, 3*3 blocks would lead tospatial searches, etc. We compare the performance of a primal and a dual method on a graded set of artificial test problems with increasing size, increasing degree of degeneracy, and increasing ill-conditioning. The observed speed-up by the dual approach varies between 2 and 30. Finally, we consider the potential of the dual approach for execution on parallel computers.

A new technique for nonconvex primal-dual decomposition of a large-scale separable optimization problem

The primal-dual approach is quite effective in decomposing a convex separable optimization problem into several subproblems of smaller sizes. In this paper, we present a new technique which extends the primal-dual approach to nonconvex problems. Since a straightforward application of the multiplier method destroys separability, a new Lagrangian function is proposed which preserves separability. Based on this new function we develop a new iterative method for finding an optimal solution to the problem and show that the method is locally convergent to an optimal solution. Furthermore, the effect of certain parameters on the ratio of convergence is investigated and simple examples are given to illustrate the proposed approach.