Convex Quadratic Semidefinite Programming Research Articles

A Three-Operator Splitting Perspective of a Three-Block ADMM for Convex Quadratic Semidefinite Programming and Beyond

In recent years, several convergent variants of the multi-block alternating direction method of multipliers (ADMM) have been proposed for solving the convex quadratic semidefinite programming via its dual, which is inherently a [Formula: see text]-block separable convex optimization problem with coupled linear constraints. Among these multi-block ADMM-type algorithms, the modified [Formula: see text]-block ADMM in [Chang, XK, SY Liu and X Li (2016). Modified alternating direction method of multipliers for convex quadratic semidefinite programming. Neurocomputing, 214, 575–586] bears a peculiar feature that the augmented Lagrangian function is not necessarily to be minimized with respect to the block-variable corresponding to the quadratic term in the objective function. In this paper, we lay the theoretical foundation of this phenomenon by interpreting this modified [Formula: see text]-block ADMM as a special implementation of the Davis–Yin [Formula: see text]-operator splitting [Davis, D and WT Yin (2017). A three-operator splitting scheme and its optimization applications. Set-Valued and Variational Analysis, 25, 829–858]. Based on this perspective, we are able to extend this modified [Formula: see text]-block ADMM to a generalized [Formula: see text]-block ADMM, in the sense of [Eckstein, J and DP Bertsekas (1992). On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming, 55, 293–318], which not only applies to the more general convex composite quadratic programming problems but also admits the flexibility of achieving even better numerical performance.

On the equivalence of inexact proximal ALM and ADMM for a class of convex composite programming

In this paper, we show that for a class of linearly constrained convex composite optimization problems, an (inexact) symmetric Gauss–Seidel based majorized multi-block proximal alternating direction method of multipliers (ADMM) is equivalent to an inexact proximal augmented Lagrangian method. This equivalence not only provides new perspectives for understanding some ADMM-type algorithms but also supplies meaningful guidelines on implementing them to achieve better computational efficiency. Even for the two-block case, a by-product of this equivalence is the convergence of the whole sequence generated by the classic ADMM with a step-length that exceeds the conventional upper bound of $$(1+\sqrt{5})/2$$ , if one part of the objective is linear. This is exactly the problem setting in which the very first convergence analysis of ADMM was conducted by Gabay and Mercier (Comput Math Appl 2(1):17–40, 1976), but, even under notably stronger assumptions, only the convergence of the primal sequence was known. A collection of illustrative examples are provided to demonstrate the breadth of applications for which our results can be used. Numerical experiments on solving a large number of linear and convex quadratic semidefinite programming problems are conducted to illustrate how the theoretical results established here can lead to improvements on the corresponding practical implementations.

On the R-superlinear convergence of the KKT residuals generated by the augmented Lagrangian method for convex composite conic programming

Due to the possible lack of primal-dual-type error bounds, it was not clear whether the Karush–Kuhn–Tucker (KKT) residuals of the sequence generated by the augmented Lagrangian method (ALM) for solving convex composite conic programming (CCCP) problems converge superlinearly. In this paper, we resolve this issue by establishing the R-superlinear convergence of the KKT residuals generated by the ALM under only a mild quadratic growth condition on the dual of CCCP, with easy-to-implement stopping criteria for the augmented Lagrangian subproblems. This discovery may help to explain the good numerical performance of several recently developed semismooth Newton-CG based ALM solvers for linear and convex quadratic semidefinite programming.

QSDPNAL: a two-phase augmented Lagrangian method for convex quadratic semidefinite programming

In this paper, we present a two-phase augmented Lagrangian method, called QSDPNAL, for solving convex quadratic semidefinite programming (QSDP) problems with constraints consisting of a large number of linear equality and inequality constraints, a simple convex polyhedral set constraint, and a positive semidefinite cone constraint. A first order algorithm which relies on the inexact Schur complement based decomposition technique is developed in QSDPNAL-Phase I with the aim of solving a QSDP problem to moderate accuracy or using it to generate a reasonably good initial point for the second phase. In QSDPNAL-Phase II, we design an augmented Lagrangian method (ALM) wherein the inner subproblem in each iteration is solved via inexact semismooth Newton based algorithms. Simple and implementable stopping criteria are designed for the ALM. Moreover, under mild conditions, we are able to establish the rate of convergence of the proposed algorithm and prove the R-(super)linear convergence of the KKT residual. In the implementation of QSDPNAL, we also develop efficient techniques for solving large scale linear systems of equations under certain subspace constraints. More specifically, simpler and yet better conditioned linear systems are carefully designed to replace the original linear systems and novel shadow sequences are constructed to alleviate the numerical difficulties brought about by the crucial subspace constraints. Extensive numerical results for various large scale QSDPs show that our two-phase algorithm is highly efficient and robust in obtaining accurate solutions. The software reviewed as part of this submission was given the DOI (Digital Object Identifier) https://doi.org/10.5281/zenodo.1206980 .

Modified alternating direction method of multipliers for convex quadratic semidefinite programming

The direct extension of alternating direction method of multipliers (ADMM) is not necessarily convergent to the dual form of convex quadratic semidefinite programming (CQSDP) problems, though the convergence of ADMM is proved when two blocks of variables are alternatively updated. In this paper, we present a modified ADMM for solving the dual form of the CQSDP problems. At each iteration, the proposed method did not have to work out the sub-problem with the primal variable, compared with the existing ADMM. Assuming that the penalty parameter satisfies the condition related to the quadratic term of the primal objective function, the convergence is proved by using a fixed-point argument. Numerical results for the nearest correlation matrix problems and the random CQSDP problems demonstrate the efficiency of our proposed algorithm.

A Schur complement based semi-proximal ADMM for convex quadratic conic programming and extensions

This paper is devoted to the design of an efficient and convergent {semi-proximal} alternating direction method of multipliers (ADMM) for finding a solution of low to medium accuracy to convex quadratic conic programming and related problems. For this class of problems, the convergent two block semi-proximal ADMM can be employed to solve their primal form in a straightforward way. However, it is known that it is more efficient to apply the directly extended multi-block semi-proximal ADMM, though its convergence is not guaranteed, to the dual form of these problems. Naturally, one may ask the following question: can one construct a convergent multi-block semi-proximal ADMM that is more efficient than the directly extended semi-proximal ADMM? Indeed, for linear conic programming with 4-block constraints this has been shown to be achievable in a recent paper by Sun, Toh and Yang [arXiv preprint arXiv:1404.5378, (2014)]. Inspired by the aforementioned work and with the convex quadratic conic programming in mind, we propose a Schur complement based convergent semi-proximal ADMM for solving convex programming problems, with a coupling linear equality constraint, whose objective function is the sum of two proper closed convex functions plus an arbitrary number of convex quadratic or linear functions. Our convergent semi-proximal ADMM is particularly suitable for solving convex quadratic semidefinite programming (QSDP) with constraints consisting of linear equalities, a positive semidefinite cone and a simple convex polyhedral set. The efficiency of our proposed algorithm is demonstrated by numerical experiments on various examples including QSDP.

Positive Semidefinite Generalized Diffusion Tensor Imaging via Quadratic Semidefinite Programming

The positive definiteness of a diffusion tensor is important in magnetic resonance imaging because it reflects the phenomenon of water molecular diffusion in complicated biological tissue environments. To preserve this property, we represent it as an explicit positive semidefinite (PSD) matrix constraint and some linear matrix equalities. The objective function is the regularized linear least squares fitting for the log-linearized Stejskal--Tanner equation. The regularization term is the heuristic nuclear norm of the PSD matrix, since we expect it to be of low rank. In this way, we establish a convex quadratic semidefinite programming (SDP) model, whose global solution exists. The optimal solution could be solved by three efficient methods. While there are two state-of-the-art solvers---SDPT3 and QSDP---for the primal problem, we design a new augmented Lagrangian based alternating direction method (ADM) for the dual problem. Sensitivity analyses on the coefficients of the optimal diffusion tensor and the ...

A convex quadratic semi-definite programming approach to the partial additive constant problem in multidimensional scaling

Bénasséni [Partial additive constant, J. Statist. Comput. Simul. 49 (1994), pp. 179–193] studied the partial additive constant problem in multidimensional scaling. This problem is quite challenging to solve, and Bénasséni proposed a numerical procedure for two special cases: the cross-set partial perturbation and the within-set partial perturbation. This paper casts the problem as a modern quadratic semi-definite programming (QSDP) problem, which is not only capable of dealing with general cases, but also enjoys a number of good properties. One of the good properties is that the proposed approach can find the minimal constant under very weak conditions. Another is that there exists a ready-to-use numerical package such as the QSDP solver in Toh [An inexact path-following algorithm for convex quadratic SDP, Math. Program. 112 (2008), pp. 221–254], allowing a great deal of flexibility in choosing the index set to which the partial constant should be added. Our numerical results show a significant improvement over that reported in Bénasséni (1994).

An Inexact Accelerated Proximal Gradient Method for Large Scale Linearly Constrained Convex SDP

The accelerated proximal gradient (APG) method, first proposed by Nesterov for minimizing smooth convex functions, later extended by Beck and Teboulle to composite convex objective functions, and studied in a unifying manner by Tseng, has proven to be highly efficient in solving some classes of large scale structured convex optimization (possibly nonsmooth) problems, including nuclear norm minimization problems in matrix completion and $l_1$ minimization problems in compressed sensing. The method has superior worst-case iteration complexity over the classical projected gradient method and usually has good practical performance on problems with appropriate structures. In this paper, we extend the APG method to the inexact setting, where the subproblem in each iteration is solved only approximately, and show that it enjoys the same worst-case iteration complexity as the exact counterpart if the subproblems are progressively solved to sufficient accuracy. We apply our inexact APG method to solve large scale convex quadratic semidefinite programming (QSDP) problems of the form $\min\{ \frac{1}{2}\langle x,\,\mathcal{Q}(x)\rangle + \langle c,\,x\rangle\mid \mathcal{A}(x) = b, x\succeq 0\}$, where $\mathcal{Q},\mathcal{A}$ are given linear maps and $b,c$ are given data. The subproblem in each iteration is solved by a semismooth Newton-CG (SSNCG) method with warm-start using the iterate from the previous iteration. Our APG-SSNCG method is demonstrated to be efficient for QSDP problems whose positive semidefinite linear maps $\mathcal{Q}$ are highly ill-conditioned or rank deficient.

An inexact spectral bundle method for convex quadratic semidefinite programming

We present an inexact spectral bundle method for solving convex quadratic semidefinite optimization problems. This method is a first-order method, hence requires much less computational cost in each iteration than second-order approaches such as interior-point methods. In each iteration of our method, we solve an eigenvalue minimization problem inexactly, and solve a small convex quadratic semidefinite program as a subproblem. We give a proof of the global convergence of this method using techniques from the analysis of the standard bundle method, and provide a global error bound under a Slater type condition for the problem in question. Numerical experiments with matrices of order up to 3000 are performed, and the computational results establish the effectiveness of this method.

Biquadratic Optimization Over Unit Spheres and Semidefinite Programming Relaxations

This paper studies the so-called biquadratic optimization over unit spheres $\min_{x\in \mathbb{R}^n,y\in \mathbb{R}^m}\sum_{1\leq i,k\leq n,\,1\leq j,l \leq m}b_{ijkl}x_{i}y_{j}x_{k}{y}_l$, subject to $\|x\| = 1$, $\|y\| =1$. We show that this problem is NP-hard, and there is no polynomial time algorithm returning a positive relative approximation bound. Then, we present various approximation methods based on semidefinite programming (SDP) relaxations. Our theoretical results are as follows: For general biquadratic forms, we develop a $\frac{1}{2\max\{m,n\}^2}$-approximation algorithm under a slightly weaker approximation notion; for biquadratic forms that are square-free, we give a relative approximation bound $\frac{1}{nm}$; when $\min\{n,m\}$ is a constant, we present two polynomial time approximation schemes (PTASs) which are based on sum of squares (SOS) relaxation hierarchy and grid sampling of the standard simplex. For practical computational purposes, we propose the first order SOS relaxation, a convex quadratic SDP relaxation, and a simple minimum eigenvalue method and show their error bounds. Some illustrative numerical examples are also included.

Estimation of Positive Semidefinite Correlation Matrices by Using Convex Quadratic Semidefinite Programming

The correlation matrix is a fundamental statistic that is used in many fields. For example, GroupLens, a collaborative filtering system, uses the correlation between users for predictive purposes. Since the correlation is a natural similarity measure between users, the correlation matrix may be used in the Gram matrix in kernel methods. However, the estimated correlation matrix sometimes has a serious defect: although the correlation matrix is originally positive semidefinite, the estimated one may not be positive semidefinite when not all ratings are observed. To obtain a positive semidefinite correlation matrix, the nearest correlation matrix problem has recently been studied in the fields of numerical analysis and optimization. However, statistical properties are not explicitly used in such studies. To obtain a positive semidefinite correlation matrix, we assume the approximate model. By using the model, an estimate is obtained as the optimal point of an optimization problem formulated with information on the variances of the estimated correlation coefficients. The problem is solved by a convex quadratic semidefinite program. A penalized likelihood approach is also examined. The MovieLens data set is used to test our approach.

Local Duality of Nonlinear Semidefinite Programming

Recently, Chan and Sun [Chan, Z. X., D. Sun. Constraint nondegeneracy, strong regularity and nonsingularity in semidefinite programming. SIAM J. Optim. 19 370–376.] reported for semidefinite programming (SDP) that the primal/dual constraint nondegeneracy is equivalent to the dual/primal strong second order sufficient condition (SSOSC). This result is responsible for a number of important results in stability analysis of SDP. In this paper, we study duality of this type in nonlinear semidefinite programming (NSDP). We introduce the dual SSOSC at a Karush-Kuhn-Tucker (KKT) triple of NSDP and study its various characterizations and relationships to the primal nondegeneracy. Although the dual SSOSC is nothing but the SSOSC for the Wolfe dual of the NSDP, it suggests new information for the primal NSDP. For example, it ensures that the inverse of the Hessian of the Lagrangian function exists at the KKT triple and the inverse is positive definite on some normal space. It also ensures the primal nondegeneracy. Some of our results generalize the corresponding classical duality results in nonlinear programming studied by Fujiwara et al. [Fujiwara, O., S.-P. Han, O. L. Mangasarian. 1984. Local duality of nonlinear programs. SIAM J. Control Optim. 22 162–169]. For the convex quadratic SDP (QSDP), we have complete characterizations for the primal and dual SSOSC. Our results reveal that the nearest correlation matrix problem satisfies not only the primal and dual SSOSC but also the primal and dual nondegeneracy at its solution, suggesting that it is a well-conditioned QSDP.