Composite Convex Research Articles

Stochastic Gradient Made Stable: A Manifold Propagation Approach for Large-Scale Optimization

Stochastic gradient descent (SGD) holds as a classical method to build large scale machine learning models over big data. A stochastic gradient is typically calculated from a limited number of samples (known as mini-batch), which potentially incurs a high variance and causes the estimated parameters to bounce around the optimal solution. To improve the stability of stochastic gradient, recent years have witnessed the proposal of several semi-stochastic gradient descent algorithms, which distinguish themselves from standard SGD by incorporating global information into gradient computation. In this paper, we contribute a novel stratified semi-stochastic gradient descent (S3GD) algorithm to this nascent research area, accelerating the optimization of a large family of composite convex functions. Though theoretically converging faster, prior semi-stochastic algorithms are found to suffer from high iteration complexity, which makes them even slower than SGD in practice on many datasets. In our proposed S3GD, the semi-stochastic gradient is calculated based on efficient manifold propagation, which can be numerically accomplished by sparse matrix multiplications. This way S3GD is able to generate a highly-accurate estimate of the exact gradient from each mini-batch with largely-reduced computational complexity. Theoretic analysis reveals that the proposed S3GD elegantly balances the geometric algorithmic convergence rate against the space and time complexities during the optimization. The efficacy of S3GD is also experimentally corroborated on several large-scale benchmark datasets.

Accelerated First-Order Primal-Dual Proximal Methods for Linearly Constrained Composite Convex Programming

Motivated by big data applications, first-order methods have been extremely popular in recent years. However, naive gradient methods generally converge slowly. Hence, much effort has been made to accelerate various first-order methods. This paper proposes two accelerated methods towards solving structured linearly constrained convex programming, for which we assume composite convex objective that is the sum of a differentiable function and a possibly nondifferentiable one. The first method is the accelerated linearized augmented Lagrangian method (LALM). At each update to the primal variable, it allows linearization to the differentiable function and also the augmented term, and thus it enables easy subproblems. Assuming merely convexity, we show that LALM owns $O(1/t)$ convergence if parameters are kept fixed during all the iterations and can be accelerated to $O(1/t^2)$ if the parameters are adapted, where $t$ is the number of total iterations. The second method is the accelerated linearized alternating direction method of multipliers (LADMM). In addition to the composite convexity, it further assumes two-block structure on the objective. Different from classic alternating direction method of multipliers, our method allows linearization to the objective and also augmented term to make the update simple. Assuming strong convexity on one block variable, we show that LADMM also enjoys $O(1/t^2)$ convergence with adaptive parameters. This result is a significant improvement over that in [Goldstein et. al, SIAM J. Imag. Sci., 7 (2014), pp. 1588--1623], which requires strong convexity on both block variables and no linearization to the objective or augmented term. Numerical experiments are performed on quadratic programming, image denoising, and support vector machine. The proposed accelerated methods are compared to nonaccelerated ones and also existing accelerated methods. The results demonstrate the validity of acceleration and superior performance of the proposed methods over existing ones.

Exact Worst-Case Performance of First-Order Methods for Composite Convex Optimization

We provide a framework for computing the exact worst-case performance of any algorithm belonging to a broad class of oracle-based first-order methods for composite convex optimization, including those performing explicit, projected, proximal, conditional and inexact (sub)gradient steps. We simultaneously obtain tight worst-case guarantees and explicit instances of optimization problems on which the algorithm reaches this worst-case. We achieve this by reducing the computation of the worst-case to solving a convex semidefinite program, generalizing previous works on performance estimation by Drori and Teboulle [13] and the authors [43]. We use these developments to obtain a tighter analysis of the proximal point algorithm and of several variants of fast proximal gradient, conditional gradient, subgradient and alternating projection methods. In particular, we present a new analytical worst-case guarantee for the proximal point algorithm that is twice better than previously known, and improve the standard worst-case guarantee for the conditional gradient method by more than a factor of two. We also show how the optimized gradient method proposed by Kim and Fessler in [22] can be extended by incorporating a projection or a proximal operator, which leads to an algorithm that converges in the worst-case twice as fast as the standard accelerated proximal gradient method [2].

An accelerated non-Euclidean hybrid proximal extragradient-type algorithm for convex–concave saddle-point problems

This paper describes an accelerated HPE-type method based on general Bregman distances for solving convex–concave saddle-point (SP) problems. The algorithm is a special instance of a non-Euclidean hybrid proximal extragradient framework introduced by Svaiter and Solodov [An inexact hybrid generalized proximal point algorithm and some new results on the theory of Bregman functions, Math. Oper. Res. 25(2) (2000), pp. 214–230] where the prox sub-inclusions are solved using an accelerated gradient method. It generalizes the accelerated HPE algorithm presented in He and Monteiro [An accelerated HPE-type algorithm for a class of composite convex–concave saddle-point problems, SIAM J. Optim. 26 (2016), pp. 29–56] in two ways, namely: (a) it deals with general monotone SP problems instead of bilinear structured SPs and (b) it is based on general Bregman distances instead of the Euclidean one. Similar to the algorithm of He and Monteiro [An accelerated HPE-type algorithm for a class of composite convex–concave saddle-point problems, SIAM J. Optim. 26 (2016), pp. 29–56], it has the advantage that it works for any constant choice of proximal stepsize. Moreover, a suitable choice of the stepsize yields a method with the best known iteration-complexity for solving monotone SP problems. Computational results show that the new method is superior to Nesterov's [Smooth minimization of non-smooth functions, Math. Program. 103(1) (2005), pp. 127–152] smoothing scheme.

A splitting primal-dual proximity algorithm for solving composite optimization problems

Our work considers the optimization of the sum of a non-smooth convex function and a finite family of composite convex functions, each one of which is composed of a convex function and a bounded linear operator. This type of problem is associated with many interesting challenges encountered in the image restoration and image reconstruction fields. We developed a splitting primal-dual proximity algorithm to solve this problem. Further, we propose a preconditioned method, of which the iterative parameters are obtained without the need to know some particular operator norm in advance. Theoretical convergence theorems are presented. We then apply the proposed methods to solve a total variation regularization model, in which the L2 data error function is added to the L1 data error function. The main advantageous feature of this model is its capability to combine different loss functions. The numerical results obtained for computed tomography (CT) image reconstruction demonstrated the ability of the proposed algorithm to reconstruct an image with few and sparse projection views while maintaining the image quality.

Approximate Optimality Conditions for Composite Convex Optimization Problems

The purpose of this paper is to study the approximate optimality condition for composite convex optimization problems with a cone-convex system in locally convex spaces, where all functions involved are not necessarily lower semi-continuous. By using the properties of the epigraph of conjugate functions, we introduce a new regularity condition and give its equivalent characterizations. Under this new regularity condition, we derive necessary and sufficient optimality conditions of \(\varepsilon \)-optimal solutions for the composite convex optimization problem. As applications of our results, we derive approximate optimality conditions to cone-convex optimization problems. Our results extend or cover many known results in the literature.

Adaptive smoothing algorithms for nonsmooth composite convex minimization

We propose an adaptive smoothing algorithm based on Nesterov's smoothing technique in Nesterov (Math Prog 103(1):127---152, 2005) for solving fully nonsmooth composite convex optimization problems. Our method combines both Nesterov's accelerated proximal gradient scheme and a new homotopy strategy for smoothness parameter. By an appropriate choice of smoothing functions, we develop a new algorithm that has the $$\mathcal {O}\left( \frac{1}{\varepsilon }\right) $$O1?-worst-case iteration-complexity while preserves the same complexity-per-iteration as in Nesterov's method and allows one to automatically update the smoothness parameter at each iteration. Then, we customize our algorithm to solve four special cases that cover various applications. We also specify our algorithm to solve constrained convex optimization problems and show its convergence guarantee on a primal sequence of iterates. We demonstrate our algorithm through three numerical examples and compare it with other related algorithms.

The stable duality of DC programs for composite convex functions

In this paper, we consider a composite DC optimization problem with a cone-convex system in locally convex Hausdorff topological vector spaces. By using the properties of the epigraph of the conjugate functions, some necessary and sufficient conditions which characterize the strong Fenchel-Lagrange duality and the stable strong Fenchel-Lagrange duality are given. We apply the results obtained to study the minmax optimization problem and $l_1$ penalty problem.

MAGMA: Multilevel Accelerated Gradient Mirror Descent Algorithm for Large-Scale Convex Composite Minimization

Composite convex optimization models arise in several applications and are especially prevalent in inverse problems with a sparsity inducing norm and in general convex optimization with simple constraints. The most widely used algorithms for convex composite models are accelerated first order methods; however, they can take a large number of iterations to compute an acceptable solution for large-scale problems. In this paper we propose speeding up first order methods by taking advantage of the structure present in many applications and in image processing in particular. Our method is based on multilevel optimization methods and exploits the fact that many applications that give rise to large-scale models can be modeled using varying degrees of fidelity. We use Nesterov's acceleration techniques together with the multilevel approach to achieve an $\mathcal{O}(1/\sqrt{\epsilon})$ convergence rate, where $\epsilon$ denotes the desired accuracy. The proposed method has a better convergence rate than any other existing multilevel method for convex problems and in addition has the same rate as accelerated methods, which is known to be optimal for first order methods. Moreover, as our numerical experiments show, on large-scale face recognition problems our algorithm is several times faster than the state of the art.

An adaptive accelerated first-order method for convex optimization

This paper presents a new accelerated variant of Nesterov's method for solving composite convex optimization problems in which certain acceleration parameters are adaptively (and aggressively) chosen so as to substantially improve its practical performance compared to existing accelerated variants while at the same time preserve the optimal iteration-complexity shared by these methods. Computational results are presented to demonstrate that the proposed adaptive accelerated method endowed with a restarting scheme outperforms other existing accelerated variants.

The Stable Farkas Lemma for composite convex functions in infinite dimensional spaces

In this paper, we consider a general composite convex optimization problem with a cone-convex system in locally convex Hausdorff topological vector spaces. Some Fenchel conjugate transforms for the composite convex functions are derived to obtain the equivalent condition of the Stable Farkas Lemma, which is formulated by using the epigraph of the conjugates for the convex functions involved and turns out to be weaker than the classic Slater condition. Moreover, we get some necessary and sufficient conditions for stable duality results of the composite convex functions and present an example to illustrate that the monotonic increasing property of the outer convex function in the objective function is essential. Our main results in this paper develop some recently results.

A Parallel Line Search Subspace Correction Method for Composite Convex Optimization

In this paper, we investigate a parallel subspace correction framework for composite convex optimization. The variables are first divided into a few blocks based on certain rules. At each iteration, the algorithms solve a suitable subproblem on each block simultaneously, construct a search direction by combining their solutions on all blocks, then identify a new point along this direction using a step size satisfying the Armijo line search condition. They are called PSCLN and PSCLO, respectively, depending on whether there are overlapping regions between two immediately adjacent blocks of variables. Their convergence is established under mild assumptions. We compare PSCLN and PSCLO with the parallel version of the fast iterative thresholding algorithm and the fixed-point continuation method using the Barzilai-Borwein step size and the greedy coordinate block descent method for solving the \(\ell _1\)-regularized minimization problems. Our numerical results show that PSCLN and PSCLO can run fast and return solutions not worse than those from the state-of-the-art algorithms on most test problems. It is also observed that the overlapping domain decomposition scheme is helpful when the data of the problem has certain special structures.

Accelerating Block-Decomposition First-Order Methods for Solving Composite Saddle-Point and Two-Player Nash Equilibrium Problems

This article considers the (two-player) composite Nash equilibrium (CNE) problem with a separable nonsmooth part, which is known to include the composite saddle-point (CSP) problem as a special case. Due to its two-block structure, this problem can be solved by any algorithm belonging to the block-decomposition hybrid proximal-extragradient (BD-HPE) framework proposed in [R. D. C. Monteiro and B. F. Svaiter, SIAM J. Optim., 23 (2013), pp. 475--507]. The framework consists of a family of inexact proximal point methods for solving a more general two-block structured monotone inclusion problem which, at every iteration, solves two prox subinclusions according to a certain relative error criterion. By exploiting the fact that the two prox subinclusions in the context of the CNE problem are equivalent to two composite convex programs, this article proposes a new instance of the BD-HPE framework that approximately solves them using an accelerated gradient method. It is shown that the new instance is able to take significantly larger prox stepsizes than other instances from this framework that perform single composite gradient steps for solving the subinclusions. As a result, it is shown that the first instance has better iteration-complexity than the latter ones. Finally, it is also shown that the new accelerated BD-HPE instance computationally outperforms several state-of-the-art algorithms on many relevant classes of CSP and CNE instances.

A total variation based nonrigid image registration by combining parametric and non-parametric transformation models

To overcome the conflict between the global robustness and the local accuracy of dense nonrigid image registration, we propose a union registration approach by combining parametric and non-parametric transformation models. On one hand, to guarantee the robustness, we constrain the displacement field @f using a mapping difference metric between the B-spline parametric space @J and the non-parametric transformation space @F. On the other hand, to correct the densely and highly localized geometrical distortions, we introduce a total variation (TV) regularization term for the displacement field @f. Accounting for the effect of spatially varying intensity distortions, the residual complexity (RC) is used as the similarity metric. Moreover, to solve the proposed union nonrigid registration, which is a composite convex optimization problem by the smooth @?2 term and the non-smooth @?1 term (TV), we design a two-stage algorithm using split Bregman iteration. Experiments with both synthetic and real images from different domains illustrate that this approach can capture the local details of transformation accurately and effectively while being robust to the spatially varying intensity distortions.

Fast First-Order Methods for Composite Convex Optimization with Backtracking

We propose new versions of accelerated first-order methods for convex composite optimization, where the prox parameter is allowed to increase from one iteration to the next. In particular, we show ...

Comments on: Farkas’ lemma: three decades of generalizations for mathematical optimization

The celebrated Farkas lemma originated in Farkas (1902) provides us an attractively simple and extremely useful characterization for a linear inequality to be a consequence of a linear inequality system on the Euclidean space Rn . This lemma has been extensively studied and extended in many directions, including conic systems (linear, sublinear, convex), infinite/semi-infinite convex inequality systems and some classes of nonconvex systems such as the difference of convex (DC) systems and composite convex systems, and the rich results are spread in the literature.

From the Farkas Lemma to the Hahn--Banach Theorem

This paper provides new versions of the Farkas lemma characterizing those inequalities of the form $f(x)\geq 0$ which are consequences of a composite convex inequality $(S\circ g)(x)\leq 0$ on a closed convex subset of a given locally convex topological vector space $X$, where $f$ is a proper lower semicontinuous convex function defined on $X,$ $S$ is an extended sublinear function, and $g$ is a vector-valued $S$-convex function. In parallel, associated versions of a stable Farkas lemma, considering arbitrary linear perturbations of $f$, are also given. These new versions of the Farkas lemma, and their corresponding stable forms, are established under the weakest constraint qualification conditions (the so-called closedness conditions), and they are actually equivalent to each other, as well as equivalent to an extended version of the so-called Hahn--Banach--Lagrange theorem, and its stable version, correspondingly. It is shown that any of them implies analytic and algebraic versions of the Hahn--Banach t...

The Toland-Fenchel-Lagrange duality of DC programs for composite convex functions

In this paper, by virtue of the epigraph technique, we construct a new kind of closedness-type constraintqualification, which is the sufficient and necessary condition to guarantee thestrong duality between a cone constraint composite optimization problem and its dual problem holds. Under this closedness-type constraint qualification condition, we obtain a formula of subdifferential for composite functions and study a cone constraint composite DC optimization problem in locally convex Hausdorff topological vector spaces.

Characterizations of Asymptotic Cone of the Solution Set of a Composite Convex Optimization Problem

We characterize the asymptotic cone of the solution set of a convex composite optimization problem. We then apply the obtained results to study the necessary and sufficient conditions for the nonemptiness and compactness of the solution set of the problem. Our results generalize and improve some known results in literature.

An Inexact Accelerated Proximal Gradient Method for Large Scale Linearly Constrained Convex SDP

The accelerated proximal gradient (APG) method, first proposed by Nesterov for minimizing smooth convex functions, later extended by Beck and Teboulle to composite convex objective functions, and studied in a unifying manner by Tseng, has proven to be highly efficient in solving some classes of large scale structured convex optimization (possibly nonsmooth) problems, including nuclear norm minimization problems in matrix completion and $l_1$ minimization problems in compressed sensing. The method has superior worst-case iteration complexity over the classical projected gradient method and usually has good practical performance on problems with appropriate structures. In this paper, we extend the APG method to the inexact setting, where the subproblem in each iteration is solved only approximately, and show that it enjoys the same worst-case iteration complexity as the exact counterpart if the subproblems are progressively solved to sufficient accuracy. We apply our inexact APG method to solve large scale convex quadratic semidefinite programming (QSDP) problems of the form $\min\{ \frac{1}{2}\langle x,\,\mathcal{Q}(x)\rangle + \langle c,\,x\rangle\mid \mathcal{A}(x) = b, x\succeq 0\}$, where $\mathcal{Q},\mathcal{A}$ are given linear maps and $b,c$ are given data. The subproblem in each iteration is solved by a semismooth Newton-CG (SSNCG) method with warm-start using the iterate from the previous iteration. Our APG-SSNCG method is demonstrated to be efficient for QSDP problems whose positive semidefinite linear maps $\mathcal{Q}$ are highly ill-conditioned or rank deficient.