Convex Composite Optimization Research Articles

Four-Operator Splitting via a Forward–Backward–Half-Forward Algorithm with Line Search

In this article, we provide a splitting method for solving monotone inclusions in a real Hilbert space involving four operators: a maximally monotone, a monotone-Lipschitzian, a cocoercive, and a monotone-continuous operator. The proposed method takes advantage of the intrinsic properties of each operator, generalizing the forward–backward–half-forward splitting and the Tseng’s algorithm with line search. At each iteration, our algorithm defines the step size by using a line search in which the monotone-Lipschitzian and the cocoercive operators need only one activation. We also derive a method for solving nonlinearly constrained composite convex optimization problems in real Hilbert spaces. Finally, we implement our algorithm in a nonlinearly constrained least-square problem and we compare its performance with available methods in the literature.

Forward-partial inverse-half-forward splitting algorithm for solving monotone inclusions

In this paper we provide a splitting algorithm for solving coupled monotone inclusions in a real Hilbert space involving the sum of a normal cone to a vector subspace, a maximally monotone, a monotone-Lipschitzian, and a cocoercive operator. The proposed method takes advantage of the intrinsic properties of each operator and generalizes the method of partial inverses and the forward-backward-half forward splitting, among other methods. At each iteration, our algorithm needs two computations of the Lipschitzian operator while the cocoercive operator is activated only once. By using product space techniques, we derive a method for solving a composite monotone primal-dual inclusions including linear operators and we apply it to solve constrained composite convex optimization problems. Finally, we apply our algorithm to a constrained total variation least-squares problem and we compare its performance with efficient methods in the literature.

A nested primal–dual FISTA-like scheme for composite convex optimization problems

We propose a nested primal–dual algorithm with extrapolation on the primal variable suited for minimizing the sum of two convex functions, one of which is continuously differentiable. The proposed algorithm can be interpreted as an inexact inertial forward–backward algorithm equipped with a prefixed number of inner primal–dual iterations for the proximal evaluation and a “warm–start” strategy for starting the inner loop, and generalizes several nested primal–dual algorithms already available in the literature. By appropriately choosing the inertial parameters, we prove the convergence of the iterates to a saddle point of the problem, and provide an O(1/n) convergence rate on the primal–dual gap evaluated at the corresponding ergodic sequences. Numerical experiments on some image restoration problems show that the combination of the “warm–start” strategy with an appropriate choice of the inertial parameters is strictly required in order to guarantee the convergence to the real minimum point of the objective function.

An accelerated forward-backward-half forward splitting algorithm for monotone inclusion with applications to image restoration

ABSTRACT Inertial methods play a vital role in accelerating the convergence speed of optimization algorithms. We present an inertial forward-backward-half forward splitting algorithm, which mainly finds a zero of the sum of three operators, where two of them are cocoercive operator and monotone-Lipschitz continuous respectively. Meanwhile, the convergence analysis of the proposed algorithm is established under mild conditions. To overcome the difficulty in the calculation for the resolvent of the composite operator, relying on a primal-dual idea, we expand the proposed algorithm to solve the composite inclusion problem involving a linearly composed monotone operator. As an application, we make use of the obtained inertial algorithm to deal with a composite convex optimization problem. We also show extensive numerical experiments on the total variation-based image deblurring problem to demonstrate the efficiency of the proposed algorithm. Specifically, the proposed algorithm not only has a better quality of the deblurring image but also converges more rapidly than the original one.

Resolvent of the parallel composition and the proximity operator of the infimal postcomposition

In this paper we provide the resolvent computation of the parallel composition of a maximally monotone operator by a linear operator under mild assumptions. Connections with a modification of the warped resolvent are provided. In the context of convex optimization, we obtain the proximity operator of the infimal postcomposition of a convex function by a linear operator and we relax full range conditions on the linear operator to mild qualification conditions. We also introduce a generalization of the proximity operator involving a general linear bounded operator leading to a generalization of Moreau’s decomposition for composite convex optimization.

Analysis of the Frank–Wolfe method for convex composite optimization involving a logarithmically-homogeneous barrier

We present and analyze a new generalized Frank–Wolfe method for the composite optimization problem (P): {min }_{xin {mathbb {R}}^n} ; f(mathsf {A} x) + h(x), where f is a theta -logarithmically-homogeneous self-concordant barrier, mathsf {A} is a linear operator and the function h has a bounded domain but is possibly non-smooth. We show that our generalized Frank–Wolfe method requires O((delta _0 + theta + R_h)ln (delta _0) + (theta + R_h)^2/varepsilon ) iterations to produce an varepsilon -approximate solution, where delta _0 denotes the initial optimality gap and R_h is the variation of h on its domain. This result establishes certain intrinsic connections between theta -logarithmically homogeneous barriers and the Frank–Wolfe method. When specialized to the D-optimal design problem, we essentially recover the complexity obtained by Khachiyan (Math Oper Res 21 (2): 307–320, 1996) using the Frank–Wolfe method with exact line-search. We also study the (Fenchel) dual problem of (P), and we show that our new method is equivalent to an adaptive-step-size mirror descent method applied to the dual problem. This enables us to provide iteration complexity bounds for the mirror descent method despite the fact that the dual objective function is non-Lipschitz and has unbounded domain. In addition, we present computational experiments that point to the potential usefulness of our generalized Frank–Wolfe method on Poisson image de-blurring problems with TV regularization, and on simulated PET problem instances.

A globally convergent proximal Newton-type method in nonsmooth convex optimization

The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structured problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumption used in the literature to ensure a superlinear convergence of other algorithms of the proximal Newton type. The conducted numerical experiments of solving the \(l_1\) regularized logistic regression model illustrate the possibility of applying the proposed algorithm to deal with practically important problems.

Multi-step inertial forward-backward-half forward algorithm for solving monotone inclusion

In this article, we study a multi-step inertial forward-backward-half forward splitting algorithm to solve the solution of the sum of three operators with two of them being respectively, cocoercive and monotone-Lipschitz continuous. Notably, the convergence of the multi-step inertial forward-backward-half forward algorithm is established under the summability condition formulated on the basis of the iterative sequence in a Hilbert space setting. We also display two applications including problems that are: a composite monotone inclusion problem involving mixtures of linearly composed with parallel-sum type operators, and a composite convex optimization problem. As far as the resolvent of the composite operator is concerned, due to difficulty in calculation, then, the composite monotone inclusion problem is addressed by the proposed multi-step inertial algorithm together with the primal-dual idea. Experiments conducted on an image deblurring problem indicate the superiority and efficiency of the proposed method.

From differential equation solvers to accelerated first-order methods for convex optimization

Convergence analysis of accelerated first-order methods for convex optimization problems are developed from the point of view of ordinary differential equation solvers. A new dynamical system, called Nesterov accelerated gradient (NAG) flow, is derived from the connection between acceleration mechanism and A-stability of ODE solvers, and the exponential decay of a tailored Lyapunov function along with the solution trajectory is proved. Numerical discretizations of NAG flow are then considered and convergence rates are established via a discrete Lyapunov function. The proposed differential equation solver approach can not only cover existing accelerated methods, such as FISTA, Güler’s proximal algorithm and Nesterov’s accelerated gradient method, but also produce new algorithms for composite convex optimization that possess accelerated convergence rates. Both the convex and the strongly convex cases are handled in a unified way in our approach.

A New Homotopy Proximal Variable-Metric Framework for Composite Convex Minimization

This paper suggests two novel ideas to develop new proximal variable-metric methods for solving a class of composite convex optimization problems. The first idea is to utilize a new parameterization strategy of the optimality condition to design a class of homotopy proximal variable-metric algorithms that can achieve linear convergence and finite global iteration-complexity bounds. We identify at least three subclasses of convex problems in which our approach can apply to achieve linear convergence rates. The second idea is a new primal-dual-primal framework for implementing proximal Newton methods that has attractive computational features for a subclass of nonsmooth composite convex minimization problems. We specialize the proposed algorithm to solve a covariance estimation problem in order to demonstrate its computational advantages. Numerical experiments on the four concrete applications are given to illustrate the theoretical and computational advances of the new methods compared with other state-of-the-art algorithms.

A Novel Quasi-Newton Method for Composite Convex Minimization

A fast parallelable Jacobi iteration type optimization method for non-smooth convex composite optimization is presented. Traditional gradient-based techniques cannot solve the problem. Smooth approximate functions are attempted to be used as a replacement of those non-smooth terms without compromising the accuracy. Recently, proximal mapping concept has been introduced into this field. Techniques which utilize proximal average based proximal gradient have been used to solve the problem. The state-of-art methods only utilize first-order information of the smooth approximate function. We integrate both first and second-order techniques to use both first and second-order information to boost the convergence speed. A convergence rate with a lower bound of O(1k2) is achieved by the proposed method and a super-linear convergence is enjoyed when there is proper second-order information. In experiments, the proposed method converges significantly better than the state of art methods which enjoy O(1k) convergence.

An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

We propose an inexact proximal augmented Lagrangian framework with explicit inner problem termination rule for composite convex optimization problems. We consider arbitrary linearly convergent inner solver including in particular stochastic algorithms, making the resulting framework more scalable facing the ever-increasing problem dimension. Each subproblem is solved inexactly with an explicit and self-adaptive stopping criterion, without requiring to set an a priori target accuracy. When the primal and dual domain are bounded, our method achieves \(O(1/\sqrt{\epsilon })\) and \(O(1/{\epsilon })\) complexity bound in terms of number of inner solver iterations, respectively for the strongly convex and non-strongly convex case. Without the boundedness assumption, only logarithm terms need to be added and the above two complexity bounds increase respectively to \({\tilde{O}}(1/\sqrt{\epsilon })\) and \({\tilde{O}}(1/{\epsilon })\), which hold both for obtaining \(\epsilon \)-optimal and \(\epsilon \)-KKT solution. Within the general framework that we propose, we also obtain \({\tilde{O}}(1/{\epsilon })\) and \({\tilde{O}}(1/{\epsilon ^2})\) complexity bounds under relative smoothness assumption on the differentiable component of the objective function. We show through theoretical analysis as well as numerical experiments the computational speedup possibly achieved by the use of randomized inner solvers for large-scale problems.

Iteration complexity analysis of a partial LQP-based alternating direction method of multipliers

In this paper, we consider a prototypical convex optimization problem with multi-block variables and separable structures. By adding the Logarithmic Quadratic Proximal (LQP) regularizer with suitable proximal parameter to each of the first grouped subproblems, we develop a partial LQP-based Alternating Direction Method of Multipliers (ADMM-LQP). The dual variable is updated twice with relatively larger stepsizes than the classical region (0,1+52). Using a prediction-correction approach to analyze properties of the iterates generated by ADMM-LQP, we establish its global convergence and sublinear convergence rate of O(1/T) in the new ergodic and nonergodic senses, where T denotes the iteration index. We also extend the algorithm to a nonsmooth composite convex optimization and establish similar convergence results as our ADMM-LQP.

General Convergence Analysis of Stochastic First-Order Methods for Composite Optimization

In this paper, we consider stochastic composite convex optimization problems with the objective function satisfying a stochastic bounded gradient condition, with or without a quadratic functional growth property. These models include the most well-known classes of objective functions analyzed in the literature: nonsmooth Lipschitz functions and composition of a (potentially) nonsmooth function and a smooth function, with or without strong convexity. Based on the flexibility offered by our optimization model, we consider several variants of stochastic first-order methods, such as the stochastic proximal gradient and the stochastic proximal point algorithms. Usually, the convergence theory for these methods has been derived for simple stochastic optimization models satisfying restrictive assumptions, and the rates are in general sublinear and hold only for specific decreasing stepsizes. Hence, we analyze the convergence rates of stochastic first-order methods with constant or variable stepsize under general assumptions covering a large class of objective functions. For constant stepsize, we show that these methods can achieve linear convergence rate up to a constant proportional to the stepsize and under some strong stochastic bounded gradient condition even pure linear convergence. Moreover, when a variable stepsize is chosen we derive sublinear convergence rates for these stochastic first-order methods. Finally, the stochastic gradient mapping and the Moreau smoothing mapping introduced in the present paper lead to simple and intuitive proofs.

A Unified Convergence Analysis of Stochastic Bregman Proximal Gradient and Extragradient Methods

We consider a mini-batch stochastic Bregman proximal gradient method and a mini-batch stochastic Bregman proximal extragradient method for stochastic convex composite optimization problems. A simplified and unified convergence analysis framework is proposed to obtain almost sure convergence properties and expected convergence rates of the mini-batch stochastic Bregman proximal gradient method and its variants. This framework can also be used to analyze the convergence of the mini-batch stochastic Bregman proximal extragradient method, which has seldom been discussed in the literature. We point out that the standard uniformly bounded variance assumption and the usual Lipschitz gradient continuity assumption are not required in the analysis.

A Penalty Alternating Direction Method of Multipliers for Convex Composite Optimization Over Decentralized Networks

Consider the problem of minimizing a sum of convex composite functions over a decentralized network, where each agent in the network holds a private function consisting of a smooth part and a nonsmooth part, and it can only exchange information with its neighbors during the optimization process. One approach to tackling this problem is to study its penalized approximation. Although such an approximation becomes more accurate as the penalty parameter becomes smaller, it is well known that the penalized objective will also become more ill-conditioned, thereby causing the popular proximal gradient descent method to slow down substantially. To break this accuracy-speed tradeoff, we propose to solve the penalized approximation with the alternating direction method of multipliers (ADMM). We also exploit the composite structure of the private functions by linearizing the smooth parts and handling the nonsmooth parts with proximal operators, which allows us to further reduce the computational costs. The proposed penalty ADMM (abbreviated as PAD) is proven to be sublinearly convergent when the private functions are convex, and linearly convergent when in addition the smooth parts are strongly convex. We present numerical results to corroborate the theoretical analyses and to further demonstrate the advantages of PAD over existing state-of-the-art algorithms such as DL-ADMM, PG-EXTRA, and NIDS.

New inertial proximal gradient methods for unconstrained convex optimization problems

The proximal gradient method is a highly powerful tool for solving the composite convex optimization problem. In this paper, firstly, we propose inexact inertial acceleration methods based on the viscosity approximation and proximal scaled gradient algorithm to accelerate the convergence of the algorithm. Under reasonable parameters, we prove that our algorithms strongly converge to some solution of the problem, which is the unique solution of a variational inequality problem. Secondly, we propose an inexact alternated inertial proximal point algorithm. Under suitable conditions, the weak convergence theorem is proved. Finally, numerical results illustrate the performances of our algorithms and present a comparison with related algorithms. Our results improve and extend the corresponding results reported by many authors recently.

Incremental proximal gradient scheme with penalization for constrained composite convex optimization problems

We consider the problem of minimizing a finite sum of convex functions subject to the set of minimizers of a convex differentiable function. In order to solve the problem, an algorithm combining the incremental proximal gradient method with smooth penalization technique is proposed. We show the convergence of the generated sequence of iterates to an optimal solution of the optimization problems, provided that a condition expressed via the Fenchel conjugate of the constraint function is fulfilled. Finally, the functionality of the method is illustrated by some numerical experiments addressing image inpainting problems and generalized Heron problems with least squares constraints.

Strong Metric (Sub)regularity of Karush–Kuhn–Tucker Mappings for Piecewise Linear-Quadratic Convex-Composite Optimization and the Quadratic Convergence of Newton’s Method

This work concerns the local convergence theory of Newton and quasi-Newton methods for convex-composite optimization: where one minimizes an objective that can be written as the composition of a convex function with one that is continuiously differentiable. We focus on the case in which the convex function is a potentially infinite-valued piecewise linear-quadratic function. Such problems include nonlinear programming, mini-max optimization, and estimation of nonlinear dynamics with non-Gaussian noise as well as many modern approaches to large-scale data analysis and machine learning. Our approach embeds the optimality conditions for convex-composite optimization problems into a generalized equation. We establish conditions for strong metric subregularity and strong metric regularity of the corresponding set-valued mappings. This allows us to extend classical convergence of Newton and quasi-Newton methods to the broader class of nonfinite valued piecewise linear-quadratic convex-composite optimization problems. In particular, we establish local quadratic convergence of the Newton method under conditions that parallel those in nonlinear programming.

An Inexact Interior-Point Lagrangian Decomposition Algorithm with Inexact Oracles

We develop a new inexact interior-point Lagrangian decomposition method to solve a wide range class of constrained composite convex optimization problems. Our method relies on four techniques: Lagrangian dual decomposition, self-concordant barrier smoothing, path-following, and proximal-Newton technique. It also allows one to approximately compute the solution of the primal subproblems (called the slave problems), which leads to inexact oracles (i.e., inexact gradients and Hessians) of the smoothed dual problem (called the master problem). The smoothed dual problem is nonsmooth, we propose to use an inexact proximal-Newton method to solve it. By appropriately controlling the inexact computation at both levels: the slave and master problems, we still estimate a polynomial-time iteration-complexity of our algorithm as in standard short-step interior-point methods. We also provide a strategy to recover primal solutions and establish complexity to achieve an approximate primal solution. We illustrate our method through two numerical examples on well-known models with both synthetic and real data and compare it with some existing state-of-the-art methods.