Composite Minimization Problems Research Articles

Minimizing Compositions of Differences-of-Convex Functions with Smooth Mappings

We address the so-called DC (difference-of-convex functions) composite minimization problems (or DC composite programs) whose objective function is a composition of a DC function with a continuously differentiable mapping. We first develop an algorithm named DC composite algorithm (DCCA in short) for unconstrained DC composite programs and further extend to DC composite programs with constraints of inclusion associated with a smooth mapping and a closed convex set. The convergence analysis of the proposed algorithms is investigated. Applications of DCCA for two different problems, computation of the numerical radius of a square matrix and minimization of composite energies, are presented.

Inexact and Stochastic Generalized Conditional Gradient with Augmented Lagrangian and Proximal Step

In this paper we propose and analyze inexact and stochastic versions of the CGALP algorithm developed in [25], which we denote ICGALP , that allow for errors in the computation of several important quantities. In particular this allows one to compute some gradients, proximal terms, and/or linear minimization oracles in an inexact fashion that facilitates the practical application of the algorithm to computationally intensive settings, e.g., in high (or possibly infinite) dimensional Hilbert spaces commonly found in machine learning problems. The algorithm is able to solve composite minimization problems involving the sum of three convex proper lower-semicontinuous functions subject to an affine constraint of the form Ax = b for some bounded linear operator A. Only one of the functions in the objective is assumed to be differentiable, the other two are assumed to have an accessible proximal operator and a linear minimization oracle. As main results, we show convergence of the Lagrangian values (so-called convergence in the Bregman sense) and asymptotic feasibility of the affine constraint as well as strong convergence of the sequence of dual variables to a solution of the dual problem, in an almost sure sense. Almost sure convergence rates are given for the Lagrangian values and the feasibility gap for the ergodic primal variables. Rates in expectation are given for the Lagrangian values and the feasibility gap subsequentially in the pointwise sense. Numerical experiments verifying the predicted rates of convergence are shown as well.

Parallel Normal S-Iteration Methods with Applications to Optimization Problems

A large number of nonlinear and optimization problems can be reduced to altering point problems. This paper aims to introduce the parallel normal S-iteration technique and study its convergence rates for solving such problems in infinite-dimensional Hilbert spaces under practical assumptions. We place particular emphasis on the parallel splitting method for the sum of two maximal monotone operators and that can apply for solving a class of convex composite minimization problems. Moreover, we present applications of our iterative methods to some nonlinear problems, such as a system of variational inequalities and a system of inclusion problems. Finally, to demonstrate the applicability of the altering point technique, the performances of our proposed parallel normal S-iteration methods are presented through numerical experiments in signal recovery problems.

Minimizing Uniformly Convex Functions by Cubic Regularization of Newton Method

In this paper, we study the iteration complexity of cubic regularization of Newton method for solving composite minimization problems with uniformly convex objective. We introduce the notion of second-order condition number of a certain degree and justify the linear rate of convergence in a nondegenerate case for the method with an adaptive estimate of the regularization parameter. The algorithm automatically achieves the best possible global complexity bound among different problem classes of uniformly convex objective functions with Hölder continuous Hessian of the smooth part of the objective. As a byproduct of our developments, we justify an intuitively plausible result that the global iteration complexity of the Newton method is always better than that of the gradient method on the class of strongly convex functions with uniformly bounded second derivative.

An Average Curvature Accelerated Composite Gradient Method for Nonconvex Smooth Composite Optimization Problems

This paper presents an accelerated composite gradient (ACG) variant, referred to as the AC-ACG method, for solving nonconvex smooth composite minimization problems. As opposed to well-known ACG variants that are based on either a known Lipschitz gradient constant or a sequence of maximum observed curvatures, the current one is based on the average of all past observed curvatures. More specifically, AC-ACG uses a positive multiple of the average of all observed curvatures until the previous iteration as a way to estimate the “function curvature” at the current point and then two resolvent evaluations to compute the next iterate. In contrast to other variable Lipschitz estimation variants, e.g., the ones based on the maximum curvature, AC-ACG always accepts the aforementioned iterate regardless of how poor the Lipschitz estimation turns out to be. Finally, computational results are presented to illustrate the efficiency of AC-ACG on both randomly generated and real-world problem instances.

General-type proximal point algorithm for solving inclusion and fixed point problems with composite operators

The main purpose of this paper is to introduce a new general-type proximal point algorithm for finding a common element of the set of solutions of monotone inclusion problem, the set of minimizers of a convex function, and the set of solutions of fixed point problem with composite operators: the composition of quasi-nonexpansive and firmly nonexpansive mappings in real Hilbert spaces. We prove that the sequence xn which is generated by the proposed iterative algorithm converges strongly to a common element of the three sets above without commuting assumption on the mappings. Finally, applications of our theorems to find a common solution of some nonlinear problems, namely, composite minimization problems, convex optimization problems, and fixed point problems, are given to validate our new findings.

Convergence rate analysis of proximal gradient methods with applications to composite minimization problems

First-order methods such as proximal gradient, which use Forward–Backward Splitting techniques have proved to be very effective in solving nonsmooth convex minimization problem, which is useful in solving various practical problems in different fields such as machine learning and image processing. In this paper, we propose few new forward–backward splitting algorithms, which consume less number of iterations to converge to an optimum. In addition, we derive convergence rates for the proposed formulations and show that the speed of convergence of these algorithms is significantly better than the traditional forward–backward algorithm. To demonstrate the practical applicability, we apply them to two real-world problems of machine learning and image processing. The first issue deals with the regression on high-dimensional datasets, whereas the second one is the image deblurring problem. Numerical experiments have been conducted on several publicly available real datasets to verify the obtained theoretical results. Results demonstrate the superiority of our algorithms in terms of accuracy, the number of iterations required to converge and the rate of convergence against the classical first-order methods.

Composite Minimization Problems in Hadamard Spaces

In this paper, we prove some Δ-convergence and strong convergence results for the sequence generated by a new algorithm to a minimizer of two convex functions and a common fixed point for quasi-pseudo-contractive mappings in Hadamard spaces. Our theorems improve and generalize some recent results in the literature.

Novel Proximal Gradient Methods for Nonnegative Matrix Factorization with Sparsity Constraints

We consider the nonnegative matrix factorization (NMF) problem with sparsity constraints formulated as a nonconvex composite minimization problem. We introduce four novel proximal gradient based al...

On the Convergence of Bregman ADMM With Variational Inequality

The alternating direction method of multipliers (ADMM) is one of most foundational algorithms for linear constrained composite minimization problems. For different specific problems, variations of ADMM (like linearized ADMM, proximal ADMM) are developed. By using the Bregman distances, lots of ADMMs can be formulated into a uniform mathematical scheme. Although variational inequalities have been well used to study ADMMs, the use for BADMM has still been missing. In this paper, we study the convergence of BADMM by variational inequalities. We present a proof framework for BADMMs. And then, we present very concise convergence proof for the basic BADMM. As applications, we consider several variations of BADMM and obtain corresponding convergence results.

Proximal point algorithm for inclusion problems in Hadamard manifolds with applications

This paper deals with the proximal point algorithm for finding a singularity of sum of a single-valued vector field and a set-valued vector field in the setting of Hadamard manifolds. The convergence analysis of the proposed algorithm is discussed. Applications to composite minimization problems and variational inequality problems are also presented.

A modified hybrid algorithm for solving a composite minimization problem in Banach spaces

A modified hybrid algorithm for solving a composite minimization problem in Banach spaces

Convergence analysis on a modified generalized alternating direction method of multipliers

The alternating direction method of multipliers (ADMM) is one of the most powerful and successful methods for solving convex composite minimization problem. The generalized ADMM relaxes both the variables and the multipliers with a common relaxation factor in (0,2), which has the potential of enhancing the performance of the classic ADMM. Very recently, two different variants of semi-proximal generalized ADMM have been proposed. They allow the weighting matrix in the proximal terms to be positive semidefinite, which makes the subproblems relatively easy to evaluate. One of the variants of semi-proximal generalized ADMMs has been analyzed theoretically, but the convergence result of the other is not known so far. This paper aims to remedy this deficiency and establish its convergence result under some mild conditions in the sense that the relaxation factor is also restricted into (0,2).

Undersampled CS image reconstruction using nonconvex nonsmooth mixed constraints

Compressed sensing magnetic resonance imaging (CS-MRI) has attracted considerable attention due to its great potential in reducing scanning time and guaranteeing high-quality reconstruction. In conventional CS-MRI framework, the total variation (TV) penalty and L1-norm constraint on wavelet coefficients are commonly combined to reduce the reconstruction error. However, TV sometimes tends to cause staircase-like artifacts due to its nature in favoring piecewise constant solution. To overcome the model-dependent deficiency, a hybrid TV (TV1,2) regularizer is introduced in this paper by combining TV with its second-order version (TV2). It is well known that the wavelet coefficients of MR images are not only approximately sparse, but also have the property of tree-structured hierarchical sparsity. Therefore, a L0-regularized tree-structured sparsity constraint is proposed to better represent the measure of sparseness in wavelet domain. In what follows, we present our new CS-MRI framework by combining the TV1,2 regularizer and L0-regularized tree-structured sparsity constraint. However, the combination makes CS-MRI problem difficult to handle due to the nonconvex and nonsmooth natures of mixed constraints. To achieve solution stability, the resulting composite minimization problem is decomposed into several simpler subproblems. Each of these subproblems has a closed-form solution or could be efficiently solved using existing numerical method. The results from simulation and in vivo experiments have demonstrated the good performance of our proposed method compared with several conventional MRI reconstruction methods.

Elastic-net regularization for nonlinear electrical impedance tomography with a splitting approach

ABSTRACTImage reconstruction of EIT mathematically is a typical nonlinear and severely ill-posed inverse problem. Appropriate priors or penalties are required to enable the reconstruction. The commonly used -norm can enforce the stability to preserve local smoothness, and the current -norm can enforce the sparsity to preserve sharp edges. Considering the fact that -norm penalty always makes the solution overly smooth and -norm penalty always makes the solution too sparse, elastic-net regularization approach with a convex combination term of -norm and -norm emerges for fully nonlinear EIT inverse problems. Our aim is to combine the strength of both terms: sparsity in the transform domain and smoothness in the physical domain, in an attempt to improve the reconstruction resolution and robustness to noise. Nonlinearity and non-smoothness of the generated composite minimization problem make it challenging to find an efficient numerical solution. Then we develop one simple and fast numerical optimization scheme based on the split Bregman technique for the joint penalties regularization. The method is validated using simulated data for some typical conductivity distributions. Results indicate that the proposed inversion model with an appropriate parameter choice achieves an efficient regularization solution and enhances the quality of the reconstructed image.

A Nonmonotone Trust Region Method for Nonsmooth Composite Programming Problems

A class of nonsmooth composite minimization problems are considered in this paper. In practice, many problems in computational science, engineering and other enormous areas fall into this class of problems. Two obvious applications of this class of problems are least square problems with nonsmooth data and the problem of solving a system of nonsmooth equations. Because of its practical importance, this class of problem has received wide attention from the mathematical optimization community. In particular, Sampaio, Yuan and Sun has proposed a trust region method for this class of problems and also provided the convergence analysis to support their algorithm. Motivated partly by their work, in this paper a nonmonotone trust region algorithm for this class of nonsmooth composite minimization problems is presented. Different from most existing monotone line search and trust region methods, this method combines the nonmonotone technique to improve the efficiency of the trust region method. After a brief introduction of the class of problems in the first section, some fundamental concepts and properties which will be used in this paper are presented. Then, the new nonmonotone trust region algorithm for the class of problem is described followed by the global convergence analysis of the new algorithm. A simple application of this algorithm is discussed in the last part of this paper.

An efficient duality-based approach for PDE-constrained sparse optimization

In this paper, elliptic optimal control problems involving the $L^1$-control cost ($L^1$-EOCP) is considered. To numerically discretize $L^1$-EOCP, the standard piecewise linear finite element is employed. However, different from the finite dimensional $l^1$-regularization optimization, the resulting discrete $L^1$-norm does not have a decoupled form. A common approach to overcome this difficulty is employing a nodal quadrature formula to approximately discretize the $L^1$-norm. It is clear that this technique will incur an additional error. To avoid the additional error, solving $L^1$-EOCP via its dual, which can be reformulated as a multi-block unconstrained convex composite minimization problem, is considered. Motivated by the success of the accelerated block coordinate descent (ABCD) method for solving large scale convex minimization problems in finite dimensional space, we consider extending this method to $L^1$-EOCP. Hence, an efficient inexact ABCD method is introduced for solving $L^1$-EOCP. The design of this method combines an inexact 2-block majorized ABCD and the recent advances in the inexact symmetric Gauss-Seidel (sGS) technique for solving a multi-block convex composite quadratic programming whose objective contains a nonsmooth term involving only the first block. The proposed algorithm (called sGS-imABCD) is illustrated at two numerical examples. Numerical results not only confirm the finite element error estimates, but also show that our proposed algorithm is more efficient than (a) the ihADMM (inexact heterogeneous alternating direction method of multipliers), (b) the APG (accelerated proximal gradient) method.

Local and global convergence of a general inertial proximal splitting scheme for minimizing composite functions

This paper is concerned with convex composite minimization problems in a Hilbert space. In these problems, the objective is the sum of two closed, proper, and convex functions where one is smooth and the other admits a computationally inexpensive proximal operator. We analyze a family of generalized inertial proximal splitting algorithms (GIPSA) for solving such problems. We establish weak convergence of the generated sequence when the minimum is attained. Our analysis unifies and extends several previous results. We then focus on $$\ell _1$$ -regularized optimization, which is the ubiquitous special case where the nonsmooth term is the $$\ell _1$$ -norm. For certain parameter choices, GIPSA is amenable to a local analysis for this problem. For these choices we show that GIPSA achieves finite “active manifold identification”, i.e. convergence in a finite number of iterations to the optimal support and sign, after which GIPSA reduces to minimizing a local smooth function. We prove local linear convergence under either restricted strong convexity or a strict complementarity condition. We determine the rate in terms of the inertia, stepsize, and local curvature. Our local analysis is applicable to certain recent variants of the Fast Iterative Shrinkage–Thresholding Algorithm (FISTA), for which we establish active manifold identification and local linear convergence. Based on our analysis we propose a momentum restart scheme in these FISTA variants to obtain the optimal local linear convergence rate while maintaining desirable global properties.

Forward-Backward Splitting with Bregman Distances

We propose a forward-backward splitting algorithm based on Bregman distances for composite minimization problems in general reflexive Banach spaces. The convergence is established using the notion of variable quasi-Bregman monotone sequences. Various examples are discussed, including some in Euclidean spaces, where new algorithms are obtained.

Projection algorithms for composite minimization

Parallel and cyclic projection algorithms are proposed for minimizing the sum of a finite family of convex functions over the intersection of a finite family of closed convex subsets of a Hilbert space. These algorithms consist of two steps. Once the kth iterate is constructed, an inner circle of gradient descent process is executed through each local function, and then a parallel or cyclic projection process is applied to produce the (k + 1) iterate. These algorithms are proved to converge to an optimal solution of the composite minimization problem under investigation upon assuming boundedness of the gradients at the iterates of the local functions and the stepsizes being chosen appropriately.