Solving Monotone Inclusions Research Articles

A Three-Operator Splitting Scheme and its Optimization Applications

Operator-splitting methods convert optimization and inclusion problems into fixed-point equations; when applied to convex optimization and monotone inclusion problems, the equations given by operator-splitting methods are often easy to solve by standard techniques. The hard part of this conversion, then, is to design nicely behaved fixed-point equations. In this paper, we design a new, and thus far, the only nicely behaved fixed-point equation for solving monotone inclusions with three operators; the equation employs resolvent and forward operators, one at a time, in succession. We show that our new equation extends the Douglas-Rachford and forward-backward equations; we prove that standard methods for solving the equation converge; and we give two accelerated methods for solving the equation.

Accelerated schemes for a class of variational inequalities

We propose a novel stochastic method, namely the stochastic accelerated mirror-prox (SAMP) method, for solving a class of monotone stochastic variational inequalities (SVI). The main idea of the proposed algorithm is to incorporate a multi-step acceleration scheme into the stochastic mirror-prox method. The developed SAMP method computes weak solutions with the optimal iteration complexity for SVIs. In particular, if the operator in SVI consists of the stochastic gradient of a smooth function, the iteration complexity of the SAMP method can be accelerated in terms of their dependence on the Lipschitz constant of the smooth function. For SVIs with bounded feasible sets, the bound of the iteration complexity of the SAMP method depends on the diameter of the feasible set. For unbounded SVIs, we adopt the modified gap function introduced by Monteiro and Svaiter for solving monotone inclusion, and show that the iteration complexity of the SAMP method depends on the distance from the initial point to the set of strong solutions. It is worth noting that our study also significantly improves a few existing complexity results for solving deterministic variational inequality problems. We demonstrate the advantages of the SAMP method over some existing algorithms through our preliminary numerical experiments.

Asymmetric forward–backward–adjoint splitting for solving monotone inclusions involving three operators

In this work we propose a new splitting technique, namely Asymmetric Forward–Backward–Adjoint splitting, for solving monotone inclusions involving three terms, a maximally monotone, a cocoercive and a bounded linear operator. Our scheme can not be recovered from existing operator splitting methods, while classical methods like Douglas–Rachford and Forward–Backward splitting are special cases of the new algorithm. Asymmetric preconditioning is the main feature of Asymmetric Forward–Backward–Adjoint splitting, that allows us to unify, extend and shed light on the connections between many seemingly unrelated primal-dual algorithms for solving structured convex optimization problems proposed in recent years. One important special case leads to a Douglas–Rachford type scheme that includes a third cocoercive operator.

Quasi-Nonexpansive Iterations on the Affine Hull of Orbits: From Mann's Mean Value Algorithm to Inertial Methods

Fixed point iterations play a central role in the design and the analysis of a large number of optimization algorithms. We study a new iterative scheme in which the update is obtained by applying a composition of quasi-nonexpansive operators to a point in the affine hull of the orbit generated up to the current iterate. This investigation unifies several algorithmic constructs, including Mann's mean value method, inertial methods, and multilayer memoryless methods. It also provides a framework for the development of new algorithms, such as those we propose for solving monotone inclusion and minimization problems.

Solving monotone inclusions involving parallel sums of linearly composed maximally monotone operators

The aim of this article is to present two different primal-dual methods for solving structured monotone inclusions involving parallel sums of compositions of maximally monotone operators with linear bounded operators. By employing some elaborated splitting techniques, all of the operators occurring in the problem formulation are processed individually via forward or backward steps. The treatment of parallel sums of linearly composed maximally monotone operators is motivated by applications in imaging which involve first- and second-order total variation functionals, to which a special attention is given.

Penalty schemes with inertial effects for monotone inclusion problems

We introduce a penalty term-based splitting algorithm with inertial effects designed for solving monotone inclusion problems involving the sum of maximally monotone operators and the convex normal cone to the (nonempty) set of zeros of a monotone and Lipschitz continuous operator. We show weak ergodic convergence of the generated sequence of iterates to a solution of the monotone inclusion problem, provided a condition expressed via the Fitzpatrick function of the operator describing the underlying set of the normal cone is verified. Under strong monotonicity assumptions we can even show strong nonergodic convergence of the iterates. This approach constitutes the starting point for investigating from a similar perspective monotone inclusion problems involving linear compositions of parallel-sum operators and, further, for the minimization of a complexly structured convex objective function subject to the set of minima of another convex and differentiable function.

Regularized HPE-Type Methods for Solving Monotone Inclusions with Improved Pointwise Iteration-Complexity Bounds

This paper studies the iteration complexity of new regularized hybrid proximal extragradient (HPE)-type methods for solving monotone inclusion problems (MIPs). The new (regularized HPE-type) methods essentially consist of instances of the standard HPE method applied to regularizations of the original MIP. It is shown that the pointwise iteration complexity of the proposed methods considerably improves upon that of the HPE method while approaching (up to a logarithmic factor) the ergodic iteration complexity of the latter method.

A Variant of the Hybrid Proximal Extragradient Method for Solving Strongly Monotone Inclusions and its Complexity Analysis

This paper presents and studies the iteration-complexity of a variant of the hybrid proximal extragradient method for solving inclusion problems with strongly (maximal) monotone operators. As applications, we propose and analyze two special cases: variants of the Tseng's forward---backward method for solving monotone inclusions with strongly monotone and Lipschitz continuous operators and of the Korpelevich extragradient method for solving (strongly monotone) variational inequalities.

An inertial forward-backward-forward primal-dual splitting algorithm for solving monotone inclusion problems

We introduce and investigate the convergence properties of an inertial forward-backward-forward splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitzian operator. By making use of the product space approach, we expand it to the solving of inclusion problems involving mixtures of linearly composed and parallel-sum type monotone operators. We obtain in this way an inertial forward-backward-forward primal-dual splitting algorithm having as main characteristic the fact that in the iterative scheme all operators are accessed separately either via forward or via backward evaluations. We present also the variational case when one is interested in the solving of a primal-dual pair of convex optimization problems with complexly structured objectives, which we also illustrate by numerical experiments in image processing.

Forward–Partial Inverse–Forward Splitting for Solving Monotone Inclusions

In this paper, we provide a splitting method for finding a zero of the sum of a maximally monotone operator, a Lipschitzian monotone operator, and a normal cone to a closed vector subspace of a real Hilbert space. The problem is characterised by a simpler monotone inclusion involving only two operators: the partial inverse of the maximally monotone operator with respect to the vector subspace and a suitable Lipschitzian monotone operator. By applying the Tseng's method in this context, we obtain a fully split algorithm that exploits the whole structure of the original problem and generalises partial inverse and Tseng's methods. Connections with other methods available in the literature are provided, and the flexibility of our setting is illustrated via applications to some inclusions involving $$m$$m maximally monotone operators, to primal-dual composite monotone inclusions, and to zero-sum games.

Backward Penalty Schemes for Monotone Inclusion Problems

In this paper, we are concerned with solving monotone inclusion problems expressed by the sum of a set-valued maximally monotone operator with a single-valued maximally monotone one and the normal cone to the nonempty set of zeros of another set-valued maximally monotone operator. Depending on the nature of the single-valued operator, we propose two iterative penalty schemes, both addressing the set-valued operators via backward steps. The single-valued operator is evaluated via a single forward step if it is cocoercive, and via two forward steps if it is monotone and Lipschitz continuous. The latter situation represents the starting point for dealing with complexly structured monotone inclusion problems from algorithmic point of view.

On the convergence rate improvement of a primal-dual splitting algorithm for solving monotone inclusion problems

We present two modified versions of the primal-dual splitting algorithm relying on forward---backward splitting proposed in V $$\tilde{\mathrm{u}}$$ u ~ (Adv Comput Math 38(3):667---681, 2013) for solving monotone inclusion problems. Under strong monotonicity assumptions for some of the operators involved we obtain for the sequences of iterates that approach the solution orders of convergence of $$\mathcal{{O}}(\frac{1}{n})$$ O ( 1 n ) and $$\mathcal{{O}}(\omega ^n)$$ O ( ? n ) , for $$\omega \in (0,1)$$ ? ? ( 0 , 1 ) , respectively. The investigated primal-dual algorithms are fully decomposable, in the sense that the operators are processed individually at each iteration. We also discuss the modified algorithms in the context of convex optimization problems and present numerical experiments in image processing and pattern recognition in cluster analysis.

Convergence Analysis for a Primal-Dual Monotone + Skew Splitting Algorithm with Applications to Total Variation Minimization

In this paper we investigate the convergence behavior of a primal-dual splitting method for solving monotone inclusions involving mixtures of composite, Lipschitzian and parallel sum type operators proposed by Combettes and Pesquet (in Set-Valued Var. Anal. 20(2):307---330, 2012). Firstly, in the particular case of convex minimization problems, we derive convergence rates for the partial primal-dual gap function associated to a primal-dual pair of optimization problems by making use of conjugate duality techniques. Secondly, we propose for the general monotone inclusion problem two new schemes which accelerate the sequences of primal and/or dual iterates, provided strong monotonicity assumptions for some of the involved operators are fulfilled. Finally, we apply the theoretical achievements in the context of different types of image restoration problems solved via total variation regularization.

Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions

We provide two weakly convergent algorithms for finding a zero of the sum of a maximally monotone operator, a cocoercive operator, and the normal cone to a closed vector subspace of a real Hilbert space. The methods exploit the intrinsic structure of the problem by activating explicitly the cocoercive operator in the first step, and taking advantage of a vector space decomposition in the second step. The second step of the first method is a Douglas–Rachford iteration involving the maximally monotone operator and the normal cone. In the second method, it is a proximal step involving the partial inverse of the maximally monotone operator with respect to the vector subspace. Connections between the proposed methods and other methods in the literature are provided. Applications to monotone inclusions with finitely many maximally monotone operators and optimization problems are examined.

A Variable Metric Extension of the Forward–Backward–Forward Algorithm for Monotone Operators

We propose a variable metric extension of the forward–backward-forward algorithm for finding a zero of the sum of a maximally monotone operator and a monotone Lipschitzian operator in Hilbert spaces. In turn, this framework provides a variable metric splitting algorithm for solving monotone inclusions involving sums of composite operators. Monotone operator splitting methods recently proposed in the literature are recovered as special cases.

Global Convergence of a Closed-Loop Regularized Newton Method for Solving Monotone Inclusions in Hilbert Spaces

We analyze the global convergence properties of some variants of regularized continuous Newton methods for convex optimization and monotone inclusions in Hilbert spaces. The regularization term is of Levenberg–Marquardt type and acts in an open-loop or closed-loop form. In the open-loop case the regularization term may be of bounded variation.

Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators

We propose a primal-dual splitting algorithm for solving monotone inclusions involving a mixture of sums, linear compositions, and parallel sums of set-valued and Lipschitzian operators. An important feature of the algorithm is that the Lipschitzian operators present in the formulation can be processed individually via explicit steps, while the set-valued operators are processed individually via their resolvents. In addition, the algorithm is highly parallel in that most of its steps can be executed simultaneously. This work brings together and notably extends various types of structured monotone inclusion problems and their solution methods. The application to convex minimization problems is given special attention.

A Continuous Dynamical Newton-Like Approach to Solving Monotone Inclusions

We introduce nonautonomous continuous dynamical systems which are linked to the Newton and Levenberg–Marquardt methods. They aim at solving inclusions governed by maximal monotone operators in Hilbert spaces. Relying on the Minty representation of maximal monotone operators as lipschitzian manifolds, we show that these dynamics can be formulated as first-order in time differential systems, which are relevant to the Cauchy–Lipschitz theorem. By using Lyapunov methods, we prove that their trajectories converge weakly to equilibria. Time discretization of these dynamics gives algorithms providing new insight into Newton's method for solving monotone inclusions.

Iterative selection methods for common fixed point problems

Many problems encountered in applied mathematics can be recast as the problem of selecting a particular common fixed point of a countable family of quasi-nonexpansive operators in a Hilbert space. We propose two iterative methods to solve such problems. Our convergence analysis is shown to cover a variety of existing results in this area. Applications to solving monotone inclusion and equilibrium problems are considered.

Solving monotone inclusions via compositions of nonexpansive averaged operators

A unified fixed point theoretic framework is proposed to investigate the asymptotic behavior of algorithms for finding solutions to monotone inclusion problems. The basic iterative scheme under consideration involves nonstationary compositions of perturbed averaged nonexpansive operators. The analysis covers proximal methods for common zero problems as well as for various splitting methods for finding a zero of the sum of monotone operators.