Cocoercive Operator Research Articles

Fast Convex Optimization via Time Scale and Averaging of the Steepest Descent

In a Hilbert setting, we develop a gradient-based dynamic approach for fast solving convex optimization problems. By applying time scaling, averaging, and perturbation techniques to the continuous steepest descent (SD), we obtain high-resolution ordinary differential equations of the Nesterov and Ravine methods. These dynamics involve asymptotically vanishing viscous damping and Hessian-driven damping (either in explicit or implicit form). Mathematical analysis does not require developing a Lyapunov analysis for inertial systems. We simply exploit classical convergence results for SD and its external perturbation version, then use tools of differential and integral calculus, including Jensen’s inequality. The method is flexible, and by way of illustration, we show how it applies starting from other important dynamics in optimization. We consider the case in which the initial dynamic is the regularized Newton method, then the case in which the starting dynamic is the differential inclusion associated with a convex lower semicontinuous potential, and finally we show that the technique can be naturally extended to the case of a monotone cocoercive operator. Our approach leads to parallel algorithmic results, which we study in the case of fast gradient and proximal algorithms. Our averaging technique shows new links between the Nesterov and Ravine methods. Funding: The research of R.I. Boţ and D.-K. Nguyen was supported by the Austrian Science Fund (FWF), projects W 1260 and P 34922-N.

Outer reflected forward–backward splitting algorithm with inertial extrapolation step

ABSTRACT This paper studies an outer reflected forward–backward splitting algorithm with an inertial step to find a zero of the sum of three monotone operators composing the maximal monotone operator, Lipschitz monotone operator, and a cocoercive operator in real Hilbert spaces. One of the interesting features of the proposed method is that both the Lipschitz monotone operator and the cocoercive operator are computed explicitly each with one evaluation per iteration. We obtain weak and strong convergence results under some easy-to-verify assumptions. We also obtain a non-asymptotic O ( 1 / n ) convergence rate of our proposed algorithm in a non-ergodic sense. We finally give some numerical illustrations arising from compressed sensing and image processing and show that our proposed method is effective and competitive with other related methods in the literature.

Gauss-Seidel Type Iterative Algorithm for a Generalized System of Extended Non-linear Variational Inequalities

This study is focused on a new generalized system of extended non-linear variational inequalities (GSENVI, for short) involving 3k-distinct non-linear relaxed cocoercive operators. We give the equivalent formulation of GSENVI in a more convenient form by using auxiliary principal technique. Through the projection technique, we demonstrate that the non-linear projection equations are analogous to equivalent form of GSENVI. By the use of alternative fixed point formulations, we proposed the k-steps Gauss-Seidel type iterative algorithms to obtain an approximate solution of the GSENVI. Further, we discuss the convergence of proposed k-step Gauss Seidel type iterative algorithms. Several special cases of GSENVI are discussed for the reliability of our findings.

A Relaxed Inertial Method for Solving Monotone Inclusion Problems with Applications

We study a relaxed inertial forward–backward–half-forward splitting approach with variable step size to solve a monotone inclusion problem involving a maximal monotone operator, a cocoercive operator, and a monotone Lipschitz operator. The convergence of the sequence of iterations generated by the discretisations of a continuous-time dynamical system is established under suitable conditions. Given the challenges associated with computing the resolvent of the composite operator, the proposed method is employed to tackle the composite monotone inclusion problem. Additionally, a convergence analysis is conducted under certain conditions. To demonstrate the effectiveness of the algorithm, numerical experiments are performed on the image deblurring problem.

Forward-backward-half forward splitting algorithm with deviations

ABSTRACT In this paper, we present a forward-backward-half forward splitting algorithm with deviations for solving the structured monotone inclusion problem composed of a maximally monotone operator, a maximally monotone and Lipschitz continuous operator and a cocoercive operator. The weak and linear convergence of the proposed algorithms is established. The aim of introducing deviations is to improve the performance of the forward-backward-half forward splitting algorithm through suitable choices of the deviations. A numerical example is given to show that the two-step inertial variant of the forward-backward-half forward splitting algorithms with deviations outperforms the existing methods in Briceño-Arias and Davis [Forward-backward-half forward algorithm for solving monotone inclusions, SIAM J Optimiz, 2017;28(4):2839–2871. doi: 10.1137/17M1120099], Zong et al. [An accelerated forward-backward-half forward splitting algorithm for monotone inclusion with applications to image restoration, Optimization, 2022;73(2):401–428. doi: 10.1080/02331934.2022.2107926] and Fan et al. [Convergence of an inertial shadow Douglas–Rachford splitting algorithm for monotone inclusions, Numer Func Anal Optim, 2021;42(14):1627–1644. doi: 10.1080/01630563.2021.2001749].

Three-operator reflected forward-backward splitting algorithm with double inertial effects

In this paper, we propose a reflected forward-backward splitting algorithm with two different inertial extrapolations to find a zero of the sum of three monotone operators consisting of the maximal monotone operator, Lipschitz continuous monotone operator, and a cocoercive operator in real Hilbert spaces. One of the interesting features of the proposed algorithm is that both the Lipschitz continuous monotone operator and the cocoercive operator are computed explicitly each with one evaluation per iteration. We then obtain weak and strong convergence results under some mild conditions. We finally give a numerical illustration to show that our proposed algorithm is effective and competitive with other related algorithms in the literature.

Solving monotone inclusions involving the sum of three maximally monotone operators and a cocoercive operator with applications

Solving monotone inclusions involving the sum of three maximally monotone operators and a cocoercive operator with applications

Four-Operator Splitting via a Forward–Backward–Half-Forward Algorithm with Line Search

In this article, we provide a splitting method for solving monotone inclusions in a real Hilbert space involving four operators: a maximally monotone, a monotone-Lipschitzian, a cocoercive, and a monotone-continuous operator. The proposed method takes advantage of the intrinsic properties of each operator, generalizing the forward–backward–half-forward splitting and the Tseng’s algorithm with line search. At each iteration, our algorithm defines the step size by using a line search in which the monotone-Lipschitzian and the cocoercive operators need only one activation. We also derive a method for solving nonlinearly constrained composite convex optimization problems in real Hilbert spaces. Finally, we implement our algorithm in a nonlinearly constrained least-square problem and we compare its performance with available methods in the literature.

Forward-partial inverse-half-forward splitting algorithm for solving monotone inclusions

In this paper we provide a splitting algorithm for solving coupled monotone inclusions in a real Hilbert space involving the sum of a normal cone to a vector subspace, a maximally monotone, a monotone-Lipschitzian, and a cocoercive operator. The proposed method takes advantage of the intrinsic properties of each operator and generalizes the method of partial inverses and the forward-backward-half forward splitting, among other methods. At each iteration, our algorithm needs two computations of the Lipschitzian operator while the cocoercive operator is activated only once. By using product space techniques, we derive a method for solving a composite monotone primal-dual inclusions including linear operators and we apply it to solve constrained composite convex optimization problems. Finally, we apply our algorithm to a constrained total variation least-squares problem and we compare its performance with efficient methods in the literature.

Distributed forward-backward methods for ring networks

In this work, we propose and analyse forward-backward-type algorithms for finding a zero of the sum of finitely many monotone operators, which are not based on reduction to a two operator inclusion in the product space. Each iteration of the studied algorithms requires one resolvent evaluation per set-valued operator, one forward evaluation per cocoercive operator, and two forward evaluations per monotone operator. Unlike existing methods, the structure of the proposed algorithms are suitable for distributed, decentralised implementation in ring networks without needing global summation to enforce consensus between nodes.

An accelerated forward-backward-half forward splitting algorithm for monotone inclusion with applications to image restoration

ABSTRACT Inertial methods play a vital role in accelerating the convergence speed of optimization algorithms. We present an inertial forward-backward-half forward splitting algorithm, which mainly finds a zero of the sum of three operators, where two of them are cocoercive operator and monotone-Lipschitz continuous respectively. Meanwhile, the convergence analysis of the proposed algorithm is established under mild conditions. To overcome the difficulty in the calculation for the resolvent of the composite operator, relying on a primal-dual idea, we expand the proposed algorithm to solve the composite inclusion problem involving a linearly composed monotone operator. As an application, we make use of the obtained inertial algorithm to deal with a composite convex optimization problem. We also show extensive numerical experiments on the total variation-based image deblurring problem to demonstrate the efficiency of the proposed algorithm. Specifically, the proposed algorithm not only has a better quality of the deblurring image but also converges more rapidly than the original one.

Potential Function-Based Framework for Minimizing Gradients in Convex and Min-Max Optimization

Making the gradients small is a fundamental optimization problem that has eluded unifying and simple convergence arguments in first-order optimization, so far primarily reserved for other convergence criteria, such as reducing the optimality gap. In particular, while many different potential function-based frameworks covering broad classes of algorithms exist for optimality gap-based convergence guarantees, we are not aware of such general frameworks addressing the gradient norm guarantees. To fill this gap, we introduce a novel potential function-based framework to study the convergence of standard methods for making the gradients small in smooth convex optimization and convex-concave min-max optimization. Our framework is intuitive and provides a lens for viewing algorithms that makes the gradients small as being driven by a trade-off between reducing either the gradient norm or a certain notion of an optimality gap. On the lower bounds side, we discuss tightness of the obtained convergence results for the convex setup and provide a new lower bound for minimizing norm of cocoercive operators that allows us to argue about optimality of methods in the min-max setup.

Newton-Type Inertial Algorithms for Solving Monotone Equations Governed by Sums of Potential and Nonpotential Operators

In a Hilbert space setting, this paper is devoted to the study of a class of first-order algorithms which aim to solve structured monotone equations involving the sum of potential and nonpotential operators. Precisely, we are looking for the zeros of an operator $$A= \nabla f +B $$ , where $$\nabla f$$ is the gradient of a differentiable convex function f, and B is a nonpotential monotone and cocoercive operator. This study is based on the inertial autonomous dynamic previously studied by the authors, which involves dampings controlled respectively by the Hessian of f, and by a Newton-type correction term attached to B. These geometric dampings attenuate the oscillations which occur with the inertial methods with viscous damping. Temporal discretization of this dynamic provides fully splitted proximal-gradient algorithms. Their convergence properties are proven using Lyapunov analysis. These results open the door to the design of first-order accelerated algorithms in numerical optimization taking into account the specific properties of potential and nonpotential terms.

Second Order Splitting Dynamics with Vanishing Damping for Additively Structured Monotone Inclusions

In the framework of a real Hilbert space, we address the problem of finding the zeros of the sum of a maximally monotone operator A and a cocoercive operator B. We study the asymptotic behaviour of the trajectories generated by a second order equation with vanishing damping, attached to this problem, and governed by a time-dependent forward–backward-type operator. This is a splitting system, as it only requires forward evaluations of B and backward evaluations of A. A proper tuning of the system parameters ensures the weak convergence of the trajectories to the set of zeros of A+B\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$A + B$$\\end{document}, as well as fast convergence of the velocities towards zero. A particular case of our system allows to derive fast convergence rates for the problem of minimizing the sum of a proper, convex and lower semicontinuous function and a smooth and convex function with Lipschitz continuous gradient. We illustrate the theoretical outcomes by numerical experiments.

Distributed [formula omitted]-winners-take-all via multiple neural networks with inertia

This paper is dedicated to solving the k-winners-take-all problem with large-scale input signals in a distributed manner. According to the decomposition of global input signals, a novel dynamical system consisting of multiple coordinated neural networks is proposed for finding the k largest inputs. In the system, each neural network is designed to tackle its available partial inputs only for a local objective ki (ki≤k). Simultaneously, a consensus-based approach is adopted to coordinate multiple neural networks for achieving the global objective k. In addition, an inertial term is introduced in each neural network for regulating its transient behavior, which has the potential of accelerating the convergence. By developing a cocoercive operator, we theoretically prove that the multiple neural networks with inertial terms converge asymptotically/exponentially to the k-winners-take-all solution exactly from arbitrary initial states for whatever decomposition of inputs and objective. Furthermore, some extensions to distributed constrained k-winners-take-all are also investigated. Finally, simulation results are presented to substantiate the effectiveness of the proposed system as well as its superior performance over existing distributed networks.

$H(.,.,.,.)$-$varphi$-$eta$-cocoercive Operator with an Application to Variational Inclusions

In this work, we study generalized $ H(.,.,.,.)$-$varphi$-$eta-$ cocoercive operator to find the solution of variational like inclusion involving an infinite family of set-valued mappings in semi-inner product spaces via resolvent equation approach. Furthermore, we established an equivalence between the set-valued variational-like inclusion problem and fixed point problem by employing generalized resolvent operator technique involving generalized $H(.,.,.,.)$-$varphi$-$eta$-cocoercive operator. Using the equivalent formulation of set-valued variational-like inclusion problem and resolvent equation problem, an iterative algorithm is developed that approximate the uniqueness of solution of the resolvent equation problem.

Forcing strong convergence of a viscosity iteration for nonexpansive and cocoercive operators in a Hilbert space

Forcing strong convergence of a viscosity iteration for nonexpansive and cocoercive operators in a Hilbert space

Primal-Dual Splitting Algorithms for Solving Structured Monotone Inclusion with Applications

This work proposes two different primal-dual splitting algorithms for solving structured monotone inclusion containing a cocoercive operator and the parallel-sum of maximally monotone operators. In particular, the parallel-sum is symmetry. The proposed primal-dual splitting algorithms are derived from two approaches: One is the preconditioned forward–backward splitting algorithm, and the other is the forward–backward–half-forward splitting algorithm. Both algorithms have a simple calculation framework. In particular, the single-valued operators are processed via explicit steps, while the set-valued operators are computed by their resolvents. Numerical experiments on constrained image denoising problems are presented to show the performance of the proposed algorithms.

Asymptotic behavior of Newton-like inertial dynamics involving the sum of potential and nonpotential terms

In a Hilbert space mathcal{H}, we study a dynamic inertial Newton method which aims to solve additively structured monotone equations involving the sum of potential and nonpotential terms. Precisely, we are looking for the zeros of an operator A= nabla f +B , where ∇f is the gradient of a continuously differentiable convex function f and B is a nonpotential monotone and cocoercive operator. Besides a viscous friction term, the dynamic involves geometric damping terms which are controlled respectively by the Hessian of the potential f and by a Newton-type correction term attached to B. Based on a fixed point argument, we show the well-posedness of the Cauchy problem. Then we show the weak convergence as tto +infty of the generated trajectories towards the zeros of nabla f +B. The convergence analysis is based on the appropriate setting of the viscous and geometric damping parameters. The introduction of these geometric dampings makes it possible to control and attenuate the known oscillations for the viscous damping of inertial methods. Rewriting the second-order evolution equation as a first-order dynamical system enables us to extend the convergence analysis to nonsmooth convex potentials. These results open the door to the design of new first-order accelerated algorithms in optimization taking into account the specific properties of potential and nonpotential terms. The proofs and techniques are original and differ from the classical ones due to the presence of the nonpotential term.

An adaptive splitting algorithm for the sum of two generalized monotone operators and one cocoercive operator

Splitting algorithms for finding a zero of sum of operators often involve multiple steps which are referred to as forward or backward steps. Forward steps are the explicit use of the operators and backward steps involve the operators implicitly via their resolvents. In this paper, we study an adaptive splitting algorithm for finding a zero of the sum of three operators. We assume that two of the operators are generalized monotone and their resolvents are computable, while the other operator is cocoercive but its resolvent is missing or costly to compute. Our splitting algorithm adapts new parameters to the generalized monotonicity of the operators and, at the same time, combines appropriate forward and backward steps to guarantee convergence to a solution of the problem.