Douglas-Rachford Algorithm Research Articles

Non-separable multidimensional multiresolution wavelets: A Douglas-Rachford approach

After re-casting the wavelet construction problem as a feasibility problem with constraints arising from the requirements of compact support, smoothness and orthogonality, the Douglas–Rachford algorithm is employed in the search for multi-dimensional, non-separable, compactly supported, smooth, orthogonal, multiresolution wavelets in the case of translations along the integer lattice and isotropic dyadic dilations. An algorithm for the numerical construction of such wavelets is described. By applying the algorithm, new one-dimensional wavelets are produced as well as genuinely non-separable two-dimensional wavelets.

Transformer based Douglas-Rachford unrolling network for compressed sensing

Compressed sensing (CS) with the binary sampling matrix is hardware-friendly and memory-saving in the signal processing field. Existing Convolutional Neural Network (CNN)-based CS methods show potential restrictions in exploiting non-local similarity and lack interpretability. In parallel, the emerging Transformer architecture performs well in modelling long-range correlations. To further improve the CS reconstruction quality from highly under-sampled CS measurements, a Transformer based deep unrolling reconstruction network abbreviated as DR-TransNet is proposed, whose design is inspired by the traditional iterative Douglas-Rachford algorithm. It combines the merits of structure insights of optimization-based methods and the speed of the network-based ones. Therein, a U-type Transformer based proximal sub-network is elaborated to reconstruct images in the wavelet domain and the spatial domain as an auxiliary mode, which aims to explore local informative details and global long-term interaction of the images. Specially, a flexible single model is trained to address the CS reconstruction with different binary CS sampling ratios. Compared with the state-of-the-art CS reconstruction methods with the binary sampling matrix, the proposed method can achieve appealing improvements in terms of Peak Signal to Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) and visual metrics. Codes are available at https://github.com/svyueming/DR-TransNet.

Weak and Strong Convergence of Split Douglas-Rachford Algorithms for Monotone Inclusions

We are concerned in this paper with the convergence analysis of the primal-dual splitting (PDS) and the split Douglas-Rachford (SDR) algorithms for monotone inclusions by using an operator-oriented approach. We shall show that both PDS and SDR algorithms can be driven by a (firmly) nonexpansive mapping in a product Hilbert space. We are then able to apply the Krasnoselskii-Mann and Halpern fixed point algorithms to PDS and SDR to get weakly and strongly convergent algorithms for finding solutions of the primal and dual monotone inclusions. Moreover, an additional projection technique is used to derive strong convergence of a modified SDR algorithm.

Optimal Control Duality and the Douglas–Rachford Algorithm

Optimal Control Duality and the Douglas–Rachford Algorithm

Douglas–Rachford algorithm for control-constrained minimum-energy control problems

Splitting and projection-type algorithms have been applied to many optimization problems due to their simplicity and efficiency, but the application of these algorithms to optimal control is less common. In this paper we utilize the Douglas–Rachford (DR) algorithm to solve control-constrained minimum-energy optimal control problems. Instead of the traditional approach where one discretizes the problem and solves it using large-scale finite-dimensional numerical optimization techniques we split the problem in two subproblems and use the DR algorithm to find an optimal point in the intersection of the solution sets of these two subproblems hence giving a solution to the original problem. We derive general expressions for the projections and propose a numerical approach. We obtain analytic closed-form expressions for the projectors of pure, under-, critically- and over-damped harmonic oscillators. We illustrate the working of our approach to solving not only these example problems but also a challenging machine tool manipulator problem. Through numerical case studies, we explore and propose desirable ranges of values of an algorithmic parameter which yield smaller number of iterations.

Douglas–Rachford algorithm for control- and state-constrained optimal control problems

<abstract><p>We consider the application of the Douglas–Rachford (DR) algorithm to solve linear-quadratic (LQ) control problems with box constraints on the state and control variables. We have split the constraints of the optimal control problem into two sets: one involving the ordinary differential equation with boundary conditions, which is affine, and the other, a box. We have rewritten the LQ control problems as the minimization of the sum of two convex functions. We have found the proximal mappings of these functions, which we then employ for the projections in the DR iterations. We propose a numerical algorithm for computing the projection onto the affine set. We present a conjecture for finding the costates and the state constraint multipliers of the optimal control problem, which can, in turn, be used to verify the optimality conditions. We conducted numerical experiments with two constrained optimal control problems to illustrate the performance and the efficiency of the DR algorithm in comparison with the traditional approach of direct discretization.</p></abstract>

Strict pseudocontractions and demicontractions, their properties, and applications

We give properties of strict pseudocontractions and demicontractions defined on a Hilbert space, which constitute wide classes of operators that arise in iterative methods for solving fixed point problems. In particular, we give necessary and sufficient conditions under which a convex combination and composition of strict pseudocontractions as well as demicontractions that share a common fixed point is again a strict pseudocontraction or a demicontraction, respectively. Moreover, we introduce a generalized relaxation of composition of demicontraction and give its properties. We“ apply these properties to prove the weak convergence of a class of algorithms that is wider than the Douglas–Rachford algorithm and projected Landweber algorithms. We have also presented two numerical examples, where we compare the behavior of the presented methods with the Douglas–Rachford method.

Tikhonov regularized iterative methods for nonlinear problems

ABSTRACT We consider the monotone inclusion problems in real Hilbert spaces. Proximal splitting algorithms are very popular technique to solve it and generally achieve weak convergence under mild assumptions. Researchers assume the strong conditions like strong convexity or strong monotonicity on the considered operators to prove strong convergence of the algorithms. Mann iteration method and normal S-iteration method are popular methods to solve fixed point problems. We propose a new common fixed point algorithm based on normal S-iteration method using Tikhonov regularization to find common fixed point of non-expansive operators and prove strong convergence of the generated sequence to the set of common fixed points without assuming strong convexity and strong monotonicity. Based on proposed fixed point algorithm, we propose a forward–backward-type algorithm and a Douglas–Rachford algorithm in connection with Tikhonov regularization to find the solution of monotone inclusion problems. Further, we consider the complexly structured monotone inclusion problems which are very popular these days. We also propose a strongly convergent forward–backward-type primal–dual algorithm and a Douglas–Rachford-type primal–dual algorithm to solve the monotone inclusion problems. Finally, we conduct a numerical experiment to solve image deblurring problems.

Strong Convergence for the Alternating Halpern–Mann Iteration in CAT(0) Spaces

In this paper we consider, in the general context of CAT(0) spaces, an iterative schema which alternates between Halpern and Krasnoselskii–Mann style iterations. We prove, under suitable conditions, the strong convergence of this algorithm, benefiting from ideas from the proof mining program. We give quantitative information in the form of effective rates of asymptotic regularity and of metastability (in the sense of Tao). Motivated by these results, we are also able to obtain strongly convergent versions of the forward-backward and the Douglas–Rachford algorithms. Our results generalize recent work by Boţ, Csetnek, and Meier and Cheval and Leuştean.

The Splitting Algorithms by Ryu, by Malitsky–Tam, and by Campoy Applied to Normal Cones of Linear Subspaces Converge Strongly to the Projection onto the Intersection

Finding a zero of a sum of maximally monotone operators is a fundamental problem in modern optimization and nonsmooth analysis. Assuming that the resolvents of the operators are available, this problem can be tackled with the Douglas–Rachford algorithm. However, when dealing with three or more operators, one must work in a product space with as many factors as there are operators. In groundbreaking recent work by Ryu and by Malitsky and Tam, it was shown that the number of factors can be reduced by one. A similar reduction was achieved recently by Campoy through a clever reformulation originally proposed by Kruger. All three splitting methods guarantee weak convergence to some solution of the underlying sum problem; strong convergence holds in the presence of uniform monotonicity. In this paper, we provide a case study when the operators involved are normal cone operators of subspaces and the solution set is thus the intersection of the subspaces. Even though these operators lack strict convexity, we show that striking conclusions are available in this case: strong (instead of weak) convergence and the solution obtained is (not arbitrary but) the projection onto the intersection. To illustrate our results, we also perform numerical experiments.

On the Douglas–Rachford Algorithm for Solving Possibly Inconsistent Optimization Problems

More than 40 years ago, Lions and Mercier introduced in a seminal paper the Douglas–Rachford algorithm. Today, this method is well-recognized as a classic and highly successful splitting method to find minimizers of the sum of two (not necessarily smooth) convex functions. Whereas the underlying theory has matured, one case remains a mystery: the behavior of the shadow sequence when the given functions have disjoint domains. Building on previous work, we establish for the first time weak and value convergence of the shadow sequence generated by the Douglas–Rachford algorithm in a setting of unprecedented generality. The weak limit point is shown to solve the associated normal problem, which is a minimal perturbation of the original optimization problem. We also present new results on the geometry of the minimal displacement vector. Funding: The research of H. H. Bauschke and W. M. Moursi was partially supported by Discovery Grants of the Natural Sciences and Engineering Research Council of Canada [Grants RGPIN-2018-03703 and RGPIN-2019-04803], respectively.

Unrestricted Douglas-Rachford algorithms for solving convex feasibility problems in Hilbert space

In this work we focus on the convex feasibility problem (CFP) in Hilbert space. A specific method in this area that has gained a lot of interest in recent years is the Douglas-Rachford (DR) algorithm. This algorithm was originally introduced in 1956 for solving stationary and non-stationary heat equations. Then in 1979, Lions and Mercier adjusted and extended the algorithm with the aim of solving CFPs and even more general problems, such as finding zeros of the sum of two maximally monotone operators. Many developments which implement various concepts concerning this algorithm have occurred during the last decade. We introduce an unrestricted DR algorithm, which provides a general framework for such concepts. Using unrestricted products of a finite number of strongly nonexpansive operators, we apply this framework to provide new iterative methods, where, inter alia, such operators may be interlaced between the operators used in the scheme of our unrestricted DR algorithm.

Approximate Douglas–Rachford algorithm for two-sets convex feasibility problems

In this paper, we propose a new algorithm combining the Douglas-Rachford (DR) algorithm and the Frank-Wolfe algorithm, also known as the conditional gradient (CondG) method, for solving the classic convex feasibility problem. Within the algorithm, which will be named {\it Approximate Douglas-Rachford (ApDR) algorithm}, the CondG method is used as a subroutine to compute feasible inexact projections on the sets under consideration, and the ApDR iteration is defined based on the DR iteration. The ApDR algorithm generates two sequences, the main sequence, based on the DR iteration, and its corresponding shadow sequence. When the intersection of the feasible sets is nonempty, the main sequence converges to a fixed point of the usual DR operator, and the shadow sequence converges to the solution set. We provide some numerical experiments to illustrate the behaviour of the sequences produced by the proposed algorithm.

Resolvent splitting for sums of monotone operators with minimal lifting

In this work, we study fixed point algorithms for finding a zero in the sum of \(n\ge 2\) maximally monotone operators by using their resolvents. More precisely, we consider the class of such algorithms where each resolvent is evaluated only once per iteration. For any algorithm from this class, we show that the underlying fixed point operator is necessarily defined on a d-fold Cartesian product space with \(d\ge n-1\). Further, we show that this bound is unimprovable by providing a family of examples for which \(d=n-1\) is attained. This family includes the Douglas–Rachford algorithm as the special case when \(n=2\). Applications of the new family of algorithms in distributed decentralised optimisation and multi-block extensions of the alternation direction method of multipliers (ADMM) are discussed.

Theoretical and numerical comparison of first order algorithms for cocoercive equations and smooth convex optimization

This paper provides a theoretical and numerical comparison of classical first-order splitting methods for solving smooth convex optimization problems and cocoercive equations. From a theoretical point of view, we compare convergence rates of gradient descent, forward-backward, Peaceman-Rachford, and Douglas-Rachford algorithms for minimizing the sum of two smooth convex functions when one of them is strongly convex. A similar comparison is given in the more general cocoercive setting under the presence of strong monotonicity and we observe that the convergence rates in optimization are strictly better than the corresponding rates for cocoercive equations for some algorithms. We obtain improved rates with respect to the literature in several instances by exploiting the structure of our problems. Moreover, we indicate which algorithm has the smallest convergence rate depending on strong convexity and cocoercive parameters. From a numerical point of view, we verify our theoretical results by implementing and comparing previous algorithms in well established signal and image inverse problems involving sparsity. We replace the widely used ℓ1 norm with the Huber loss and we observe that fully proximal-based strategies have numerical and theoretical advantages with respect to methods using gradient steps.

An Adaptive Alternating Direction Method of Multipliers

The alternating direction method of multipliers (ADMM) is a powerful splitting algorithm for linearly constrained convex optimization problems. In view of its popularity and applicability, a growing attention is drawn toward the ADMM in nonconvex settings. Recent studies of minimization problems for nonconvex functions include various combinations of assumptions on the objective function including, in particular, a Lipschitz gradient assumption. We consider the case where the objective is the sum of a strongly convex function and a weakly convex function. To this end, we present and study an adaptive version of the ADMM which incorporates generalized notions of convexity and penalty parameters adapted to the convexity constants of the functions. We prove convergence of the scheme under natural assumptions. To this end, we employ the recent adaptive Douglas–Rachford algorithm by revisiting the well-known duality relation between the classical ADMM and the Douglas–Rachford splitting algorithm, generalizing this connection to our setting. We illustrate our approach by relating and comparing to alternatives, and by numerical experiments on a signal denoising problem.

The Douglas-Rachford algorithm with new error sequences for an inclusion problem

The Douglas-Rachford algorithm with new error sequences for an inclusion problem

Convergence of an Inertial Shadow Douglas-Rachford Splitting Algorithm for Monotone Inclusions

An inertial shadow Douglas-Rachford splitting algorithm for finding zeros of the sum of monotone operators is proposed in Hilbert spaces. Moreover, a three-operator splitting algorithm for solving a class of monotone inclusion problems is also concerned. The weak convergence of the algorithms is investigated under mild assumptions. Some numerical experiments are implemented to illustrate our main convergence results.

A unified Douglas–Rachford algorithm for generalized DC programming

A unified Douglas–Rachford algorithm for generalized DC programming

Robust classification with feature selection using an application of the Douglas-Rachford splitting algorithm

This paper deals with supervised classification and feature selection with application in the context of high dimensional features. A classical approach leads to an optimization problem minimizing the within sum of squares in the clusters (I2 norm) with an I1 penalty in order to promote sparsity. It has been known for decades that I1 norm is more robust than I2 norm to outliers. In this paper, we deal with this issue using a new proximal splitting method for the minimization of a criterion using I2 norm both for the constraint and the loss function. Since the I1 criterion is only convex and not gradient Lipschitz, we advocate the use of a Douglas-Rachford minimization solution. We take advantage of the particular form of the cost and, using a change of variable, we provide a new efficient tailored primal Douglas-Rachford splitting algorithm which is very effective on high dimensional dataset. We also provide an efficient classifier in the projected space based on medoid modeling. Experiments on two biological datasets and a computer vision dataset show that our method significantly improves the results compared to those obtained using a quadratic loss function.