Douglas-Rachford Splitting Method Research Articles

Customized Douglas-Rachford splitting methods for structured inverse variational inequality problems

Recently, structured inverse variational inequality (SIVI) problems have attracted much attention. In this paper, we propose new splitting methods to solve SIVI problems by employing the idea of the classical Douglas-Rachford splitting method (DRSM). In particular, the proposed methods can be regarded as a novel application of the DRSM to SIVI problems by decoupling the linear equality constraint, leading to smaller and easier subproblems. The main computational tasks per iteration are the evaluations of certain resolvent operators, which are much cheaper than those methods without taking advantage of the problem structures. To make the methods more implementable in the general cases where the resolvent operator is evaluated in an iterative scheme, we further propose to solve the subproblems in an approximate manner. Under quite mild conditions, global convergence, sublinear rate of convergence, and linear rate of convergence results are established for both the exact and the inexact methods. Finally, we present preliminary numerical results to illustrate the performance of the proposed methods.

Distributed GNE Seeking under Global-Decision and Partial-Decision Information over Douglas-Rachford Splitting Method

This paper develops an algorithm for solving the generalized Nash equilibrium problem (GNEP) in non-cooperative games. The problem involves a set of players, each with a cost function that depends on their own decision as well as the decisions of other players. The goal is to find a decision vector that minimizes the cost for each player. Unlike most of the existing algorithms for GNEP, which require full information exchange among all players, this paper considers a more realistic scenario where players can only communicate with a subset of players through a connectivity graph. The proposed algorithm enables each player to estimate the decisions of other players and update their own and others’ estimates through local communication with their neighbors. By introducing a network Lagrangian function and applying the Douglas-Rachford splitting method (DR), the GNEP is reformulated as a zero-finding problem. It is shown that the DR method can find the generalized Nash equilibrium (GNE) of the original problem under some mild conditions.

A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

In this work, we propose solving the Information Bottleneck (IB) and Privacy Funnel (PF) problems with Douglas-Rachford Splitting methods (DRS). We study a general Markovian information-theoretic Lagrangian that includes IB and PF into a unified framework. We prove the linear convergence of the proposed solvers using the Kurdyka-Łojasiewicz inequality. Moreover, our analysis is beyond IB and PF and applies to any convex-weakly convex pair objectives. Based on the results, we develop two types of linearly convergent IB solvers, with one improves the performance of convergence over existing solvers while the other can be independent to the relevance-compression trade-off. Moreover, our results apply to PF, yielding a new class of linearly convergent PF solvers. Empirically, the proposed IB solvers IB obtain solutions that are comparable to the Blahut-Arimoto-based benchmark and is convergent for a wider range of the penalty coefficients than existing solvers. For PF, our non-greedy solvers can characterize the privacy-utility trade-off better than the clustering-based greedy solvers.

Convergence analysis of the generalized Douglas-Rachford splitting method under Hölder subregularity assumptions

<p style='text-indent:20px;'>In solving convex optimization and monotone inclusion problems, operator-splitting methods are often employed to transform optimization and inclusion problems into fixed-point equations as the equations obtained from operator-splitting methods are often easy to be solved by standard techniques. For the inclusion problem involving two maximally monotone operators, under the Hölder metric subregularity of the concerned operator, which is weaker than the strong monotonicity of the operator, we derive relationships between the convergence rate of the generalized Douglas-Rachford splitting algorithm and the order of the Hölder metric subregularity of the concerned operator. Moreover, for general multifunctions in Hilbert spaces, by proximal coderivative, we provided some dual sufficient conditions for Hölder metric subregularity.</p>

Exact and inexact Douglas–Rachford splitting methods for solving large-scale sparse absolute value equations

Abstract Exact and inexact Douglas–Rachford splitting methods are developed to solve the large-scale sparse absolute value equation (AVE) $Ax - |x| =b$, where $A\in \mathbb {R}^{n\times n}$ and $b\in \mathbb {R}^n$. The inexact method adopts a relative error tolerance and, therefore, in the inner iterative processes, the LSQR method is employed to find a qualified approximate solution of each subproblem, resulting in a lower cost for each iteration. When $\|A^{-1}\|\le 1$ and the solution set of the AVE is nonempty, the algorithms are globally and linearly convergent. When $\|A^{-1}\|= 1$ and the solution set of the AVE is empty, the sequence generated by the exact algorithm diverges to infinity on a trivial example. Numerical examples are presented to demonstrate the viability and robustness of the proposed methods.

A Dynamical Splitting Method for Minimizing the Sum of Three Convex Functions

In this paper, we develop a dynamical splitting method based on a nonlinear dynamical system with two proximal operators for solving the minimum value problem of the sum of three convex functions. The method endows some existing classical splitting methods such as the Forward-Backward splitting method and the Douglas-Rachford splitting method with continuous time behaviors. We demonstrate that the proposed dynamical system converges globally to a point whose image, imaged by each of the two proximal operators, is a solution to the minimum value problem. We also shows that both the residual of the two proximal operators and the objective error enjoy the convergence rates of where t denotes time. This rate can be improved to by using an ergodic average technique. Moreover, under certain smoothness and metric subregularity conditions, we further establish the global exponential convergence of the dynamical splitting method. Finally, we apply the dynamical splitting method to deal with the problem of compressive sensing.

Convergence of Halpern’s Iteration Method with Applications in Optimization

Halpern’s iteration method, discovered by Halpern in 1967, is an iterative algorithm for finding fixed points of a nonexpansive mapping in Hilbert and Banach spaces. Since many optimization problems can be cast into fixed point problems of nonexpansive mappings, Halpern’s method plays an important role in optimization methods. This paper discusses recent advances in convergence and rate of convergence results of Halpern’s method, and applications in optimization problems, including variational inequalities, monotone inclusions, Douglas-Rachford splitting method, and minimax problems.

The fun is finite: Douglas-Rachford and Sudoku puzzle – Finite termination and local linear convergence

In recent years, the Douglas-Rachford splitting method has been shown to be effective at solving many non-convex optimization problems. In this paper we present a local convergence analysis for non-convex feasibility problems and show that both finite termination and local linear convergence are obtained. For a generalization of the Sudoku puzzle, we prove that the local linear rate of convergence of Douglas-Rachford is exactly $\frac{\sqrt{5}}{5}$ and independent of puzzle size. For the $s$-queens problem we prove that Douglas-Rachford converges after a finite number of iterations. Numerical results on solving Sudoku puzzles and $s$-queens puzzles are provided to support our theoretical findings.

Relative-error inertial-relaxed inexact versions of Douglas-Rachford and ADMM splitting algorithms

This paper derives new inexact variants of the Douglas-Rachford splitting method for maximal monotone operators and the alternating direction method of multipliers (ADMM) for convex optimization. The analysis is based on a new inexact version of the proximal point algorithm that includes both an inertial step and overrelaxation. We apply our new inexact ADMM method to LASSO and logistic regression problems and obtain somewhat better computational performance than earlier inexact ADMM methods.

Analysis of Fully Preconditioned Alternating Direction Method of Multipliers with Relaxation in Hilbert Spaces

Alternating direction method of multipliers is a powerful first-order method for non-smooth optimization problems including various applications in inverse problems and imaging. However, there is no clear result on the weak convergence of alternating direction method of multipliers in infinite-dimensional Hilbert spaces with relaxation. In this paper, by employing a kind of partial gap analysis, we prove the weak convergence of a general preconditioned and relaxed version in infinite-dimensional Hilbert spaces, with preconditioning for solving all the involved implicit equations under mild conditions. We also give the corresponding ergodic convergence rates respecting to the partial gap function. Furthermore, the connections between certain preconditioned and relaxed alternating direction method of multipliers and the corresponding Douglas–Rachford splitting methods are discussed. Numerical tests show the efficiency of the proposed overrelaxation variants with preconditioning.

A random block-coordinate Douglas–Rachford splitting method with low computational complexity for binary logistic regression

In this paper, we propose a new optimization algorithm for sparse logistic regression based on a stochastic version of the Douglas-Rachford splitting method. Our algorithm sweeps the training set by randomly selecting a mini-batch of data at each iteration, and it allows us to update the variables in a block coordinate manner. Our approach leverages the proximity operator of the logistic loss, which is expressed with the generalized Lambert W function. Experiments carried out on standard datasets demonstrate the efficiency of our approach w.r.t. stochastic gradient-like methods.

Iteration complexity of an inexact Douglas-Rachford method and of a Douglas-Rachford-Tseng’s F-B four-operator splitting method for solving monotone inclusions

In this paper, we propose and study the iteration complexity of an inexact Douglas-Rachford splitting (DRS) method and a Douglas-Rachford-Tseng's forward-backward (F-B) splitting method for solving two-operator and four-operator monotone inclusions, respectively. The former method (although based on a slightly different mechanism of iteration) is motivated by the recent work of J. Eckstein and W. Yao, in which an inexact DRS method is derived from a special instance of the hybrid proximal extragradient (HPE) method of Solodov and Svaiter, while the latter one combines the proposed inexact DRS method (used as an outer iteration) with a Tseng's F-B splitting type method (used as an inner iteration) for solving the corresponding subproblems. We prove iteration complexity bounds for both algorithms in the pointwise (non-ergodic) as well as in the ergodic sense by showing that they admit two different iterations: one that can be embedded into the HPE method, for which the iteration complexity is known since the work of Monteiro and Svaiter, and another one which demands a separate analysis. Finally, we perform simple numerical experiments %on three-operator and four-operator monotone inclusions to show the performance of the proposed methods when compared with other existing algorithms.

A note on the Douglas–Rachford splitting method for optimization problems involving hypoconvex functions

Recently, the convergence of the Douglas–Rachford splitting method (DRSM) was established for minimizing the sum of a nonsmooth strongly convex function and a nonsmooth hypoconvex function under the assumption that the strong convexity constant $$\beta $$ is larger than the hypoconvexity constant $$\omega $$ . Such an assumption, implying the strong convexity of the objective function, precludes many interesting applications. In this paper, we prove the convergence of the DRSM for the case $$\beta =\omega $$ , under relatively mild assumptions compared with some existing work in the literature.

Convergence Analysis of Douglas--Rachford Splitting Method for “Strongly + Weakly” Convex Programming

We consider the convergence of the Douglas--Rachford splitting method (DRSM) for minimizing the sum of a strongly convex function and a weakly convex function; this setting has various applications, especially in some sparsity-driven scenarios with the purpose of avoiding biased estimates which usually occur when convex penalties are used. Though the convergence of the DRSM has been well studied for the case where both functions are convex, its results for some nonconvex-function-involved cases, including the “strongly + weakly” convex case, are still in their infancy. In this paper, we prove the convergence of the DRSM for the “strongly + weakly” convex setting under relatively mild assumptions compared with some existing work in the literature. Moreover, we establish the rate of asymptotic regularity and the local linear convergence rate in the asymptotical sense under some regularity conditions. (A corrected version is attached.)

Optimal Rates of Linear Convergence of Relaxed Alternating Projections and Generalized Douglas-Rachford Methods for Two Subspaces

We systematically study the optimal linear convergence rates for several relaxed alternating projection methods and the generalized Douglas-Rachford splitting methods for finding the projection on the intersection of two subspaces. Our analysis is based on a study on the linear convergence rates of the powers of matrices. We show that the optimal linear convergence rate of powers of matrices is attained if and only if all subdominant eigenvalues of the matrix are semisimple. For the convenience of computation, a nonlinear approach to the partially relaxed alternating projection method with at least the same optimal convergence rate is also provided. Numerical experiments validate our convergence analysis

A distributed Douglas-Rachford splitting method for multi-block convex minimization problems

A customized Douglas-Rachford splitting method (DRSM) was recently proposed to solve two-block separable convex optimization problems with linear constraints and simple abstract constraints. The algorithm has advantage over the well-known alternating direction method of multipliers (ADMM), the dual application of DRSM to the two-block convex minimization problem, in the sense that the subproblems can have larger opportunity of possessing closed-form solutions since they are unconstrained. In this paper, we further study along this way by considering the primal application of DRSM for the general case m?3, i.e., we consider the multi-block separable convex minimization problem with linear constraints where the objective function is separable into m individual convex functions without coupled variables. The resulting method fully exploits the separable structure and enjoys decoupled subproblems which can be solved simultaneously. Both the exact and inexact versions of the new method are presented in a unified framework. Under mild conditions, we manage to prove the global convergence of the algorithm. Preliminary numerical experiments for extracting the background from corrupted surveillance video verify the encouraging efficiency of the new algorithm.

Proximity point algorithm for low-rank matrix recovery from sparse noise corrupted data

The method of recovering a low-rank matrix with an unknown fraction whose entries are arbitrarily corrupted is known as the robust principal component analysis (RPCA). This RPCA problem, under some conditions, can be exactly solved via convex optimization by minimizing a combination of the nuclear norm and the l1 norm. In this paper, an algorithm based on the Douglas-Rachford splitting method is proposed for solving the RPCA problem. First, the convex optimization problem is solved by canceling the constraint of the variables, and then the proximity operators of the objective function are computed alternately. The new algorithm can exactly recover the low-rank and sparse components simultaneously, and it is proved to be convergent. Numerical simulations demonstrate the practical utility of the proposed algorithm.

A STRICTLY CONTRACTIVE PEACEMAN-RACHFORD SPLITTING METHOD FOR CONVEX PROGRAMMING.

In this paper, we focus on the application of the Peaceman-Rachford splitting method (PRSM) to a convex minimization model with linear constraints and a separable objective function. Compared to the Douglas-Rachford splitting method (DRSM), another splitting method from which the alternating direction method of multipliers originates, PRSM requires more restrictive assumptions to ensure its convergence, while it is always faster whenever it is convergent. We first illustrate that the reason for this difference is that the iterative sequence generated by DRSM is strictly contractive, while that generated by PRSM is only contractive with respect to the solution set of the model. With only the convexity assumption on the objective function of the model under consideration, the convergence of PRSM is not guaranteed. But for this case, we show that the first t iterations of PRSM still enable us to find an approximate solution with an accuracy of O(1/t). A worst-case O(1/t) convergence rate of PRSM in the ergodic sense is thus established under mild assumptions. After that, we suggest attaching an underdetermined relaxation factor with PRSM to guarantee the strict contraction of its iterative sequence and thus propose a strictly contractive PRSM. A worst-case O(1/t) convergence rate of this strictly contractive PRSM in a nonergodic sense is established. We show the numerical efficiency of the strictly contractive PRSM by some applications in statistical learning and image processing.

A customized Douglas–Rachford splitting algorithm for separable convex minimization with linear constraints

We consider applying the Douglas---Rachford splitting method (DRSM) to the convex minimization problem with linear constraints and a separable objective function. The dual application of DRSM has been well studied in the literature, resulting in the well known alternating direction method of multipliers (ADMM). In this paper, we show that the primal application of DRSM in combination with an appropriate decomposition can yield an efficient structure-exploiting algorithm for the model under consideration, whose subproblems could be easier than those of ADMM. Both the exact and inexact versions of this customized DRSM are studied; and their numerical efficiency is demonstrated by some preliminary numerical results.

Recovering low‐rank and sparse components of matrices for object detection

It is shown that object detection can be addressed in the authors' unified framework, where the observed video matrix is decomposed into the low-rank matrix and the sparse matrix. The recovering problem can be solved by the proposed variant of the Douglas-Rachford splitting method, which accomplishes recovery by exploiting the separable structure property of the model. The effectiveness of the proposed object detection scheme is illustrated on two data: simulated data and real sequences applications. The numerical experiments verify that the proposed algorithm has attractive robustness and high accuracy for illumination variation and dynamic texture.