Convex-concave Saddle Point Problems Research Articles

Splitting methods for a class of non-potential mean field games

<p style='text-indent:20px;'>We extend the methods from [<xref ref-type="bibr" rid="b39">39</xref>, <xref ref-type="bibr" rid="b37">37</xref>] to a class of <i>non-potential</i> mean-field game (MFG) systems with mixed couplings. Up to now, splitting methods have been applied to <i>potential</i> MFG systems that can be cast as convex-concave saddle-point problems. Here, we show that a class of non-potential MFG can be cast as primal-dual pairs of monotone inclusions and solved via extensions of convex optimization algorithms such as the primal-dual hybrid gradient (PDHG) algorithm. A critical feature of our approach is in considering dual variables of nonlocal couplings in <i>Fourier</i> or <i>feature spaces</i>.</p>

A Primal-Dual Algorithm with Line Search for General Convex-Concave Saddle Point Problems

In this paper, we propose a primal-dual algorithm with a novel momentum term using the partial gradients of the coupling function that can be viewed as a generalization of the method proposed by Ch...

Non-Euclidean proximal methods for convex-concave saddle-point problems

Non-Euclidean proximal methods for convex-concave saddle-point problems

Accelerated Methods for Saddle-Point Problem

Recently, it has been shown how, on the basis of the usual accelerated gradient method for solving problems of smooth convex optimization, accelerated methods for more complex problems (with a structure) and problems that are solved using various local information about the behavior of a function (stochastic gradient, Hessian, etc.) can be obtained. The term “accelerated methods” here means, on the one hand, the presence of some unified and fairly general way of acceleration. On the other hand, this also means the optimality of the methods, which can often be proved rigorously. In the present work, an attempt is made to construct in the same way a theory of accelerated methods for solving smooth convex-concave saddle-point problems with a structure. The main result of this article is the obtainment of in some sense necessary and sufficient conditions under which the complexity of solving nonlinear convex-concave saddle-point problems with a structure in the number of calculations of the gradients of composites in direct variables is equal in order of magnitude to the complexity of solving bilinear problems with a structure.

Structure–texture image decomposition using a new non‐local TV‐Hilbert model

Combining the advantages of the non-local total variation (TV) and the Gabor function, a new Gabor function based non-local TV-Hilbert model is presented to separate the structure and texture components of the image. Computationally, by introducing the dual form of the non-local TV, the authors reformulate the non-local TV-Hilbert minimisation problem into a convex–concave saddle-point problem. In the aspect of solving algorithm, by transforming the Chambolle–Pock's first-order primal–dual algorithm into a different equivalent form. The authors propose a proximal-based primal–dual algorithm to solve the convex–concave saddle-point problem. At last, experimental results demonstrate that the proposed new model outperforms several existing state-of-the-art variational models.

Parameter selection and solution algorithm for TGV-based image restoration model

In this paper, image restoration problem is formulated to solve a total generalized variation (TGV)-based minimization problem. The minimization problem includes an unknown regularization parameter. A Morozov’s discrepancy principle-based method is used to choose a suitable regularization parameter. Computationally, by introducing two dual variables, the TGV-based image restoration problem is reformulated as a convex-concave saddle-point problem. Meanwhile, the Chambolle–Pock’s first-order primal–dual algorithm is transformed into a different equivalent form which can be seen as a proximal-based primal–dual algorithm. Then, the different equivalent form is used to solve the saddle-point problem. At last, compared with several existing state-of-the-art methods, experimental results demonstrate the performance of our proposed method.

Stochastic AUC Optimization Algorithms With Linear Convergence

Area under the ROC curve (AUC) is a standard metric that is used to measure classification performance for imbalanced class data. Developing stochastic learning algorithms that maximize AUC over accuracy is of practical interest. However, AUC maximization presents a challenge since the learning objective function is defined over a pair of instances of opposite classes. Existing methods circumvent this issue but with high space and time complexity. From our previous work of redefining AUC optimization as a convex-concave saddle point problem, we propose a new stochastic batch learning algorithm for AUC maximization. The key difference from our previous work is that we assume that the underlying distribution of the data is uniform, and we develop a batch learning algorithm that is a stochastic primal-dual algorithm (SPDAM) that achieves a linear convergence rate. We establish the theoretical convergence of SPDAM with high probability and demonstrate its effectiveness on standard benchmark datasets.

Distributionally robust optimization with correlated data from vector autoregressive processes

We present a distributionally robust formulation of a stochastic optimization problem for non-i.i.d vector autoregressive data. We use the Wasserstein distance to define robustness in the space of distributions and we show, using duality theory, that the problem is equivalent to a finite convex–concave saddle point problem. The performance of the method is demonstrated on both synthetic and real data.

Primal–Dual Algorithms for Convex Optimization via Regret Minimization

We consider a large-scale convex program with functional constraints, where interior point methods are intractable due to the problem size. The effective solution techniques for these problems permit only simple operations at each iteration, and thus are based on primal–dual first-order methods such as the Arrow–Hurwicz–Uzawa subgradient method, which utilize only gradient computations and projections at each iteration. Such primal–dual algorithms admit the interpretation of solving the associated saddle point problem arising from the Lagrange dual. We revisit these methods through the lens of regret minimization from online learning and present a flexible framework. While it is well known that two regret-minimizing algorithms can be used to solve a convex–concave saddle point problem at the standard rate of $O$ ( $1/\sqrt {T}$ ), our framework for primal–dual algorithms allows us to exploit structural properties such as smoothness and/or strong convexity and achieve better convergence rates in favorable cases. In particular, for non-smooth problems with strongly convex objectives, our primal–dual framework equipped with an appropriate modification of Nesterov’s dual averaging algorithm achieves $O$ ( $1/T$ ) convergence rate.

Self-concordant inclusions: a unified framework for path-following generalized Newton-type algorithms

We study a class of monotone inclusions called “self-concordant inclusion” which covers three fundamental convex optimization formulations as special cases. We develop a new generalized Newton-type framework to solve this inclusion. Our framework subsumes three schemes: full-step, damped-step, and path-following methods as specific instances, while allows one to use inexact computation to form generalized Newton directions. We prove the local quadratic convergence of both full-step and damped-step algorithms. Then, we propose a new two-phase inexact path-following scheme for solving this monotone inclusion which possesses an $${\mathcal {O}}(\sqrt{\nu }\log (1/\varepsilon ))$$-worst-case iteration-complexity to achieve an $$\varepsilon $$-solution, where $$\nu $$ is the barrier parameter and $$\varepsilon $$ is a desired accuracy. As byproducts, we customize our scheme to solve three convex problems: the convex–concave saddle-point problem, the nonsmooth constrained convex program, and the nonsmooth convex program with linear constraints. We also provide three numerical examples to illustrate our theory and compare with existing methods.

Distributed Saddle-Point Subgradient Algorithms With Laplacian Averaging

We present distributed subgradient methods for min-max problems with agreement constraints on a subset of the arguments of both the convex and concave parts. Applications include constrained minimization problems where each constraint is a sum of convex functions in the local variables of the agents. In the latter case, the proposed algorithm reduces to primal-dual updates using local subgradients and Laplacian averaging on local copies of the multipliers associated to the global constraints. For the case of general convex-concave saddle-point problems, our analysis establishes the convergence of the running time-averages of the local estimates to a saddle point under periodic connectivity of the communication digraphs. Specifically, choosing the gradient step-sizes in a suitable way, we show that the evaluation error is proportional to $1/\sqrt{t}$ , where $t$ is the iteration step. We illustrate our results in simulation for an optimization scenario with nonlinear constraints coupling the decisions of agents that cannot communicate directly.

A globally linearly convergent method for pointwise quadratically supportable convex–concave saddle point problems

We study the Proximal Alternating Predictor–Corrector (PAPC) algorithm introduced recently by Drori, Sabach and Teboulle [8] to solve nonsmooth structured convex–concave saddle point problems consisting of the sum of a smooth convex function, a finite collection of nonsmooth convex functions and bilinear terms. We introduce the notion of pointwise quadratic supportability, which is a relaxation of a standard strong convexity assumption and allows us to show that the primal sequence is R-linearly convergent to an optimal solution and the primal-dual sequence is globally Q-linearly convergent. We illustrate the proposed method on total variation denoising problems and on locally adaptive estimation in signal/image deconvolution and denoising with multiresolution statistical constraints.

An accelerated non-Euclidean hybrid proximal extragradient-type algorithm for convex–concave saddle-point problems

This paper describes an accelerated HPE-type method based on general Bregman distances for solving convex–concave saddle-point (SP) problems. The algorithm is a special instance of a non-Euclidean hybrid proximal extragradient framework introduced by Svaiter and Solodov [An inexact hybrid generalized proximal point algorithm and some new results on the theory of Bregman functions, Math. Oper. Res. 25(2) (2000), pp. 214–230] where the prox sub-inclusions are solved using an accelerated gradient method. It generalizes the accelerated HPE algorithm presented in He and Monteiro [An accelerated HPE-type algorithm for a class of composite convex–concave saddle-point problems, SIAM J. Optim. 26 (2016), pp. 29–56] in two ways, namely: (a) it deals with general monotone SP problems instead of bilinear structured SPs and (b) it is based on general Bregman distances instead of the Euclidean one. Similar to the algorithm of He and Monteiro [An accelerated HPE-type algorithm for a class of composite convex–concave saddle-point problems, SIAM J. Optim. 26 (2016), pp. 29–56], it has the advantage that it works for any constant choice of proximal stepsize. Moreover, a suitable choice of the stepsize yields a method with the best known iteration-complexity for solving monotone SP problems. Computational results show that the new method is superior to Nesterov's [Smooth minimization of non-smooth functions, Math. Program. 103(1) (2005), pp. 127–152] smoothing scheme.

Stochastic Parallel Block Coordinate Descent for Large-Scale Saddle Point Problems

We consider convex-concave saddle point problems with a separable structure and non-strongly convex functions. We propose an efficient stochastic block coordinate descent method using adaptive primal-dual updates, which enables flexible parallel optimization for large-scale problems. Our method shares the efficiency and flexibility of block coordinate descent methods with the simplicity of primal-dual methods and utilizing the structure of the separable convex-concave saddle point problem. It is capable of solving a wide range of machine learning applications, including robust principal component analysis, Lasso, and feature selection by group Lasso, etc. Theoretically and empirically, we demonstrate significantly better performance than state-of-the-art methods in all these applications.

An Accelerated HPE-Type Algorithm for a Class of Composite Convex-Concave Saddle-Point Problems

This paper proposes a new algorithm for solving a class of composite convex-concave saddle-point problems. The new algorithm is a special instance of the hybrid proximal extragradient framework in which a Nesterov accelerated variant is used to approximately solve the prox subproblems. One of the advantages of the new method is that it works for any constant choice of proximal stepsize. Moreover, a suitable choice of the latter stepsize yields a method with the best known (accelerated inner) iteration complexity for the aforementioned class of saddle-point problems. In contrast to the smoothing technique of [Y. Nesterov, Math. Program., 103 (2005), pp. 127--152], our accelerated method does not assume that a feasible set is bounded due to its proximal point nature. Experiment results on three problem sets show that the new method outperforms Nesterov's smoothing technique of [Y. Nesterov, Math. Program., 103 (2005), pp. 127--152].

An approach toward fast gradient-based image segmentation.

In this paper, we present and investigate an approach to fast multilabel color image segmentation using convex optimization techniques. The presented model is in some ways related to the well-known Mumford-Shah model, but deviates in certain important aspects. The optimization problem has been designed with two goals in mind. The objective function should represent fundamental concepts of image segmentation, such as incorporation of weighted curve length and variation of intensity in the segmented regions, while allowing transformation into a convex concave saddle point problem that is computationally inexpensive to solve. This paper introduces such a model, the nontrivial transformation of this model into a convex-concave saddle point problem, and the numerical treatment of the problem. We evaluate our approach by applying our algorithm to various images and show that our results are competitive in terms of quality at unprecedentedly low computation times. Our algorithm allows high-quality segmentation of megapixel images in a few seconds and achieves interactive performance for low resolution images.

A simple algorithm for a class of nonsmooth convex–concave saddle-point problems

We introduce a novel algorithm for solving a class of structured nonsmooth convex–concave saddle-point problems involving a smooth function and a sum of finitely many bilinear terms and nonsmooth functions. The proposed method is simple and proven to globally converge to a saddle-point with an O(1/ε) efficiency estimate. We demonstrate its usefulness for tackling a broad class of minimization models with a finitely sum of composite nonsmooth functions.

Preconditioned Douglas--Rachford Splitting Methods for Convex-concave Saddle-point Problems

We propose a preconditioned version of the Douglas--Rachford splitting method for solving convex-concave saddle-point problems associated with Fenchel--Rockafellar duality. Our approach makes it possible to use approximate solvers for the linear subproblem arising in this context. We prove weak convergence in Hilbert space under minimal assumptions. In particular, various efficient preconditioners are introduced in this framework for which only a few inner iterations are needed instead of computing an exact solution or controlling the error. The method is applied to a discrete total-variation denoising problem. Numerical experiments show that the proposed algorithms with appropriate preconditioners are very competitive with existing fast algorithms including the first-order primal-dual algorithm for saddle-point problems of Chambolle and Pock.

An Inertial Forward-Backward Algorithm for Monotone Inclusions

In this paper, we propose an inertial forward backward splitting algorithm to compute a zero of the sum of two monotone operators, with one of the two operators being co-coercive. The algorithm is inspired by the accelerated gradient method of Nesterov, but can be applied to a much larger class of problems including convex-concave saddle point problems and general monotone inclusions. We prove convergence of the algorithm in a Hilbert space setting and show that several recently proposed first-order methods can be obtained as special cases of the general algorithm. Numerical results show that the proposed algorithm converges faster than existing methods, while keeping the computational cost of each iteration basically unchanged.

Imaging with Kantorovich--Rubinstein Discrepancy

We propose the use of the Kantorovich-Rubinstein norm from optimal transport in imaging problems. In particular, we discuss a variational regularisation model endowed with a Kantorovich-Rubinstein discrepancy term and total variation regularization in the context of image denoising and cartoon-texture decomposition. We point out connections of this approach to several other recently proposed methods such as total generalized variation and norms capturing oscillating patterns. We also show that the respective optimization problem can be turned into a convex-concave saddle point problem with simple constraints and hence, can be solved by standard tools. Numerical examples exhibit interesting features and favourable performance for denoising and cartoon-texture decomposition.