Primal-dual Hybrid Gradient Research Articles

Worst-case analysis of restarted primal-dual hybrid gradient on totally unimodular linear programs

We analyze restarted PDHG on totally unimodular linear programs. In particular, we show that restarted PDHG finds an ϵ-optimal solution in O(Hm12.5nnz(A)log⁡(Hm2/ϵ)) matrix-vector multiplies where m1 is the number of constraints, m2 the number of variables, nnz(A) is the number of nonzeros in the constraint matrix, H is the largest absolute coefficient in the right hand side or objective vector, and ϵ is the distance to optimality of the outputted solution.

Primal-dual hybrid gradient image denoising algorithm based on overlapping group sparsity and fractional-order total variation

This study introduces a non-convex fractional-order hyper-Laplacian variational model for Gaussian noise removal. It employs first the primal-dual hybrid gradient algorithm to solve problems involving overlapping group sparse structures. Additionally, this paper designs a new algorithm leveraging the framework of the Chambolle-Pock algorithm with convergence and aims to recover high-quality images. The model, integrating the overlapping group sparse structure of the hyper-Laplacian prior with the non-convex fractional-order total variation, exhibits superior performance in reducing the staircase effect and maintaining sharp edge contours. To further improve the performance of the algorithm, a semi-adaptive p(x) non-convex penalty weight assignment mechanism is designed by introducing the structure tensor, which according to the characteristics of each region of the image and the noise level. The effectiveness and superiority of the proposed algorithm in image denoising with simulation experiments and comparative analyses are fully verified.

On the geometry and refined rate of primal–dual hybrid gradient for linear programming

AbstractWe study the convergence behaviors of primal–dual hybrid gradient (PDHG) for solving linear programming (LP). PDHG is the base algorithm of a new general-purpose first-order method LP solver, PDLP, which aims to scale up LP by taking advantage of modern computing architectures. Despite its numerical success, the theoretical understanding of PDHG for LP is still very limited; the previous complexity result relies on the global Hoffman constant of the KKT system, which is known to be very loose and uninformative. In this work, we aim to develop a fundamental understanding of the convergence behaviors of PDHG for LP and to develop a refined complexity rate that does not rely on the global Hoffman constant. We show that there are two major stages of PDHG for LP: in Stage I, PDHG identifies active variables and the length of the first stage is driven by a certain quantity which measures how close the non-degeneracy part of the LP instance is to degeneracy; in Stage II, PDHG effectively solves a homogeneous linear inequality system, and the complexity of the second stage is driven by a well-behaved local sharpness constant of the system. This finding is closely related to the concept of partial smoothness in non-smooth optimization, and it is the first complexity result of finite time identification without the non-degeneracy assumption. An interesting implication of our results is that degeneracy itself does not slow down the convergence of PDHG for LP, but near-degeneracy does.

On dynamical system modeling of learned primal-dual with a linear operator K : stability and convergence properties

Learned Primal-Dual (LPD) is a deep learning based method for composite optimization problems that is based on unrolling/unfolding the primal-dual hybrid gradient algorithm. While achieving great successes in applications, the mathematical interpretation of LPD as a truncated iterative scheme is not necessarily sufficient to fully understand its properties. In this paper, we study the LPD with a general linear operator. We model the forward propagation of LPD as a system of difference equations and a system of differential equations in discrete- and continuous-time settings (for primal and dual variables/trajectories), which are named discrete-time LPD and continuous-time LPD, respectively. Forward analyses such as stabilities and the convergence of the state variables of the discrete-time LPD to the solution of continuous-time LPD are given. Moreover, we analyze the learning problems with/without regularization terms of both discrete-time and continuous-time LPD from the optimal control viewpoint. We prove convergence results of their optimal solutions with respect to the network state initialization and training data, showing in some sense the topological stability of the learning problems. We also establish convergence from the solution of the discrete-time LPD learning problem to that of the continuous-time LPD learning problem through a piecewise linear extension, under some appropriate assumptions on the space of learnable parameters. This study demonstrates theoretically the robustness of the LPD structure and the associated training process, and can induce some future research and applications.

Stochastic Primal–Dual Hybrid Gradient Algorithm with Adaptive Step Sizes

In this work, we propose a new primal–dual algorithm with adaptive step sizes. The stochastic primal–dual hybrid gradient (SPDHG) algorithm with constant step sizes has become widely applied in large-scale convex optimization across many scientific fields due to its scalability. While the product of the primal and dual step sizes is subject to an upper-bound in order to ensure convergence, the selection of the ratio of the step sizes is critical in applications. Up-to-now there is no systematic and successful way of selecting the primal and dual step sizes for SPDHG. In this work, we propose a general class of adaptive SPDHG (A-SPDHG) algorithms and prove their convergence under weak assumptions. We also propose concrete parameters-updating strategies which satisfy the assumptions of our theory and thereby lead to convergent algorithms. Numerical examples on computed tomography demonstrate the effectiveness of the proposed schemes.

Generalized variational framework with minimax optimization for parametric blind deconvolution

Blind deconvolution (BD), which aims to separate unknown convolved signals, is a fundamental problem in signal processing. Due to the ill-posedness and underdetermination of the convolution system, it is a challenging nonlinear inverse problem. This paper is devoted to the algorithmic studies of parametric BD, which is typically applied to recover images from ad hoc optical modalities. We propose a generalized variational framework for parametric BD with various priors and potential functions. By using the conjugate theory in convex analysis, the framework can be cast into a nonlinear saddle point problem. We employ the recent advances in minimax optimization to solve the parametric BD by the nonlinear primal-dual hybrid gradient method, with all subproblems admitting closed-form solutions. Numerical simulations on synthetic and real datasets demonstrate the compelling performance of the minimax optimization approach for solving parametric BD.

Infeasibility Detection with Primal-Dual Hybrid Gradient for Large-Scale Linear Programming

Infeasibility Detection with Primal-Dual Hybrid Gradient for Large-Scale Linear Programming

A first-order computational algorithm for reaction-diffusion type equations via primal-dual hybrid gradient method

We propose an easy-to-implement iterative method for resolving the implicit (or semi-implicit) schemes arising in solving reaction-diffusion (RD) type equations. We formulate the nonlinear time implicit scheme as a min-max saddle point problem and then apply the primal-dual hybrid gradient (PDHG) method. Suitable precondition matrices are applied to the PDHG method to accelerate the convergence of algorithms under different circumstances. Furthermore, our method is applicable to various discrete numerical schemes with high flexibility. From various numerical examples tested in this paper, the proposed method converges properly and can efficiently produce numerical solutions with sufficient accuracy.

A distributed primal-dual hybrid gradient algorithm for fair resource allocation

A distributed primal-dual hybrid gradient algorithm for fair resource allocation

Robust noise-power estimation via primal–dual algorithms with measurements from a subset of network nodes

Noise power or signal variance is an important performance parameter or indicator used for many applications in signal processing and wireless communications. This paper investigates power estimation of random noises on sensor networks by using measurements taken over a subset of nodes. In many circumstances, the power values are typically smooth on the network, that is, the powers measured by neighboring nodes tend to have similar values. Using this smoothness, the power-estimation problem is formulated as a regularized optimization problem involving minimization of a sum of non-negative log-likelihood and the total variation. To find its optimal solution, a primal–dual hybrid gradient (PDHG) algorithm is developed, which uses primal–dual updates to achieve good estimation results. To further improve the estimation performance, two variants of the PDHG algorithm are also proposed: one has faster convergence by using a variable step-size searching strategy, and the other is robust to outliers and can identify abnormal nodes by using an outlier-rejection scheme. Several numerical experiments confirm the efficiency of our proposed algorithms.

Quadratic error bound of the smoothed gap and the restarted averaged primal-dual hybrid gradient

Quadratic error bound of the smoothed gap and the restarted averaged primal-dual hybrid gradient

A partially inexact generalized primal-dual hybrid gradient method for saddle point problems with bilinear couplings

A partially inexact generalized primal-dual hybrid gradient method for saddle point problems with bilinear couplings

Time Efficient Offloading Optimization in Automotive Multi-Access Edge Computing Networks Using Mean-Field Games

Emerging connected vehicular services, such as intelligent driving and high-definition (HD) map, are gaining increasing interest with the fast development of multi-access edge computing (MEC). For most time-sensitive and computation-intensive vehicular services, the data offloading process significantly influences the capacity and performance of MEC, especially when the number of connected vehicles is enormous. In this work, we consider data offloading optimization for a large-scale automotive MEC network. The problem is challenging due to the large number of connected vehicles and the complicated interaction between vehicles and edge servers. To tackle the scalability problem, we reformulate the original offloading optimization problem into a Mean-Field-Game (MFG) problem by abstracting the interaction among the connected vehicles as a distribution over their state spaces of task sizes, known as the mean-field term. To solve the problem efficiently, we propose a G-prox Primal-Dual-Hybrid-Gradient (PDHG) algorithm that transforms the MFG problem into a saddle-point problem. Based on our developed MFG model and G-prox PDHG algorithm, we propose the first data offloading scheme whose computation time is independent of the number of connected vehicles in automotive MEC systems. Extensive evaluation results corroborate the superior performance of our proposed scheme compared with the state-of-the-art methods.

Two Convergent Primal–Dual Hybrid Gradient Type Methods for Convex Programming with Linear Constraints

Two Convergent Primal–Dual Hybrid Gradient Type Methods for Convex Programming with Linear Constraints

Understanding the Convergence of the Preconditioned PDHG Method: A View of Indefinite Proximal ADMM

The primal-dual hybrid gradient (PDHG) algorithm is popular in solving min-max problems which are being widely used in a variety of areas. To improve the applicability and efficiency of PDHG for different application scenarios, we focus on the preconditioned PDHG (PrePDHG) algorithm, which is a framework covering PDHG, alternating direction method of multipliers (ADMM), and other methods. We give the optimal convergence condition of PrePDHG in the sense that the key parameters in the condition can not be further improved, which fills the theoretical gap in the-state-of-art convergence results of PrePDHG, and obtain the ergodic and non-ergodic sublinear convergence rates of PrePDHG. The theoretical analysis is achieved by establishing the equivalence between PrePDHG and indefinite proximal ADMM. Besides, we discuss various choices of the proximal matrices in PrePDHG and derive some interesting results. For example, the convergence condition of diagonal PrePDHG is improved to be tight, the dual stepsize of the balanced augmented Lagrangian method can be enlarged to 4/3 from 1, and a balanced augmented Lagrangian method with symmetric Gauss-Seidel iterations is also explored. Numerical results on the matrix game, projection onto the Birkhoff polytope, earth mover’s distance, and CT reconstruction verify the effectiveness and superiority of PrePDHG.

Distributed Data Offloading in Ultra-Dense LEO Satellite Networks: A Stackelberg Mean-Field Game Approach

The low earth orbit (LEO) satellite network is regarded as a promising technology for delivering seamless service to remote areas such as rural areas. In this paper, we consider a dynamic data offloading problem in ultra-dense LEO satellites networks with large-scale ground users, where each ground user makes the distributed offloading decision based on its state information, the influence from other ground users, and the fees paid to satellites. To investigate the interaction problem between ground users and satellites, we formulate the problem as a Stackelberg game. Specifically, the satellites serve as the leaders, who decide the data service price at each time slot. The ground users, on the contrary, are the followers who decide the power control based on their states, the influence from other ground users and the fees paid to satellites. Since the influence is difficult to estimate due to the large number of ground users, we employ the mean field game algorithm to transform the influence from others and satellites into the mean field term, and reformulate the optimization problem as a Stackelberg mean field game (SMFG). Each ground user makes the data offloading decision by learning the future impact of the whole network distributively. For ground users, we solve the power control optimization problem by utilizing the G-prox primal-dual hybrid gradient (PDHG) algorithm, where the Fokker-Planck-Kolmogorov (FPK) equation is converted into a linear form via the Taylor expansion. For satellites, we address the service pricing optimization problem by using the adjoint algorithm. Finally, the numerical results demonstrate the effectiveness of the proposed algorithm.

A note on the linear convergence of generalized primal-dual hybrid gradient methods

A note on the linear convergence of generalized primal-dual hybrid gradient methods

A relaxed parameter condition for the primal-dual hybrid gradient method for saddle-point problem

<p style='text-indent:20px;'>The primal-dual hybrid gradient method and the primal-dual algorithm proposed by Chambolle and Pock are both efficient methods for solving saddle point problem. However, the convergence of both methods depends on some assumptions which can be too restrictive or impractical in real applications. In this paper, we propose a new parameter condition for the primal-dual hybrid gradient method. This improvement only requires either the primal or the dual objective function to be strongly convex. The relaxed parameter condition leads to convergence acceleration. Although counter-example shows that the PDHG method is not necessarily convergent with constant step size, it becomes convergent with our relaxed parameter condition. Preliminary experimental results show that PDHG method with our relaxed parameter condition is more efficient than several state-of-art methods.</p>

A numerical algorithm for inverse problem from partial boundary measurement arising from mean field game problem

In this work, we consider a novel inverse problem in mean-field games (MFGs). We aim to recover the MFG model parameters that govern the underlying interactions among the population based on a limited set of noisy partial observations of the population dynamics under the limited aperture. Due to its severe ill-posedness, obtaining a good quality reconstruction is very difficult. Nonetheless, it is vital to recover the model parameters stably and efficiently in order to uncover the underlying causes for population dynamics for practical needs. Our work focuses on the simultaneous recovery of running cost and interaction energy in the MFG equations from a finite number of boundary measurements of population profile and boundary movement. To achieve this goal, we formalize the inverse problem as a constrained optimization problem of a least squares residual functional under suitable norms. We then develop a fast and robust operator splitting algorithm to solve the optimization using techniques including harmonic extensions, three-operator splitting scheme, and primal-dual hybrid gradient method. Numerical experiments illustrate the effectiveness and robustness of the algorithm.

Faster first-order primal-dual methods for linear programming using restarts and sharpness

First-order primal-dual methods are appealing for their low memory overhead, fast iterations, and effective parallelization. However, they are often slow at finding high accuracy solutions, which creates a barrier to their use in traditional linear programming (LP) applications. This paper exploits the sharpness of primal-dual formulations of LP instances to achieve linear convergence using restarts in a general setting that applies to alternating direction method of multipliers (ADMM), primal-dual hybrid gradient method (PDHG) and extragradient method (EGM). In the special case of PDHG, without restarts we show an iteration count lower bound of \(\Omega (\kappa ^2 \log (1/\epsilon ))\), while with restarts we show an iteration count upper bound of \(O(\kappa \log (1/\epsilon ))\), where \(\kappa \) is a condition number and \(\epsilon \) is the desired accuracy. Moreover, the upper bound is optimal for a wide class of primal-dual methods, and applies to the strictly more general class of sharp primal-dual problems. We develop an adaptive restart scheme and verify that restarts significantly improve the ability of PDHG, EGM, and ADMM to find high accuracy solutions to LP problems.