Linear Convergence Research Articles

Faster augmented Lagrangian method with inertial steps for solving convex optimization problems with linear constraints

A faster augmented Lagrangian method (Faster ALM) with constant inertial parameters for solving convex optimization problems with linear equality constraint is presented in this paper. The proposed faster ALM exhibits linear convergence rates in non-ergodic (the last iteration) sense of the Lagrangian primal–dual gap, the objective function value and the feasibility measure. In addition, we prove that the sequence generated by the faster ALM converges to a saddle point of the linear equality constrained optimization problem. This is the first result that provides both linear non-ergodic convergence rate and the convergence of the iterative sequence when solving linearly constrained convex optimization problems by inertial ALM-type algorithms without additional assumptions such as strong convexity or Lipschitz continuous of gradient. As an extension, we present contracting ALM for solving convex optimization problems with linear equality constraint with O ( 1 / ∑ i = 0 k a i ) ( a i > 0 is an arbitrarily constant) non-ergodic convergence rates of the Lagrangian primal–dual gap, the objective function value and the feasibility measure.

Complexity guarantees for nonconvex Newton-MR under inexact Hessian information

Abstract We consider an extension of the Newton-MR algorithm for nonconvex unconstrained optimization to the settings where Hessian information is approximated. Under a particular noise model on the Hessian matrix, we investigate the iteration and operation complexities of this variant to achieve appropriate sub-optimality criteria in several nonconvex settings. We do this by first considering functions that satisfy the (generalized) Polyak–Łojasiewicz condition, a special sub-class of nonconvex functions. We show that, under certain conditions, our algorithm achieves global linear convergence rate. We then consider more general nonconvex settings where the rate to obtain first-order sub-optimality is shown to be sub-linear. In all these settings we show that our algorithm converges regardless of the degree of approximation of the Hessian as well as the accuracy of the solution to the sub-problem. Finally, we compare the performance of our algorithm with several alternatives on a few machine learning problems.

An accelerated Dai–Yuan conjugate gradient projection method with the optimal choice

This article proposes an accelerated Dai–Yuan (DY) conjugate gradient (CG) projection method with the optimal choice for solving unconstrained pseudo-monotone nonlinear equations. Specifically, it first introduces a modified DY-type CG parameter with an indefinite factor and derives an optimal selection for this factor by minimizing the measure function. By combining the hyperplane projection approach, the inertial strategy, and a newly improved adaptive line search, the article presents an accelerated CG projection method. The proposed method is characterized by two main features: (i) the search direction is independent of any line search and possesses sufficient descent and trust region properties; and (ii) global convergence is achieved without requiring the underlying mapping to satisfy Lipschitz continuity. Additionally, the linear convergence rate of the method is proven under standard conditions. Numerical experiments on unconstrained nonlinear equations, as well as applications in signal restoration, demonstrate the effectiveness of the proposed method.

Linear Convergence of Asynchronous Gradient Push Algorithm for Distributed Optimization

Linear Convergence of Asynchronous Gradient Push Algorithm for Distributed Optimization

On linear convergence of exponential sign-based gradient descent

This paper aims to investigate sign-based methods for unconstrained optimisation problems, which utilise the sign of gradients, instead of full gradients, to reduce the communication cost in distributed optimisation. The methods introduced in this paper also provide an avenue to conquer the issue that classical sign gradient descent with a constant step size generally oscillates around the minimum value of the objective function. To this end, two sign-based algorithms are proposed and analysed. The first one, called ES-GD, is demonstrated to be linearly convergent to the optimal value if the objective function is strongly convex, smooth and separable. For the second variant, called ES-SGD, where the gradient estimates do not need to be unbiased, the algorithm is shown to converge linearly at the same rate as ES-GD to the optimal value under the same assumptions on the objective function. Numerical experiments are conducted to validate the theoretical results.

E-3SFC: Communication-Efficient Federated Learning With Double-Way Features Synthesizing.

The exponential growth in model sizes has significantly increased the communication burden in federated learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed extended single-step synthetic features compressing (E-3SFC), which consists of three subcomponents, i.e., the single-step synthetic features compressor (3SFC), a double-way compression (DWC) algorithm, and a communication budget scheduler (BS). First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. The 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback (EF) to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing DWC and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and nonconvex conditions demonstrates that 3SFC achieves linear and sublinear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms the state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

A three-term CG projected-based method with control factor and its inertial version for constrained nonlinear equations

Narushima et al. [SIAM J. Optim. 21(1), 212-230, 2011] proposed a three-term search direction that ensures the sufficient descent condition. In this study, we introduce a control factor to enhance the search direction, which not only possesses the sufficient descent property but also automatically incorporates the trust region feature, independent of the choice of line search. We then extend the modified search direction to solve constrained nonlinear equations using hyperplane projection technique. Moreover, we incorporate an inertial acceleration technique to develop an inertial version of the proposed method. The global convergence of the inertial method is theoretically established, even in the absence of Lipschitz continuity in the underlying mapping. Additionally, under suitable conditions, we prove its linear convergence rate. Numerical experiments demonstrate the excellent performance of our proposed methods in solving constrained nonlinear equations and restoring blurred images contaminated by Gaussian noise.

Decentralized Learning of Quantile Regression: A Smoothing Approach

Distributed estimation has attracted a significant amount of attention recently due to its advantages in computational efficiency and data privacy preservation. In this article, we focus on quantile regression over a decentralized network. Without a coordinating central node, a decentralized network improves system stability and increases efficiency by communicating with fewer nodes per round. However, existing related works on decentralized quantile regression have slow (sub-linear) convergence speed. We propose a novel method for decentralized quantile regression which is built upon the smoothed quantile loss. However, we argue that the smoothed loss proposed in the existing literature using a single smoothing bandwidth parameter fails to achieve fast convergence and statistical efficiency simultaneously in the decentralized setting. We propose a novel quadratic approximation of the quantile loss using a big bandwidth for the Hessian and a small bandwidth for the gradient. Our method enjoys a linear convergence rate and has optimal statistical efficiency. Numerical experiments and real data analysis are conducted to demonstrate the effectiveness of our method. Supplementary materials for this article are available online.

A Semismooth Newton-Based Augmented Lagrangian Algorithm for the Generalized Convex Nearly Isotonic Regression Problem

The generalized convex nearly isotonic regression problem addresses a least squares regression model that incorporates both sparsity and monotonicity constraints on the regression coefficients. In this paper, we introduce an efficient semismooth Newton-based augmented Lagrangian (Ssnal) algorithm to solve this problem. We demonstrate that, under reasonable assumptions, the Ssnal algorithm achieves global convergence and exhibits a linear convergence rate. Computationally, we derive the generalized Jacobian matrix associated with the proximal mapping of the generalized convex nearly isotonic regression regularizer and leverage the second-order sparsity when applying the semismooth Newton method to the subproblems in the Ssnal algorithm. Numerical experiments conducted on both synthetic and real datasets clearly demonstrate that our algorithm significantly outperforms first-order methods in terms of efficiency and robustness.

On the convergence of Newton-type proximal gradient method for multiobjective optimization problems

Recently, Ansary (Optim. Methods Softw. 38(2023), pp. 570–590) proposed a Newton-type proximal gradient method for nonlinear multiobjective optimization problems (NPGMO). However, the favourable convergence properties typically associated with Newton-type methods for NPGMO were not established in Ansary's work. In response to this gap, we develop a new analytic framework, rooted in an auxiliary inequality, to analyse the convergence behaviour of NPGMO. Specifically, we establish the quadratic termination of NPGMO for problems with strongly convex quadratic smooth terms. Under the assumption of strong convexity, we demonstrate that the NPGMO enjoys superlinear convergence, and quadratic convergence for problems that involve twice continuously differentiable, and twice Lipschitz continuously differentiable smooth terms, respectively. The novel study on the analytic framework simplifies the existing convergence analysis of second-order methods for multiobjective optimization problems.

On full linear convergence and optimal complexity of adaptive FEM with inexact solver

On full linear convergence and optimal complexity of adaptive FEM with inexact solver

Distributed algorithms with linear convergence for aggregative games over time-varying networks

Distributed algorithms with linear convergence for aggregative games over time-varying networks

Krasnoselskii–Mann Iterations: Inertia, Perturbations and Approximation

This paper is concerned with the study of a family of fixed point iterations combining relaxation with different inertial (acceleration) principles. We provide a systematic, unified and insightful analysis of the hypotheses that ensure their weak, strong and linear convergence, either matching or improving previous results obtained by analysing particular cases separately. We also show that these methods are robust with respect to different kinds of perturbations–which may come from computational errors, intentional deviations, as well as regularisation or approximation schemes–under surprisingly weak assumptions. Although we mostly focus on theoretical aspects, numerical illustrations in image inpainting and electricity production markets reveal possible trends in the behaviour of these types of methods.

New Analysis of an Away-Step Frank–Wolfe Method for Minimizing Log-Homogeneous Barriers

We present and analyze an away-step Frank–Wolfe method for the convex optimization problem [Formula: see text], where f is a θ-logarithmically homogeneous self-concordant barrier, [Formula: see text] is a linear operator that may be noninvertible, [Formula: see text] is a linear function, and [Formula: see text] is a nonempty polytope. The applications of primary interest include D-optimal design, inference of multivariate Hawkes processes, and total variation-regularized Poisson image deblurring. We establish affine-invariant and norm-independent global linear convergence rates of our method in terms of both the objective gap and the Frank–Wolfe gap. When specialized to the D-optimal design problem, our results settle a question left open since Ahipasaoglu et al. [Ahipasaoglu SD, Sun P, Todd MJ (2008) Linear convergence of a modified Frank–Wolfe algorithm for computing minimum-volume enclosing ellipsoids. Optim. Methods Software 23(1):5–19]. We also show that the iterates generated by our method will land on and remain in a face of [Formula: see text] within a bounded number of iterations, which can lead to improved local linear convergence rates (for both the objective gap and the Frank–Wolfe gap). We conduct numerical experiments on D-optimal design and inference of multivariate Hawkes processes, and our results not only demonstrate the efficiency and effectiveness of our method compared with other principled first-order methods but also, corroborate our theoretical results quite well. Funding: This work was supported by the Air Force Office of Scientific Research [Grant FA9550-22-1-0356].

On the linear convergence of policy gradient under Hadamard parameterization

Abstract The convergence of deterministic policy gradient under the Hadamard parameterization is studied in the tabular setting and the linear convergence of the algorithm is established. To this end, we first show that the error decreases at an $O(\frac{1}{k})$ rate for all the iterations. Based on this result, we further show that the algorithm has a faster local linear convergence rate after $k_{0}$ iterations, where $k_{0}$ is a constant that only depends on the MDP problem and the initialization. To show the local linear convergence of the algorithm, we have indeed established the contraction of the sub-optimal probability $b_{s}^{k}$ (i.e. the probability of the output policy $\pi ^{k}$ on non-optimal actions) when $k\ge k_{0}$.

Understanding the Douglas–Rachford splitting method through the lenses of Moreau-type envelopes

We analyze the Douglas–Rachford splitting method for weakly convex optimization problems, by the token of the Douglas–Rachford envelope, a merit function akin to the Moreau envelope. First, we use epi-convergence techniques to show that this artifact approximates the original objective function via epigraphs. Secondly, we present how global convergence and local linear convergence rates for Douglas–Rachford splitting can be obtained using such envelope, under mild regularity assumptions. The keystone of the convergence analysis is the fact that the Douglas–Rachford envelope satisfies a sufficient descent inequality alongside the generated sequence, a feature that allows us to use arguments usually employed for descent methods. We report numerical experiments that use weakly convex penalty functions, which are comparable with the known behavior of the method in the convex case.

Calculating Maximum Eigenvalues in Pairwise Comparison Matrices for the Analytic Hierarchy Process

This paper focuses on a numerical method for calculating the maximum eigenvalue of a pairwise comparison matrix in the analytic hierarchy process. Two contributions are made: First, the first and second differentials of the characteristic polynomial of a pairwise comparison matrix are demonstrated to be always positive in the region larger than the maximum eigenvalue. This is proven by effectively utilizing the Gauss–Lucas theorem, which is a mathematical principle that helps in analyzing polynomials. By leveraging this fact, both Newton’s method and the secant method are shown to generate sequences converging to the maximum eigenvalue, bringing a new perspective to the numerical computation of the maximum eigenvalue. The second contribution is the judicious selection of initial points. Using an upper bound for the maximum eigenvalue as initial points, Newton’s method and the secant method generate a decreasing sequence that is bounded below. This allows generalizing our results for lower-order pairwise comparison matrices independent of the matrix order. The positivity of the first and second differentials of the characteristic polynomial also guarantees that Newton’s method and the secant method have quadratic and super-linear convergence, respectively. Numerical simulations confirm the superiority of Newton’s method in terms of convergence.

Distributed Constrained Optimization Algorithms for Drones

The present study addresses a critical issue within the realm of drones: the challenge of distributed constrained optimization. Our research delves into an optimization scenario where the decision variable is confined to a closed convex set. The primary objective is to develop a distributed algorithm capable of tackling this optimization problem. To achieve this, we have crafted distributed algorithms for both balanced graphs and unbalanced graphs, with the method of feasible direction employed to address the considered constraint, and the method of estimating left eigenvector to address the unbalance, incorporating momentum elements. We have demonstrated that the algorithms exhibit linear convergence when the local objective functions are both smooth and strongly convex, and when the step-sizes are appropriately chosen. Additionally, the simulation outcomes validate the efficacy of our distributed algorithms.

A generalization of the forward-reflected-backward splitting method for monotone inclusions

In this paper, we propose a general forward-reflected-backward splitting method for solving monotone inclusions in Hilbert spaces. Our method extends and improves the one of Malitsky and Tam (SIAM J. Optim., 2020). The weak convergence of the proposed algorithm is established under standard conditions. The linear convergence of the proposed method is derived under an additional condition like the strong monotonicity. We also give some theoretical comparisons to demonstrate the efficiency of the proposed algorithm.

Robust Federated Learning: Maximum Correntropy Aggregation Against Byzantine Attacks.

As an emerging decentralized machine learning technique, federated learning organizes collaborative training and preserves the privacy and security of participants. However, untrustworthy devices, typically Byzantine attackers, pose a significant challenge to federated learning since they can upload malicious parameters to corrupt the global model. To defend against such attacks, we propose a novel robust aggregation method-maximum correntropy aggregation (MCA), which applies the maximum correntropy criterion (MCC) to derive a central value from parameters. Different from the previous use of MCC for denoising, we utilize it as a similarity metric to measure parameter distribution and aggregate a robust center. Correntropy in MCC, with all even-order moments of the parameter, contains high-order statistical properties, which allows for a comprehensive capture of parameter characteristics, thus helping to prevent interference from attackers. Meanwhile, correntropy extracts information from the parameters themselves, without requiring the proportion of malicious attackers. Through the fixed-point iteration, we solve the optimization objective, demonstrating the linear convergence of the iteration formula. Theoretical analysis reveals the robustness aggregation property of MCA and the error bound between MCA and the global optimal solution, with linear convergence to the optimal solution neighborhood. By performing independent identically distribution (IID) and non-IID experiments on three different datasets, we show that MCA exhibits significant robustness under mainstream attacks, whereas other methods cannot withstand all of them.