Convergence Rate Analysis Research Articles

A numerical method for a nonlocal diffusion equation with additive noise

We consider a nonlocal evolution equation representing the continuum limit of a large ensemble of interacting particles on graphs forced by noise. The two principle ingredients of the continuum model are a nonlocal term and a Q-Wiener process describing the interactions among the particles in the network and stochastic forcing, respectively. The network connectivity is given by a square integrable function called a graphon. We prove that the initial value problem for the continuum model is well-posed. Further, we construct semidiscrete (discrete in space and continuous in time) and fully discrete schemes for the nonlocal model. The former is obtained by a discontinuous Galerkin method and the latter is based on further discretizing time using the Euler–Maruyama method. We prove convergence and estimate the rate of convergence in each case. For the semidiscrete scheme, the rate of convergence is expressed in terms of the regularity of the graphon, the Q-Wiener process, and the initial data. We work in generalized Lipschitz spaces, which allows us to treat models with data of lower regularity. This is important for applications as many interesting types of connectivity, including small-world and power-law, are expressed by graphons that are not smooth. The error analysis of the fully discrete scheme, on the other hand, reveals that for some models common in applied science, one has a higher speed of convergence than that predicted by the standard estimates for the Euler–Maruyama method. The rate of convergence analysis is supplemented with detailed numerical experiments, which are consistent with our analytical results. As a by-product, this work presents a rigorous justification for taking continuum limit for a large class of interacting dynamical systems on graphs subject to noise.

Several accelerated subspace minimization conjugate gradient methods based on regularization model and convergence rate analysis for nonconvex problems

Several accelerated subspace minimization conjugate gradient methods based on regularization model and convergence rate analysis for nonconvex problems

Revisiting Analog Over-the-Air Machine Learning: The Blessing and Curse of Interference

We study a distributed machine learning problem carried out by an edge server and multiple agents in a wireless network. The objective is to minimize a global function that is a sum of the agents&#x2019; local loss functions. And the optimization is conducted by analog over-the-air model training. Specifically, each agent modulates its local gradient onto a set of waveforms and transmits to the edge server simultaneously. From the received analog signal the edge server extracts a noisy aggregated gradient which is distorted by the channel fading and interference, and uses it to update the global model and feedbacks to all the agents for another round of local computing. Since the electromagnetic interference generally exhibits a heavy-tailed intrinsic, we use the <inline-formula><tex-math notation="LaTeX">$\alpha$</tex-math></inline-formula>-stable distribution to model its statistic. In consequence, the global gradient has an infinite variance that hinders the use of conventional techniques for convergence analysis that rely on second-order moments&#x2019; existence. To circumvent this challenge, we take a new route to establish the analysis of convergence rate, as well as generalization error, of the algorithm. We also show that the training algorithm can be run in tandem with the momentum scheme to accelerate the convergence. Our analyses reveal a two-sided effect of the interference on the overall training procedure. On the negative side, heavy tail noise slows down the convergence rate of the model training: the heavier the tail in the distribution of interference, the slower the algorithm converges. On the positive side, heavy tail noise has the potential to increase the generalization power of the trained model: the heavier the tail, the better the model generalizes. This perhaps counterintuitive conclusion implies that the prevailing thinking on interference &#x2013; that it is only detrimental to the edge learning system &#x2013; is outdated and we shall seek new techniques that exploit, rather than simply mitigate, the interference for better machine learning in wireless networks.

Error estimates for finite differences approximations of the total variation

AbstractWe present a convergence rate analysis of the Rudin–Osher–Fatemi (ROF) denoising problem for two different discretizations of the total variation. The first is the standard discretization, which induces blurring in some particular diagonal directions. We prove that in a simplified setting corresponding to such a direction, the discrete ROF energy converges to the continuous one with the rate $h^{2/3}$. The second discretization is based on dual Raviart–Thomas fields and achieves an optimal $O(h)$ convergence rate for the same quantity, for discontinuous solutions with some standard hypotheses.

The Role of Communication Time in the Convergence of Federated Edge Learning

Federated Edge Learning (FEL) enables a massive number of edge devices (e.g. smart phones) to train machine learning models collaboratively. Due to the inherent unreliability of participating edge devices and the unpredictable network conditions, the final model accuracy is largely determined by the communication time between edge devices and the parameter server (PS). However, there exists very limited work to quantify the influence of the communication time in FEL and show how to minimize its negative impacts from a theoretic perspective. In this paper, we are among the first to develop a formal model of the communication time in FEL and its influence on the final model accuracy. In our work, the set of edge devices involved in each global iteration is defined as the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ECP (Engaged Client Pool)</i> . We model the communication time cost as a function with respect to the response time distribution of individual devices and the ECP size. By incorporating communication time cost into the convergence rate analysis, we propose the ECPA and H-ECPA algorithms to automatically adjust the size of the ECP so as to maximize the model accuracy in both homogeneous and heterogeneous networks. We also analyze how the tail shape of response time affects the convergence rate, and prove that the heavy tail shape can significantly lower the model accuracy. Finally, we conduct extensive experiments with real datasets, and the results confirm the correctness of our analysis and demonstrate the superiority of our proposed algorithms.

Faster Lagrangian-Based Methods in Convex Optimization

In this paper, we aim at unifying, simplifying, and improving the convergence rate analysis of Lagrangian-based methods for convex optimization problems. We first introduce the notion of nice primal algorithmic map, which plays a central role in the unification and in the simplification of the analysis of most Lagrangian-based methods. Equipped with a nice primal algorithmic map, we then introduce a versatile generic scheme, which allows for the design and analysis of faster Lagrangian (FLAG) methods with new provably sublinear rate of convergence expressed in terms of function values and feasibility violation of the original (nonergodic) generated sequence. To demonstrate the power and versatility of our approach and results, we show that most well-known iconic Lagrangian-based schemes admit a nice primal algorithmic map and hence share the new faster rate of convergence results within their corresponding FLAG.

Adaptive mutation quantum-inspired squirrel search algorithm for global optimization problems

This paper presents a novel Adaptive Mutation Quantum-inspired Squirrel Search Algorithm (AM-QSSA). Firstly, based on the population mutation, a location-update of quantum state correlative and attractor method is proposed. By introducing a random process to modify the sliding factor of the local attractor, the inevitable lack of diversity in the population renewal method is solved. The premature convergence problem of adaptive mutation rate improvement algorithm based on squirrel position update mode is introduced. Meanwhile, the paper decomposes the location update process of the SSA, and improves it with quantum-behavior. Furthermore, it proposes a novel quantum-inspired squirrels search algorithm. This method finds the complementary effect between quantum behavior and squirrel search algorithm, and solves the problem of premature convergence probability of SSA. In addition, it improves population diversity, and achieves the balance between global and local search. The efficiency of the proposed AM-QSSA is evaluated using exploitation analysis, exploration analysis, success rate analysis, convergence rate analysis on classical benchmark functions as well as Congress on Evolutionary Computation (CEC) 2017 test functions. For further study, AM-QSSA optimizes an image registration problem for an extensive study to check its applicability. The results reveal that AM-QSSA is more efficient and stable than SSA. And it is comparable to the most advanced optimization algorithms.

Rate of convergence for geometric inference based on the empirical Christoffel function

We consider the problem of estimating the support of a measure from a finite, independent, sample. The estimators which are considered are constructed based on the empirical Christoffel function. Such estimators have been proposed for the problem of set estimation with heuristic justifications. We carry out a detailed finite sample analysis, that allows us to select the threshold and degree parameters as a function of the sample size. We provide a convergence rate analysis of the resulting support estimation procedure. Our analysis establishes that we may obtain finite sample bounds which are comparable to existing rates for different set estimation procedures. Our results rely on concentration inequalities for the empirical Christoffel function and on estimates of the supremum of the Christoffel-Darboux kernel on sets with smooth boundaries, that can be considered of independent interest.

Lévy flight and chaos theory-based gravitational search algorithm for mechanical and structural engineering design optimization

Abstract The main aim of this article is to explore the real-life problem-solving potential of the proposed Lévy flight-based chaotic gravitational search algorithm (LCGSA) for the minimization of engineering design variables of speed reducer design (SRD), three bar truss design (TBTD), and hydrodynamic thrust bearing design (HTBD) problems. In LCGSA, the diversification of the search space is carried out by Lévy flight distribution. Simultaneously, chaotic maps have been utilized for the intensification of the candidate solutions towards the global optimum. Moreover, the penalty function method has been used to deal with the non-linear and fractional design constraints. The investigation of experimental outcomes has been performed through various performance metrics like statistical measures, run time analysis, convergence rate, and box plot analysis. Moreover, statistical verification of experimental results is carried out using a signed Wilcoxon rank-sum test. Furthermore, eleven heuristic algorithms were employed for comparative analysis of the simulation results. The simulation outcomes clearly show that LCGSA provides better values for TBTD and HTBD benchmarks than standard GSA and most of the competing algorithms. Besides, all the participating algorithms, including LCGSA, have the same results for the SRD problem. On the qualitative side, LCGSA has successfully resolved entrapment in local minima and convergence issues of standard GSA.

Expected complexity analysis of stochastic direct-search

This work presents the convergence rate analysis of stochastic variants of the broad class of direct-search methods of directional type. It introduces an algorithm designed to optimize differentiable objective functions f whose values can only be computed through a stochastically noisy blackbox. The proposed stochastic directional direct-search (SDDS) algorithm accepts new iterates by imposing a sufficient decrease condition on so called probabilistic estimates of the corresponding unavailable objective function values. The accuracy of such estimates is required to hold with a sufficiently large but fixed probability \(\beta\). The analysis of this method utilizes an existing supermartingale-based framework proposed for the convergence rates analysis of stochastic optimization methods that use adaptive step sizes. It aims to show that the expected number of iterations required to drive the norm of the gradient of f below a given threshold \(\epsilon\) is bounded in \({\mathcal {O}}\left( \epsilon ^{\frac{-p}{\min (p-1,1)}}/(2\beta -1)\right)\) with \(p>1\). Unlike prior analysis using the same aforementioned framework such as those of stochastic trust-region methods and stochastic line search methods, SDDS does not use any gradient information to find descent directions. However, its convergence rate is similar to those of both latter methods with a dependence on \(\epsilon\) that also matches that of the broad class of deterministic directional direct-search methods which accept new iterates by imposing a sufficient decrease condition.

A Class of Inexact Secant Algorithms with Line Search Filter Method for Nonlinear Programming

We propose a class of inexact secant methods in association with the line search filter technique for solving nonlinear equality constrained optimization. Compared with other filter methods that combine the line search method applied in most large-scale optimization problems, the inexact line search filter algorithm is more flexible and realizable. In this paper, we focus on the analysis of the local superlinear convergence rate of the algorithms, while their global convergence properties can be obtained by making an analogy with our previous work. These methods have been implemented in a Matlab code, and detailed numerical results indicate that the proposed algorithms are efficient for 43 problems from the CUTEr test set.

Convergence and stability analysis of iteratively reweighted least squares for noisy block sparse recovery

This paper considers the theoretical properties of iteratively reweighted least squares algorithm for noisy block sparse recovery problem (BIRLS for short). Li et al. used numerical experiments to show the remarkable performance of BIRLS algorithm for recovering a block sparse signal in noiseless measurement case, but no convergence analysis was given. In this paper, we focus on providing convergence, convergence rate and stability analysis of BIRLS algorithm for block sparse recovery in the presence of noise. The convergence of BIRLS is proved strictly. Furthermore, when the linear measurement matrix A satisfies the block restricted isometry property (abbreviated as block RIP), we show that BIRLS algorithm is stable and give the error analysis of BIRLS algorithm. We also characterize the convergence rate of the BIRLS algorithm, which implies global linear convergence for p=1 and local super-linear convergence for 0<p<1. The simplicity of BIRLS algorithm, along with the theoretical guarantees provided in this paper, make a compelling case for its adoption as a standard tool for block sparse recovery.

A unified convergence rate analysis of the accelerated smoothed gap reduction algorithm

In this paper, we develop a unified convergence analysis framework for the Accelerated Smoothed GAp ReDuction algorithm (ASGARD) introduced in Tran-Dinh et al. (SIAM J Optim 28(1):96–134, 2018). Unlike Tran-Dinh et al. (SIAM J Optim 28(1):96–134, 2018), the new analysis covers three settings in a single algorithm: general convexity, strong convexity, and strong convexity and smoothness. Moreover, we establish the convergence guarantees on three criteria: (i) gap function, (ii) primal objective residual, and (iii) dual objective residual. Our convergence rates are optimal (up to a constant factor) in all cases. While the convergence rate on the primal objective residual for the general convex case has been established in Tran-Dinh et al. (SIAM J Optim 28(1):96–134, 2018), we prove additional convergence rates on the gap function and the dual objective residual. The analysis for the last two cases is completely new. Our results provide a complete picture on the convergence guarantees of ASGARD. Finally, we present four different numerical experiments on a representative optimization model to verify our algorithm and compare it with the well-known Nesterov’s smoothing algorithm.

Defending against Backdoors in Federated Learning with Robust Learning Rate

Federated learning (FL) allows a set of agents to collaboratively train a model without sharing their potentially sensitive data. This makes FL suitable for privacy-preserving applications. At the same time, FL is susceptible to adversarial attacks due to decentralized and unvetted data. One important line of attacks against FL is the backdoor attacks. In a backdoor attack, an adversary tries to embed a backdoor functionality to the model during training that can later be activated to cause a desired misclassification. To prevent backdoor attacks, we propose a lightweight defense that requires minimal change to the FL protocol. At a high level, our defense is based on carefully adjusting the aggregation server's learning rate, per dimension and per round, based on the sign information of agents' updates. We first conjecture the necessary steps to carry a successful backdoor attack in FL setting, and then, explicitly formulate the defense based on our conjecture. Through experiments, we provide empirical evidence that supports our conjecture, and we test our defense against backdoor attacks under different settings. We observe that either backdoor is completely eliminated, or its accuracy is significantly reduced. Overall, our experiments suggest that our defense significantly outperforms some of the recently proposed defenses in the literature. We achieve this by having minimal influence over the accuracy of the trained models. In addition, we also provide convergence rate analysis for our proposed scheme.

A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization

In this paper, we analyze several methods for approximating gradients of noisy functions using only function values. These methods include finite differences, linear interpolation, Gaussian smoothing, and smoothing on a sphere. The methods differ in the number of functions sampled, the choice of the sample points, and the way in which the gradient approximations are derived. For each method, we derive bounds on the number of samples and the sampling radius which guarantee favorable convergence properties for a line search or fixed step size descent method. To this end, we use the results in Berahas et al. (Global convergence rate analysis of a generic line search algorithm with noise, arXiv:1910.04055 , 2019) and show how each method can satisfy the sufficient conditions, possibly only with some sufficiently large probability at each iteration, as happens to be the case with Gaussian smoothing and smoothing on a sphere. Finally, we present numerical results evaluating the quality of the gradient approximations as well as their performance in conjunction with a line search derivative-free optimization algorithm.

Block FETI–DP/BDDC preconditioners for mixed isogeometric discretizations of three-dimensional almost incompressible elasticity

A block FETI–DP/BDDC preconditioner for mixed formulations of almost incompressible elasticity is constructed and analyzed; FETI–DP (Finite Element Tearing and Interconnecting Dual-Primal) and BDDC (Balancing Domain Decomposition by Constraints) are two very successful domain decomposition algorithms for a variety of elliptic problems. The saddle point problems of the mixed problems are discretized with mixed isogeometric analysis with continuous pressure fields. As in previous work by Tu and Li (2015) for finite element discretizations of the incompressible Stokes system, the proposed preconditioner is applied to a reduced positive definite system involving only the pressure interface variable and the Lagrange multiplier of the FETI–DP algorithm. In this work, we extend the theory to a wider class of saddle point problems and we propose a novel block-preconditioning strategy, which consists in using BDDC with deluxe scaling for the interface pressure block as well as deluxe scaling for the FETI–DP preconditioner for the Lagrange multiplier block. A convergence rate analysis is presented with a condition number bound for the preconditioned operator which depends on the inf-sup parameter of the fully assembled problem and the condition number of a closely related BDDC algorithm for compressible elasticity. This bound is scalable in the number of subdomains, poly-logarithmic in the ratio of subdomain and element sizes, and robust with respect to material incompressibility. Parallel numerical experiments validate the theory, demonstrate robustness in the presence of discontinuities of the Lamé parameters, and indicate how the rate of convergence varies with respect to the spline polynomial degree and regularity and the deformation of the domain.

Improved Gray Wolf Algorithm Based On Dynamic Inertia Weight and Adaptive Search

An improved ISAGWO algorithm is proposed to solve the problem that the traditional grey Wolf algorithm is easy to fall into local optimum, which leads to low search accuracy. By introducing the weight factor before the vector coefficient, the position vector updating formula and the adaptive search strategy of the dynamic modulation algorithm, the search accuracy of the algorithm and the ability of jumping out of the local optimum are improved. The efficiency of the improved grey Wolf algorithm is evaluated by statistical analysis, convergence rate analysis and classical reference function. The results show that the improved grey Wolf algorithm has better global search ability and solution robustness, and the optimization precision and convergence speed are also improved compared with the standard grey Wolf algorithm.

Stochastic analysis of an adaptive cubic regularization method under inexact gradient evaluations and dynamic Hessian accuracy

We here adapt an extended version of the adaptive cubic regularization method with dynamic inexact Hessian information for nonconvex optimization in Bellavia et al. [Adaptive cubic regularization methods with dynamic inexact hessian information and applications to finite-sum minimization. IMA Journal of Numerical Analysis. 2021;41(1):764–799] to the stochastic optimization setting. While exact function evaluations are still considered, this novel variant inherits the innovative use of adaptive accuracy requirements for Hessian approximations introduced in the just quoted paper and additionally employs inexact computations of the gradient. Without restrictions on the variance of the errors, we assume that these approximations are available within a sufficiently large, but fixed, probability and we extend, in the spirit of Cartis and Scheinberg [Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Math Program Ser A. 2018;159(2):337–375], the deterministic analysis of the framework to its stochastic counterpart, showing that the expected number of iterations to reach a first-order stationary point matches the well-known worst-case optimal complexity. This is, in fact, still given by , with respect to the first-order ϵ tolerance. Finally, numerical tests on nonconvex finite-sum minimization confirm that using inexact first- and second-order derivatives can be beneficial in terms of the computational savings.

Distributed Nonsmooth Convex Optimization over Markovian Switching Random Networks with Two Step-Sizes

This paper investigates the distributed convex optimization problem over a multi-agent system with Markovian switching communication networks. The objective function is the sum of each agent’s local nonsmooth objective function, which cannot be known by other agents. The communication network is assumed to switch over a set of weight-balanced directed graphs with a Markovian property. The authors propose a consensus sub-gradient algorithm with two time-scale step-sizes to handle the Markovian switching topologies and the absence of global gradient information. With proper selection of step-sizes, the authors prove the almost sure convergence of all agents’ local estimates to the same optimal solution when the union graph of the Markovian network’ states is strongly connected and the Markovian chain is irreducible. The convergence rate analysis is also given for specific cases. Simulations are given to demonstrate the results.

Inexact stochastic subgradient projection method for stochastic equilibrium problems with nonmonotone bifunctions: application to expected risk minimization in machine learning

This paper discusses a stochastic equilibrium problem for which the function is in the form of the expectation of nonmonotone bifunctions and the constraint set is closed and convex. This problem includes various applications such as stochastic variational inequalities, stochastic Nash equilibrium problems, and nonconvex stochastic optimization problems. For solving this stochastic equilibrium problem, we propose an inexact stochastic subgradient projection method. The proposed method sets a random realization of the bifunction and then updates its approximation by using both its stochastic subgradient and the projection onto the constraint set. The main contribution of this paper is to present a convergence analysis showing that, under certain assumptions, any accumulation point of the sequence generated by the proposed method using a constant step size almost surely belongs to the solution set of the stochastic equilibrium problem. A convergence rate analysis of the method is also provided to illustrate the method’s efficiency. Another contribution of this paper is to show that a machine learning algorithm based on the proposed method achieves the expected risk minimization for a class of least absolute selection and shrinkage operator (lasso) problems in statistical learning with sparsity. Numerical comparisons of the proposed machine learning algorithm with existing machine learning algorithms for the expected risk minimization using LIBSVM datasets demonstrate the effectiveness and superior classification accuracy of the proposed algorithm.