Convergence Rate Analysis Research Articles

Convergence rate analysis in limit theorems for nonlinear functionals of the second Wiener chaos

This paper analyzes the distribution distance between random vectors from the analytic wavelet transform of squared envelopes of Gaussian processes and their large-scale limits. For Gaussian processes with a long-memory parameter below 1/2, the limit combines the second and fourth Wiener chaos. Using a non-Stein approach, we determine the convergence rate in the Kolmogorov metric. When the long-memory parameter exceeds 1/2, the limit is a chi-distributed random process, and the convergence rate in the Wasserstein metric is determined using multidimensional Stein’s method. Notable differences in convergence rate upper bounds are observed for long-memory parameters within (1/2,3/4) and (3/4,1).

Stability and convergence rate analysis of continuous-time delay-difference systems with both point and distributed delays

This paper studies the exponential stability and convergence rate analysis of continuous-time delay-difference systems. Firstly, stability and convergence rate analysis of delay-difference systems with both point delays and distributed delays having exponential integral kernels are studied by using the weighted Lyapunov–Krasovskii functionals (LKFs) approach and the state transformation approach, respectively. Different sufficient stability conditions expressed by linear matrix inequalities (LMIs) are presented. Secondly, as a particular case, LMIs based conditions and spectral radius based conditions are given to ensure the exponential stability with a guaranteed convergence rate for delay-difference systems with both point delays and distributed delays having constant integral kernels, respectively. Finally, numerical examples illustrate the effectiveness of the obtained results.

Efficient computation of highly oscillatory finite-part integrals

This study is devoted to the numerical evaluation of finite-part integrals given by⨎0+∞xαf(x)(x−τ)m+1S(ωx)dx,τ>0,α>−1,m∈N, where S(ωx) represents highly oscillatory functions of the form S(ωx)=Hν(1)(ωx) and S(ωx)=eiωx with ω≫1. Through singularity separation, we introduce a uniform approximation scheme utilizing the complex integration method. The study includes an analysis of error estimates and convergence rates, demonstrating the superior effectiveness of the proposed algorithm for handling oscillatory integrands. Plentiful numerical results highlight the stability and efficiency of the developed schemes.

Iteratively reweighted least squares for block sparse signal recovery with unconstrained l2,p minimization

In this paper, we study an unconstrained [Formula: see text] minimization and its associated iteratively reweighted least squares algorithm (UBIRLS) for recovering block sparse signals. Wang et al. [Y. Wang, J. Wang and Z. Xu, On recovery of block-sparse signals via mixed [Formula: see text] [Formula: see text] norm minimization, EURASIP J. Adv. Signal Process. 2013(76) (2013) 76] have used numerical experiments to show the remarkable performance of UBIRLS algorithm for recovering a block sparse signal, but no theoretical analysis such as convergence and convergence rate analysis of UBIRLS algorithm was given. We focus on providing convergence and convergence rate analysis of UBIRLS algorithm for block sparse recovery problem. First, the convergence of UBIRLS is proved strictly. Second, based on the block restricted isometry property (block RIP) of linear measurement matrix [Formula: see text], we give the error bound analysis of the UBIRLS algorithm. Lastly, we also characterize the local convergence behavior of the UBIRLS algorithm. The simplicity of UBIRLS algorithm, along with the theoretical guarantees provided in this paper, will make a compelling case for its adoption as a standard tool for block sparse recovery.

Adaptive optimization federated learning enabled digital twins in industrial IoT

The Industrial Internet of Things (IIoT) plays a pivotal role in steering enterprises towards comprehensive digital transformation and fostering intelligent production, which serves as a critical pillar of Industry 4.0. Digital twin (DT) emerges as a highly promising technology, enabling the digital transformation of the IIoT by seamlessly bridging physical systems with digital spaces. However, the overall service quality of the IIoT is severely impacted by the resource-limited devices and the massive, heterogeneous and sensitive data in the IIoT. As an innovative distributed machine learning paradigm, federated learning (FL) inherently possesses advantages in handling private and heterogeneous data. In this paper, we propose a novel framework integrating FL with DT-enabled IIoT, termed FDEI, which combines the merits of both to improve service quality while maintaining trustworthiness. To enhance the modeling efficiency, we develop FedOA, an adaptive optimization FL method that dynamically adjusts the local update coefficient and model compression rate in resource-limited IIoT scenarios, to construct the FDEI model. Specifically, leveraging the interdependence between the two variables, we conduct a theoretical analysis of the model convergence rate and derive the associated convergence bounds. Building upon the theoretical analysis, we further propose a joint adaptive adjustment strategy by optimizing the two variables across various clients to minimize runtime differences and accelerate the convergence rate. Numerical results demonstrate that our proposed approach achieves an approximate 68% improvement in convergence speed and a reduction of approximately 66% in traffic consumption compared to the benchmarks (e.g., FedAvg, AFL, and CSFL).

Convergence rate analysis of the gradient descent–ascent method for convex–concave saddle-point problems

In this paper, we study the gradient descent–ascent method for convex–concave saddle-point problems. We derive a new non-asymptotic global convergence rate in terms of distance to the solution set by using the semidefinite programming performance estimation method. The given convergence rate incorporates most parameters of the problem and it is exact for a large class of strongly convex-strongly concave saddle-point problems for one iteration. We also investigate the algorithm without strong convexity and we provide some necessary and sufficient conditions under which the gradient descent–ascent enjoys linear convergence.

Bolstering stochastic gradient descent with model building

Stochastic gradient descent method and its variants constitute the core optimization algorithms that achieve good convergence rates for solving machine learning problems. These rates are obtained especially when these algorithms are fine-tuned for the application at hand. Although this tuning process can require large computational costs, recent work has shown that these costs can be reduced by line search methods that iteratively adjust the step length. We propose an alternative approach to stochastic line search by using a new algorithm based on forward step model building. This model building step incorporates second-order information that allows adjusting not only the step length but also the search direction. Noting that deep learning model parameters come in groups (layers of tensors), our method builds its model and calculates a new step for each parameter group. This novel diagonalization approach makes the selected step lengths adaptive. We provide convergence rate analysis, and experimentally show that the proposed algorithm achieves faster convergence and better generalization in well-known test problems. More precisely, SMB requires less tuning, and shows comparable performance to other adaptive methods.

Differentially private distributed estimation and learning

We study distributed estimation and learning problems in a networked environment where agents exchange information to estimate unknown statistical properties of random variables from their privately observed samples. The agents can collectively estimate the unknown quantities by exchanging information about their private observations, but they also face privacy risks. Our novel algorithms extend the existing distributed estimation literature and enable the participating agents to estimate the expected value of a complete sufficient statistic from private signals acquired offline or online over time and to preserve the privacy of their signals and network neighborhoods. This is achieved through linear aggregation schemes with adjusted randomization schemes that add noise to the exchanged estimates subject to differential privacy constraints, both in an offline and online manner. We provide convergence rate analysis and tight finite-time convergence bounds. We show that the noise that minimizes the convergence time to the best estimates is the Laplace noise, with parameters corresponding to each agent’s sensitivity to their signal and network characteristics. Our algorithms are amenable to dynamic topologies and balancing privacy and accuracy trade-offs. Finally, to supplement and validate our theoretical results, we run experiments on real-world data from the US Power Grid Network and electric consumption data from German Households to estimate the average power consumption of power stations and households under all privacy regimes and show that our method outperforms existing first-order, privacy-aware, distributed optimization methods.

An efficient inertial subspace minimization CG algorithm with convergence rate analysis for constrained nonlinear monotone equations

Conjugate gradient (CG) methods are efficient algorithms for unconstrained optimization. An inertial step strategy is usually used to accelerate iterative methods. In 1995, Yuan and Stoer (1995) introduced a subspace study on CG algorithms and presented some subspace minimization CG (SMCG) methods. Motivated by these approaches, we incorporate a new inertial step strategy in SMCG methods and present an efficient inertial SMCG method for solving constrained nonlinear monotone equations by combining with the projection technique. The proposed inertial step strategy makes use of information from recent iterations. It can be considered as an extension of the known inertial step strategy. The resulting direction always satisfies the sufficient descent property and that of trust region. Indeed, these properties do not depend on line search rules. The global convergence and Q-linear convergence rate of the proposed method are established under mild assumptions, without any Lipschitz continuity. Numerical results indicate that the proposed method is very promising.

Real-Time Ambient Seismic Noise Tomography of the Hillside Iron Oxide–Copper–Gold Deposit

We conduct an exploration-scale ambient noise tomography (ANT) survey over the Hillside Iron Oxide–Copper–Gold (IOCG) deposit in South Australia, leveraging Fleet’s direct-to-satellite technology for real-time data analysis. The acquisition array consisted of 100 sensors spaced 260 m apart which recorded continuous vertical-component seismic ambient noise for 14 days. High quality Rayleigh wave signals, with a mean signal-to-noise ratio (SNR) of 40, were recovered in the frequency band 1–4 Hz after processing the recorded data between 0.1–9 Hz. Our modelling results capture aspects of the deposit’s known geology, including depth of cover, structures linked to mineralisation, and the mineralised host rock, down to approximately 1 km depth. We compare our velocity model with existing magnetic, gravity, induced polarisation and drilling data, showing strong correlation with each. We identify several new features of the local geology, including the behaviour of key structures down to 1 km, and highlight the significance of a Cambrian-age dolomite that cuts across the main structural corridor that hosts the Hillside deposit. An analysis of model convergence rates with respect to Rayleigh wave SNRs shows that real-time data analysis can reduce recording duration at the site by 65% compared to traditional deployment durations, from ∼14 days to ∼5 days. Finally, we conclude by commenting on the efficacy of the ANT technique for the exploration of IOCG systems more broadly.

Adaptive Localized Cayley Parametrization for Optimization Over Stiefel Manifold and Its Convergence Rate Analysis

Adaptive Localized Cayley Parametrization for Optimization Over Stiefel Manifold and Its Convergence Rate Analysis

The $ AA $-iterative algorithm in hyperbolic spaces with applications to integral equations on time scales

<p>We explored the $ AA $-iterative algorithm within the hyperbolic spaces (HSs), aiming to unveil a stability outcome for contraction maps and convergence outcomes for generalized $ (\alpha, \beta) $-nonexpansive ($ G\alpha \beta N $) maps in such spaces. Through this algorithm, we derived compelling outcomes for both strong and $ \Delta $-convergence and weak $ w^2 $-stability. Furthermore, we provided an illustrative example of $ G\alpha \beta N $ maps and conducted a comparative analysis of convergence rates against alternative iterative methods. Additionally, we demonstrated the practical relevance of our findings by applying them to solve the linear Fredholm integral equations (FIEs) and nonlinear Fredholm-Hammerstein integral equations (FHIEs) on time scales.</p>

The convergence rate analysis of conjugate gradient method in typhoon forecasting

The convergence rate analysis of conjugate gradient method in typhoon forecasting

Adaptive guaranteed lower eigenvalue bounds with optimal convergence rates

AbstractGuaranteed lower Dirichlet eigenvalue bounds (GLB) can be computed for the m-th Laplace operator with a recently introduced extra-stabilized nonconforming Crouzeix–Raviart ($$m=1$$ m = 1 ) or Morley ($$m=2$$ m = 2 ) finite element eigensolver. Striking numerical evidence for the superiority of a new adaptive eigensolver motivates the convergence analysis in this paper with a proof of optimal convergence rates of the GLB towards a simple eigenvalue. The proof is based on (a generalization of) known abstract arguments entitled as the axioms of adaptivity. Beyond the known a priori convergence rates, a medius analysis is enfolded in this paper for the proof of best-approximation results. This and subordinated $$L^2$$ L 2 error estimates for locally refined triangulations appear of independent interest. The analysis of optimal convergence rates of an adaptive mesh-refining algorithm is performed in 3D and highlights a new version of discrete reliability.

A new inversion-free iterative algorithm for the discrete algebraic Riccati equation

Abstract In this paper, by the transformation form of the discrete algebraic Riccati equation (DARE), we propose a new inverse-free iterative algorithm to obtain the positive definite solution of the DARE. Furthermore, the monotone convergence is proved and convergence rate analysis is presented for the derived algorithm. Compared with some existing algorithms, numerical examples demonstrate the feasibility and effectiveness of our algorithm.

Robust Stability Analysis of Sampled-Data Systems With Uncertainties Characterized by the ${\mathcal {L}}_\infty$-Induced Norm: Gridding Treatment With Convergence Rate Analysis

This paper tackles the problem of stability analysis for uncertain sampled-data systems consisting of the feedback interconnection between a nominal linear time-invariant (LTI) sampled-data system and a structured linear time-varying (LTV) model uncertainty. When the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\mathcal {L}}_\infty$</tex-math></inline-formula> norm and its induced norm are taken into account to characterize such uncertain systems, it has been shown that the necessary and sufficient condition for robust stability of such uncertain sampled-data systems is described through the supremum of the spectral radii of a certain nonnegative matrix defined on the sampling interval <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$[0,h)$</tex-math></inline-formula> . However, a direct computation of the supremum for practical uncertain sampled-data systems is quite difficult since it intrinsically involves an infinite number of computations of spectral radii over the continuous-time domain <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$[0,h)$</tex-math></inline-formula> . To solve this problem, we introduce a gridding approach, by which an upper bound, as well as a lower bound on the supremum of the spectral radii, are obtained. The upper and lower bounds also lead to a sufficient condition and a necessary condition for the stability of uncertain sampled-data systems, respectively. Furthermore, the gap between the original supremum of the spectral radii and the upper bound (as well as the lower bound) is shown to converge to 0 at the rate no smaller than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(1/M)^{1/n}$</tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$M$</tex-math></inline-formula> is the gridding parameter and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> is the number of perturbation blocks in the structured model uncertainty. To put it another way, both the upper bound used in the sufficient condition and the lower bound taken in the necessary condition tend to the exact supremum of the spectral radii by taking <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$M$</tex-math></inline-formula> larger. Finally, a numerical example is given to demonstrate the effectiveness of the overall arguments.

Novel regularized dynamical systems for solving hierarchical fixed point problems

In this paper, we study some Krasnoselskii-Mann type dynamical systems in solving fixed point problems. The first one can be regarded as a continuous version of the Krasnoselskii-Mann iterations. We prove that the solution of this dynamical system converges weakly to a fixed point of the involving mapping. Next, we focus our attention on a regularized Krasnoselskii-Mann type dynamical system. Besides proving existence and uniqueness of strong global solutions, we show that the generated trajectories converge strongly to a unique solution of a variational inequality over the fixed point set. Also, we provide a convergence rate analysis for the regularized dynamical system.

Optimizing the Numbers of Queries and Replies in Convex Federated Learning With Differential Privacy

Federated learning (FL) empowers distributed clients to collaboratively train a shared machine learning model through exchanging parameter information. Despite the fact that FL can protect clients' raw data, malicious users can still crack original data with disclosed parameters. To amend this flaw, differential privacy (DP) is incorporated into FL clients to disturb original parameters, which however can significantly impair the accuracy of the trained model. In this work, we study an imperative question which has been vastly overlooked by existing works: what are the optimal numbers of queries and replies in FL with DP so that the final model accuracy is maximized. In FL, the parameter server (PS) needs to query participating clients for multiple global iterations to complete training. Each client responds a query from the PS by conducting a local iteration. We consider FL that will uniformly and randomly select participating clients to conduct local iterations with the FedSGD algorithm. Our work investigates how many times the PS should query clients and how many times each client should reply the PS by incorporating two most extensively used DP mechanisms (i.e., the Laplace mechanism and Gaussian mechanisms). Through conducting convergence rate analysis, we can determine the optimal numbers of queries and replies in FL with DP so that the final model accuracy can be maximized. Finally, extensive experiments are conducted with publicly available datasets: MNIST and FEMNIST, to verify our analysis and the results demonstrate that properly setting the numbers of queries and replies can significantly improve the final model accuracy in FL with DP.

Linear convergence rate analysis of proximal generalized ADMM for convex composite programming

Linear convergence rate analysis of proximal generalized ADMM for convex composite programming

Joint Gradient Sparsification and Device Scheduling for Federated Learning

Federated learning is an attractive distributed learning framework where edge server coordinates edge devices to collaboratively train an artificial intelligence (AI) model while training data stays in edge devices. However, implementing federated learning in an energy-efficient way still faces some problems in practice: i) uploading high-dimensional parameters of edge devices is energy-costly and causes communication congestion; ii) The heterogeneity of edge devices, i.e. non-i.i.d. data, different software/hardware configurations, and time-varying channel conditions, affects training accuracy and convergence rate of federated learning. To address these issues, in this paper, we aim at minimizing energy use of the heterogeneous edge devices while maintaining the learning performance. Note that the effects of sparsification and the number of participating devices are highly coupled, and they have opposite effects on both learning performance and energy consumption. Therefore, the gradient sparsification and device scheduling should be jointly optimized for federated learning. We consider the case where a fraction of heterogeneous devices are scheduled to upload their sparsified local gradients for global aggregation at the server. First, a tradeoff between the sparsification and the number of participating devices is revealed through the convergence rate analysis. Then, a joint optimization problem is formulated to minimize energy consumption under the convergence guarantee. An efficient algorithm is proposed to find the globally optimal solution of the considered non-convex problem. Simulations verify the efficiency of the proposed scheme.