Sublinear Convergence Rate Research Articles

E-3SFC: Communication-Efficient Federated Learning With Double-Way Features Synthesizing.

The exponential growth in model sizes has significantly increased the communication burden in federated learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed extended single-step synthetic features compressing (E-3SFC), which consists of three subcomponents, i.e., the single-step synthetic features compressor (3SFC), a double-way compression (DWC) algorithm, and a communication budget scheduler (BS). First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. The 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback (EF) to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing DWC and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and nonconvex conditions demonstrates that 3SFC achieves linear and sublinear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms the state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

An Adapted Proximal Point Algorithm Utilizing the Golden Ratio Technique for Solving Equilibrium Problems in Banach Spaces

This paper explores the iterative approximation of solutions to equilibrium problems and proposes a simple proximal point method for addressing them. We incorporate the golden ratio technique as an extrapolation method, resulting in a two-step iterative process. This method is self-adaptive and does not require any Lipschitz-type conditions for implementation. We present and prove a weak convergence theorem along with a sublinear convergence rate for our method. The results extend some previously published findings from Hilbert spaces to 2-uniformly convex Banach spaces. To demonstrate the effectiveness of the method, we provide several numerical illustrations and compare the results with those from other methods available in the literature.

Leveraging user’s preference and social circle for personalized recommendation via matrix factorization with sub-linear convergence rate

Recommender systems aim to filter information effectively and recommend useful sources to match users’ requirements. However, the exponential growth of information in recent social networks may cause low prediction accuracy for recommendation systems. This article proposes a unified personalized recommendation architecture referred to as PSRec, which incorporates user preference and social relationship into matrix factorization framework. Specifically, PSRec generates two collections for textual reviews and contextual information respectively, and performs preference learning for each user via the Latent Dirichlet Allocation topic model. Moreover, PSRec exploits the inner relations within the social circle for recommendation, including direct trust relationship and indirect trust relationship. Additionally, it’s certificated that PSRec can converge at a sub-linear rate via theoretical analysis. Experimental results over DoubanMovie, CiaoDVDs and Yelp demonstrate the superiority of the proposed PSRec, which can achieve significant improvements and provide much better user experience while compared with other benchmark models.

On the linear convergence of additive Schwarz methods for the p-Laplacian

Abstract We consider additive Schwarz methods for boundary value problems involving the $p$-Laplacian. While existing theoretical estimates suggest a sublinear convergence rate for these methods, empirical evidence from numerical experiments demonstrates a linear convergence rate. In this paper we narrow the gap between these theoretical and empirical results by presenting a novel convergence analysis. First, we present a new convergence theory for additive Schwarz methods written in terms of a quasi-norm. This quasi-norm exhibits behaviour akin to the Bregman distance of the convex energy functional associated with the problem. Secondly, we provide a quasi-norm version of the Poincaré–Friedrichs inequality, which plays a crucial role in deriving a quasi-norm stable decomposition for a two-level domain decomposition setting. By utilizing these key elements we establish the asymptotic linear convergence of additive Schwarz methods for the $p$-Laplacian.

Convergence of Peaceman-Rachford splitting method with Bregman distance for three-block nonconvex nonseparable optimization

Abstract It is of strong theoretical significance and application prospects to explore three-block nonconvex optimization with nonseparable structure, which are often modeled for many problems in machine learning, statistics, and image and signal processing. In this article, by combining the Bregman distance and Peaceman-Rachford splitting method, we propose a novel three-block Bregman Peaceman-Rachford splitting method (3-BPRSM). Under a general assumption, global convergence is presented via optimality conditions. Furthermore, we prove strong convergence when the augmented Lagrange function satisfies Kurdyka-Łojasiewicz property. In addition, if the association function possessing the Kurdyka-Łojasiewicz property exhibits a distinctive structure, then linear and sublinear convergence rate of 3-BPRSM can be guaranteed. Finally, a preliminary numerical experiment demonstrates the effectiveness.

The Proximal Gradient Method for Composite Optimization Problems on Riemannian Manifolds

In this paper, the composite optimization problem is studied on Riemannian manifolds. To tackle this problem, the proximal gradient method to solve composite optimization problems is proposed on Riemannian manifolds. Under some reasonable conditions, the convergence of the proximal gradient method with the backtracking procedure in the nonconvex case is presented. Furthermore, a sublinear convergence rate and the complexity result of the proximal gradient method for convex case are also established on Riemannian manifolds.

Variance Reduction Techniques for Stochastic Proximal Point Algorithms

In the context of finite sums minimization, variance reduction techniques are widely used to improve the performance of state-of-the-art stochastic gradient methods. Their practical impact is clear, as well as their theoretical properties. Stochastic proximal point algorithms have been studied as an alternative to stochastic gradient algorithms since they are more stable with respect to the choice of the step size. However, their variance-reduced versions are not as well studied as the gradient ones. In this work, we propose the first unified study of variance reduction techniques for stochastic proximal point algorithms. We introduce a generic stochastic proximal-based algorithm that can be specified to give the proximal version of SVRG, SAGA, and some of their variants. For this algorithm, in the smooth setting, we provide several convergence rates for the iterates and the objective function values, which are faster than those of the vanilla stochastic proximal point algorithm. More specifically, for convex functions, we prove a sublinear convergence rate of O(1/k). In addition, under the Polyak-łojasiewicz condition, we obtain linear convergence rates. Finally, our numerical experiments demonstrate the advantages of the proximal variance reduction methods over their gradient counterparts in terms of the stability with respect to the choice of the step size in most cases, especially for difficult problems.

On Mean-Optimal Robust Linear Discriminant Analysis

Linear discriminant analysis (LDA) is widely used for dimensionality reduction under supervised learning settings. Traditional LDA objective aims to minimize the ratio of the squared Euclidean distances that may not perform optimally on noisy datasets. Multiple robust LDA objectives have been proposed to address this problem, but their implementations have two major limitations. One is that their mean calculations use the squared \(\ell_{2}\) -norm distance to center the data, which is not valid when the objective depends on other distance functions. The second problem is that there is no generalized optimization algorithm to solve different robust LDA objectives. In addition, most existing algorithms can only guarantee that the solution is locally optimal rather than globally optimal. In this article, we review multiple robust loss functions and propose a new and generalized robust objective for LDA. Besides, to remove the mean value within data better, our objective uses an optimal way to center the data through learning. As one important algorithmic contribution, we derive an efficient iterative algorithm to optimize the resulting non-smooth and non-convex objective function. We theoretically prove that our solution algorithm guarantees that both the objective and the solution sequences converge to globally optimal solutions at a sub-linear convergence rate. The results of comprehensive experimental evaluations demonstrate the effectiveness of our new method, achieving significant improvements compared to the other competing methods.

Distributed stochastic optimization algorithm with non-consistent constraints in time-varying unbalanced networks

Distributed stochastic optimization algorithm with non-consistent constraints in time-varying unbalanced networks

Shuffling-type gradient method with bandwidth-based step sizes for finite-sum optimization

Shuffling-type gradient method with bandwidth-based step sizes for finite-sum optimization

Efficient Computation of the Quantum Rate-Distortion Function

The quantum rate-distortion function plays a fundamental role in quantum information theory, however there is currently no practical algorithm which can efficiently compute this function to high accuracy for moderate channel dimensions. In this paper, we show how symmetry reduction can significantly simplify common instances of the entanglement-assisted quantum rate-distortion problems. This allows us to better understand the properties of the quantum channels which obtain the optimal rate-distortion trade-off, while also allowing for more efficient computation of the quantum rate-distortion function regardless of the numerical algorithm being used. Additionally, we propose an inexact variant of the mirror descent algorithm to compute the quantum rate-distortion function with provable sublinear convergence rates. We show how this mirror descent algorithm is related to Blahut-Arimoto and expectation-maximization methods previously used to solve similar problems in information theory. Using these techniques, we present the first numerical experiments to compute a multi-qubit quantum rate-distortion function, and show that our proposed algorithm solves faster and to higher accuracy when compared to existing methods.

Toward Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework.

While graph neural networks (GNNs) are popular in the deep learning community, they suffer from several challenges including over-smoothing, over-squashing, and gradient vanishing. Recently, a series of models have attempted to relieve these issues by first augmenting the node features and then imposing node-wise functions based on multilayer perceptron (MLP), which are widely referred to as graph-augmented MLP (GA-MLP) models. However, while GA-MLP models enjoy deeper architectures for better accuracy, their efficiency largely deteriorates. Moreover, popular acceleration techniques such as stochastic-version or data-parallelism cannot be effectively applied due to the dependency among samples (i.e., nodes) in graphs. To address these issues, in this article, instead of data parallelism, we propose a parallel graph deep learning Alternating Direction Method of Multipliers (pdADMM-G) framework to achieve model parallelism: parameters in each layer of GA-MLP models can be updated in parallel. The extended pdADMM-G-Q algorithm reduces communication costs by introducing the quantization technique. Theoretical convergence to a (quantized) stationary point of the pdADMM-G algorithm and the pdADMM-G-Q algorithm is provided with a sublinear convergence rate o(1/k) , where k is the number of iterations. Extensive experiments demonstrate the convergence of two proposed algorithms. Moreover, they lead to a more massive speedup and better performance than all state-of-the-art comparison methods on nine benchmark datasets. Last but not least, the proposed pdADMM-G-Q algorithm reduces communication overheads by up to 45% without loss of performance. Our code is available at https://github.com/xianggebenben/pdADMM-G.

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

In this paper, we explore a specific optimization problem that involves the combination of a differentiable nonconvex function and a nondifferentiable function. The differentiable component lacks a global Lipschitz continuous gradient, posing challenges for optimization. To address this issue and accelerate the convergence, we propose a Bregman proximal stochastic gradient method with extrapolation (BPSGE), which only requires smooth adaptivity of the differentiable part. Under variance reduction framework, we not only analyze the subsequential and global convergence of the proposed algorithm under certain conditions, but also analyze the sublinear convergence rate of the subsequence, and the complexity of the algorithm, revealing that the BPSGE algorithm requires at most O(epsilon\^\,(-2)) iterations in expectation to attain an epsilon-stationary point. To validate the effectiveness of our proposed algorithm, we conduct numerical experiments on three real-world applications: graph regularized nonnegative matrix factorization (NMF), matrix factorization with weakly-convex regularization, and NMF with nonconvex sparsity constraints. These experiments demonstrate that BPSGE is faster than the baselines without extrapolation. The code is available at: https://github.com/nothing2wang/BPSGE-Algorithm.

Stochastic alternating structure-adapted proximal gradient descent method with variance reduction for nonconvex nonsmooth optimization

The blocky optimization has gained a significant amount of attention in far-reaching practical applications. Following the recent work (M. Nikolova and P. Tan [SIAM J. Optim. 29 (2019), pp. 2053–2078]) on solving a class of nonconvex nonsmooth optimization, we develop a stochastic alternating structure-adapted proximal (s-ASAP) gradient descent method for solving blocky optimization problems. By deploying some state-of-the-art variance reduced gradient estimators (rather than full gradient) in stochastic optimization, the s-ASAP method is applicable to nonconvex optimization whose objective is the sum of a nonsmooth data-fitting term and a finite number of differentiable functions. The sublinear convergence rate of s-ASAP is built upon the proximal point algorithmic framework, whilst the linear convergence rate of s-ASAP is achieved under the error bound condition. Furthermore, the convergence of the sequence produced by s-ASAP is established under the Kurdyka-Łojasiewicz property. Preliminary numerical simulations on some image processing applications demonstrate the compelling performance of the proposed method.

Distributed Stochastic Consensus Optimization Using Communication-Censoring Strategy

In this paper, a novel communication-efficient distributed stochastic algorithm (referred as CO-DSA) is proposed for solving large-scale consensus optimization problems. As compared to the existing relevant work where only a sublinear convergence rate is obtained for strongly convex and smooth objective functions, CO-DSA achieves a linear convergence rate even in the presence of an event-triggered based communication-censoring strategy. Moreover, by properly setting the threshold function of the event-triggered communication scheme, CO-DSA maintains the same convergence rate as the algorithm without event-triggered communication. This means CO-DSA theoretically yields communication efficiency for free. Numerical experiments verify the theoretical findings and also show the excellent communication saving effect of CO-DSA in large distributed networks.

Toward Fast Personalized Semi-Supervised Federated Learning in Edge Networks: Algorithm Design and Theoretical Guarantee

Recent years have witnessed a huge demand for artificial intelligence and machine learning applications in wireless edge networks to assist individuals with real-time services. Federated learning (FL) has emerged as a suitable and appealing distributed learning paradigm to deploy these applications at the network edge. Despite the many successful efforts made to apply FL to wireless edge networks, the adopted algorithms mostly follow the same spirit as FedAvg, thereby heavily suffering from the practical challenges of label deficiency and device heterogeneity. These challenges not only decelerate the model training in FL but also downgrade the application performance. In this paper, we focus on the algorithm design and address these challenges by investigating the personalized semi-supervised FL problem and proposing an effective algorithm, named FedCPSL. In particular, the techniques of pseudo-labeling, and interpolation-based model personalization are judiciously combined to provide a new problem formulation for personalized semi-supervised FL. The proposed FedCPSL algorithm employs novel strategies, including adaptive client variance reduction, local momentum, and normalized global aggregation, to combat the challenge of device heterogeneity and boost algorithm convergence. The convergence property of FedCPSL is also thoroughly analyzed and shows that FedCPSL is resilient to both statistical and system heterogeneity, obtaining a sublinear convergence rate. Experimental results on image classification tasks are presented to demonstrate that the proposed approach outperforms its counterparts in terms of both convergence speed and application performance.

On fast convergence rates for generalized conditional gradient methods with backtracking stepsize

<p style='text-indent:20px;'>A generalized conditional gradient method for minimizing the sum of two convex functions, one of them differentiable, is presented. This iterative method relies on two main ingredients: First, the minimization of a partially linearized objective functional to compute a descent direction and, second, a stepsize choice based on an Armijo-like condition to ensure sufficient descent in every iteration. We provide several convergence results. Under mild assumptions, the method generates sequences of iterates which converge, on subsequences, towards minimizers. Moreover, a sublinear rate of convergence for the objective functional values is derived. Second, we show that the method enjoys improved rates of convergence if the partially linearized problem fulfills certain growth estimates. Most notably these results do not require strong convexity of the objective functional. Numerical tests on a variety of challenging PDE-constrained optimization problems confirm the practical efficiency of the proposed algorithm.</p>

Distributed Model Predictive Control via Separable Optimization in Multiagent Networks

We present a distributed model predictive control method which enables a group of agents to compute their control inputs locally, while communicating with their neighbors over a communication network. While many distributed model predictive control methods require a central station for some coordination or computation of the optimization variables, our method does not require a central station, making our approach applicable to a variety of communication network topologies. With our method, each agent solves for its control inputs without solving for the control inputs of other agents, allowing for efficient optimization by each agent, unlike some other distributed methods. Further, our method attains linear convergence to the optimal control inputs in convex model predictive control problems, improving upon the sub-linear convergence rates provided by some other distributed methods such as dual decomposition methods. Moreover, our algorithm provides a closed-loop controller for convex model predictive control problems with affine constraints. We demonstrate our method in both convex and non-convex model predictive control problems in wireless transceiver alignment and satellite deployment, where we show robustness of our method to time delays.

Decentralized Asynchronous Nonconvex Stochastic Optimization on Directed Graphs

We consider a decentralized stochastic optimization problem over a network of agents, modeled as a directed graph: Agents aim to asynchronously minimize the average of their individual losses (possibly non-convex), each one having access only to a noisy estimate of the gradient of its own function. We propose an asynchronous distributed algorithm for such a class of problems. The algorithm combines stochastic gradients with tracking in an asynchronous push-sum framework and obtains a sublinear convergence rate, matching the rate of the centralized stochastic gradient descent applied to the nonconvex minimization. Our experiments on a non-convex image classification task using convolutional neural network validate the convergence of our proposed algorithm across different number of nodes and graph connectivity percentages.

Customized Douglas-Rachford splitting methods for structured inverse variational inequality problems

Recently, structured inverse variational inequality (SIVI) problems have attracted much attention. In this paper, we propose new splitting methods to solve SIVI problems by employing the idea of the classical Douglas-Rachford splitting method (DRSM). In particular, the proposed methods can be regarded as a novel application of the DRSM to SIVI problems by decoupling the linear equality constraint, leading to smaller and easier subproblems. The main computational tasks per iteration are the evaluations of certain resolvent operators, which are much cheaper than those methods without taking advantage of the problem structures. To make the methods more implementable in the general cases where the resolvent operator is evaluated in an iterative scheme, we further propose to solve the subproblems in an approximate manner. Under quite mild conditions, global convergence, sublinear rate of convergence, and linear rate of convergence results are established for both the exact and the inexact methods. Finally, we present preliminary numerical results to illustrate the performance of the proposed methods.