Communication-efficient Algorithm Research Articles

M22: A Communication-Efficient Algorithm for Federated Learning Inspired by Rate-Distortion

M22: A Communication-Efficient Algorithm for Federated Learning Inspired by Rate-Distortion

Decentralized ADMM with compressed and event-triggered communication

This paper considers the decentralized optimization problem, where agents in a network cooperate to minimize the sum of their local objective functions by communication and local computation. We propose a decentralized second-order communication-efficient algorithm called communication-censored and communication-compressed quadratically approximated alternating direction method of multipliers (ADMM), termed as CC-DQM, by combining event-triggered communication with compressed communication. In CC-DQM, agents are allowed to transmit the compressed message only when the current primal variables have changed greatly compared to its last estimate. Moreover, to relieve the computation cost, the update of Hessian is also scheduled by the trigger condition. Theoretical analysis shows that the proposed algorithm can still maintain an exact linear convergence, despite the existence of compression error and intermittent communication, if the local objective functions are strongly convex and smooth. Finally, numerical experiments demonstrate its satisfactory communication efficiency.

Decentralized Federated Averaging.

Federated averaging (FedAvg) is a communication-efficient algorithm for distributed training with an enormous number of clients. In FedAvg, clients keep their data locally for privacy protection; a central parameter server is used to communicate between clients. This central server distributes the parameters to each client and collects the updated parameters from clients. FedAvg is mostly studied in centralized fashions, requiring massive communications between the central server and clients, which leads to possible channel blocking. Moreover, attacking the central server can break the whole system's privacy. Indeed, decentralization can significantly reduce the communication of the busiest node (the central one) because all nodes only communicate with their neighbors. To this end, in this paper, we study the decentralized FedAvg with momentum (DFedAvgM), implemented on clients that are connected by an undirected graph. In DFedAvgM, all clients perform stochastic gradient descent with momentum and communicate with their neighbors only. To further reduce the communication cost, we also consider the quantized DFedAvgM. The proposed algorithm involves the mixing matrix, momentum, client training with multiple local iterations, and quantization, introducing extra items in the Lyapunov analysis. Thus, the analysis of this paper is much more challenging than previous decentralized (momentum) SGD or FedAvg. We prove convergence of the (quantized) DFedAvgM under trivial assumptions; the convergence rate can be improved to sublinear when the loss function satisfies the PŁ property. Numerically, we find that the proposed algorithm outperforms FedAvg in both convergence speed and communication cost.

Communication-Censored Distributed Stochastic Gradient Descent.

This article develops a communication-efficient algorithm to solve the stochastic optimization problem defined over a distributed network, aiming at reducing the burdensome communication in applications, such as distributed machine learning. Different from the existing works based on quantization and sparsification, we introduce a communication-censoring technique to reduce the transmissions of variables, which leads to our communication-censored distributed stochastic gradient descent (CSGD) algorithm. Specifically, in CSGD, the latest minibatch stochastic gradient at a worker will be transmitted to the server if and only if it is sufficiently informative. When the latest gradient is not available, the stale one will be reused at the server. To implement this communication-censoring strategy, the batch size is increasing in order to alleviate the effect of stochastic gradient noise. Theoretically, CSGD enjoys the same order of convergence rate as that of SGD but effectively reduces communication. Numerical experiments demonstrate the sizable communication saving of CSGD.

Resource-Constrained Federated Edge Learning With Heterogeneous Data: Formulation and Analysis

Efficient collaboration between collaborative machine learning and wireless communication technology, forming a Federated Edge Learning (FEEL), has spawned a series of next-generation intelligent applications. However, due to the openness of network connections, the FEEL framework generally involves hundreds of remote devices (or clients), resulting in expensive communication costs, which is not friendly to resource-constrained FEEL. To address this issue, we propose a distributed approximate Newton-type algorithm with fast convergence speed to alleviate the problem of FEEL resource (in terms of communication resources) constraints. Specifically, the proposed algorithm is improved based on distributed L-BFGS algorithm and allows each client to approximate the high-cost Hessian matrix by computing the low-cost Fisher matrix in a distributed manner to find a “better” descent direction, thereby speeding up convergence. Second, we prove that the proposed algorithm has linear convergence in strongly convex and non-convex cases and analyze its computational and communication complexity. Similarly, due to the heterogeneity of the connected remote devices, FEEL faces the challenge of heterogeneous data and non-IID (Independent and Identically Distributed) data. To this end, we design a simple but elegant training scheme, namely FedOVA (Federated One-vs-All), to solve the heterogeneous statistical challenge brought by heterogeneous data. In this way, FedOVA first decomposes a multi-class classification problem into more straightforward binary classification problems and then combines their respective outputs using ensemble learning. In particular, the scheme can be well integrated with our communication efficient algorithm to serve FEEL. Numerical results verify the effectiveness and superiority of the proposed algorithm.

DQC-ADMM: Decentralized Dynamic ADMM With Quantized and Censored Communications.

In distributed learning and optimization, a network of multiple computing units coordinates to solve a large-scale problem. This article focuses on dynamic optimization over a decentralized network. We develop a communication-efficient algorithm based on the alternating direction method of multipliers (ADMM) with quantized and censored communications, termed DQC-ADMM. At each time of the algorithm, the nodes collaborate to minimize the summation of their time-varying, local objective functions. Through local iterative computation and communication, DQC-ADMM is able to track the time-varying optimal solution. Different from traditional approaches requiring transmissions of the exact local iterates among the neighbors at every time, we propose to quantize the transmitted information, as well as adopt a communication-censoring strategy for the sake of reducing the communication cost in the optimization process. To be specific, a node transmits the quantized version of the local information to its neighbors, if and only if the value sufficiently deviates from the one previously transmitted. We theoretically justify that the proposed DQC-ADMM is capable of tracking the time-varying optimal solution, subject to a bounded error caused by the quantized and censored communications, as well as the system dynamics. Through numerical experiments, we evaluate the tracking performance and communication savings of the proposed DQC-ADMM.

Auto-weighted Robust Federated Learning with Corrupted Data Sources

Federated learning provides a communication-efficient and privacy-preserving training process by enabling learning statistical models with massive participants without accessing their local data. Standard federated learning techniques that naively minimize an average loss function are vulnerable to data corruptions from outliers, systematic mislabeling, or even adversaries. In this article, we address this challenge by proposing Auto-weighted Robust Federated Learning ( ARFL ), a novel approach that jointly learns the global model and the weights of local updates to provide robustness against corrupted data sources. We prove a learning bound on the expected loss with respect to the predictor and the weights of clients, which guides the definition of the objective for robust federated learning. We present an objective that minimizes the weighted sum of empirical risk of clients with a regularization term, where the weights can be allocated by comparing the empirical risk of each client with the average empirical risk of the best \( p \) clients. This method can downweight the clients with significantly higher losses, thereby lowering their contributions to the global model. We show that this approach achieves robustness when the data of corrupted clients is distributed differently from the benign ones. To optimize the objective function, we propose a communication-efficient algorithm based on the blockwise minimization paradigm. We conduct extensive experiments on multiple benchmark datasets, including CIFAR-10, FEMNIST, and Shakespeare, considering different neural network models. The results show that our solution is robust against different scenarios, including label shuffling, label flipping, and noisy features, and outperforms the state-of-the-art methods in most scenarios.

Privacy-Preserving Communication-Efficient Federated Multi-Armed Bandits

Communication bottleneck and data privacy are two critical concerns in federated multi-armed bandit (MAB) problems, such as situations in decision-making and recommendations of connected vehicles via wireless. In this paper, we design the privacy-preserving communication-efficient algorithm in such problems and study the interactions among privacy, communication and learning performance in terms of the regret. To be specific, we design privacy-preserving learning algorithms and communication protocols and derive the learning regret when networked private agents are performing online bandit learning in a master-worker, a decentralized and a hybrid structure. Our bandit learning algorithms are based on epoch-wise sub-optimal arm eliminations at each agent and agents exchange learning knowledge with the server/each other at the end of each epoch. Furthermore, we adopt the differential privacy (DP) approach to protect the data privacy at each agent when exchanging information; and we curtail communication costs by making less frequent communications with fewer agents participation. By analyzing the regret of our proposed algorithmic framework in the master-worker, decentralized and hybrid structures, we theoretically show trade-offs between regret and communication costs/privacy. Finally, we empirically show these trade-offs which are consistent with our theoretical analysis.

An Event-Triggered Approach for Gradient Tracking in Consensus-Based Distributed Optimization

This paper is concerned with a communication-efficient algorithm update scheme for solving distributed convex optimization problems by introducing a distributed event-triggered approach. Compared with real-time consensus-based distributed optimization algorithms in the literature, this paper focuses on extending the real-time gradient tracking scheme and proposes a novel distributed event-triggering condition to reduce the frequency of information exchange between agents in a network. The proposed event-triggered approach for consensus-based distributed optimization algorithms not only avoids the real-time consecutive communication and the coordinated computation between agents but reduces the computation load of algorithm execution. Furthermore, the proposed event-triggering condition depends on only local states from neighbors at only their event times and does not require a global and homogeneous sampling period. In addition, this paper analytically shows that the proposed consensus-based distributed optimization algorithm based on an event-triggered approach can also converge to the exact optimal solution with a linear convergence rate even if the real-time consecutive communication between agents is replaced with sporadic communication.

Memory and communication efficient algorithm for decentralized counting of nodes in networks.

Node counting on a graph is subject to some fundamental theoretical limitations, yet a solution to such problems is necessary in many applications of graph theory to real-world systems, such as collective robotics and distributed sensor networks. Thus several stochastic and naïve deterministic algorithms for distributed graph size estimation or calculation have been provided. Here we present a deterministic and distributed algorithm that allows every node of a connected graph to determine the graph size in finite time, if an upper bound on the graph size is provided. The algorithm consists in the iterative aggregation of information in local hubs which then broadcast it throughout the whole graph. The proposed node-counting algorithm is on average more efficient in terms of node memory and communication cost than its previous deterministic counterpart for node counting, and appears comparable or more efficient in terms of average-case time complexity. As well as node counting, the algorithm is more broadly applicable to problems such as summation over graphs, quorum sensing, and spontaneous hierarchy creation.

SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization

In this paper, we propose and analyze SQuARM-SGD, a communication-efficient algorithm for decentralized training of large-scale machine learning models over a network. In SQuARM-SGD, each node performs a fixed number of local SGD steps using Nesterov's momentum and then sends sparsified and quantized updates to its neighbors regulated by a locally computable triggering criterion. We provide convergence guarantees of our algorithm for general (non-convex) and convex smooth objectives, which, to the best of our knowledge, is the first theoretical analysis for compressed decentralized SGD with momentum updates. We show that the convergence rate of SQuARM-SGD matches that of vanilla SGD. We empirically show that including momentum updates in SQuARM-SGD can lead to better test performance than the current state-of-the-art which does not consider momentum updates.

A novel communication efficient peer-to-peer energy trading scheme for enhanced privacy in microgrids

Peer-to-peer energy trading is an emerging concept for implementing power markets with a large number of distributed energy resources. The success of such trading depends on the participation of lots of prosumers. Therefore, the trading scheme should motivate peers to participate in the market through fair pricing while guaranteeing their privacy. A novel communication efficient algorithm, called Energy Trading Distributed Alternating Direction Method of Multipliers (ETD-ADMM), is proposed to solve the social welfare maximization problem in a decentralized manner without requiring any central authority. The algorithm is formulated using the concept of node coloring, where each node only needs to negotiate with its neighbors to reach the global optimal solution. The decrease in communications makes the trading scheme more privacy-preserving and scalable than other contemporary approaches. The proposed algorithm is tested on a wide range of scenarios that prove its optimality and fast convergence. The comparison of the algorithm with other state-of-the-art distributed approaches shows that it requires much fewer communications and iterations to get similar optimal outcomes.

Wireless Sensor Network-Based Distributed Approach to Identify Spatio-Temporal Volterra Model for Industrial Distributed Parameter Systems

The prodigious amount of data movement among sources, data centers, or processing elements precludes the utilization of least-squares (LS) and fusion-center (FC)-based modeling and control. The LS methods are offline in nature, hence may face difficulty in real-time implementation. FC-based methods that are nonrobust due to single point failure, require large communication bandwidth and computationally fast processing unit. To curb these limitations, this article identifies distributed parameter systems by estimating the parameters of spatio-temporal Volterra model using in-network data processing. It can handle the immense volume of data by distributing the processing tasks of FC among the wireless sensor network nodes. To facilitate distributed optimization, the global objective function is reformulated as a multiple constrained separable problem which is then decomposed into augmented Lagrangian form. Then, alternating direction method of multipliers along with coordinate descent method is employed to obtain the global optimal solution collaboratively. Further, a communication-efficient algorithm is designed for the proposed approach to deploy in an ad-hoc network. Simulations are carried out on two industrial distributed parameter systems (catalytic rod and tubular reactor) to illustrate the practicality of the proposed algorithm.

Fast and Communication-Efficient Algorithm for Distributed Support Vector Machine Training

Support Vector Machines (SVM) are widely used as supervised learning models to solve the classification problem in machine learning. Training SVMs for large datasets is an extremely challenging task due to excessive storage and computational requirements. To tackle so-called big data problems, one needs to design scalable distributed algorithms to parallelize the model training and to develop efficient implementations of these algorithms. In this paper, we propose a distributed algorithm for SVM training that is scalable and communication-efficient. The algorithm uses a compact representation of the kernel matrix, which is based on the QR decomposition of low-rank approximations, to reduce both computation and storage requirements for the training stage. This is accompanied by considerable reduction in communication required for a distributed implementation of the algorithm. Experiments on benchmark data sets with up to five million samples demonstrate negligible communication overhead and scalability on up to 64 cores. Execution times are vast improvements over other widely used packages. Furthermore, the proposed algorithm has linear time complexity with respect to the number of samples making it ideal for SVM training on decentralized environments such as smart embedded systems and edge-based internet of things, IoT.

Communication efficient distributed weighted non-linear least squares estimation

The paper addresses design and analysis of communication-efficient distributed algorithms for solving weighted non-linear least squares problems in multi-agent networks. Communication efficiency is highly relevant in modern applications like cyber-physical systems and the Internet of things, where a significant portion of the involved devices have energy constraints in terms of limited battery power. Furthermore, non-linear models arise frequently in such systems, e.g., with power grid state estimation. In this paper, we develop and analyze a non-linear communication-efficient distributed algorithm dubbed mathcal {CREDO-NL} (non-linear mathcal {CREDO}). mathcal {CREDO-NL} generalizes the recently proposed linear method mathcal {CREDO} (Communication efficient REcursive Distributed estimatOr) to non-linear models. We establish for a broad class of non-linear least squares problems and generic underlying multi-agent network topologies mathcal {CREDO-NL}’s strong consistency. Furthermore, we demonstrate communication efficiency of the method, both theoretically and by simulation examples. For the former, we rigorously prove that mathcal {CREDO-NL} achieves significantly faster mean squared error rates in terms of the elapsed communication cost over existing alternatives. For the latter, the considered simulation experiments show communication savings by at least an order of magnitude.

LDA classifier monitoring in distributed streaming systems

An important problem in real systems for mining data streams is to detect changes in the dynamic model describing the temporal data. Such changes indicate that the underlying data has undergone a transition which may well require attention. A distributed setting poses one of the main challenges in this type of change detection. In a distributed setting, model training requires centralizing the data from all nodes (hereafter, synchronization), which is very costly in terms of communication. In order to minimize communication, a monitoring algorithm should be executed locally at each node, while preserving the validity of the global model (that is, the model that will be computed if a synchronization takes place). To achieve this goal, we propose the first communication-efficient algorithm for monitoring a classification model over distributed, dynamic data streams. While the approach is general, here we concentrate on Linear Discriminant Analysis (LDA), a popular method for classification and dimensionality reduction in many fields. We mainly apply tools from the realms of linear algebra and multi-variate analysis in order to solve the problem at hand. The resulting implementation is quite straightforward.The emphasis of this work is not on solving the distributed optimization problem that corresponds to finding a classifier over the distributed data; instead, we continuously monitor the current classifier to check that it still fits the data. In addition to the theoretical guarantee of the proposed monitoring algorithm, we demonstrate how it reduces communication volume by up to two orders of magnitude (compared to synchronization in every round) on synthetic data as well as three real datasets from different worlds of content. Our approach monitors the classification model itself as opposed to its misclassifications, which makes it possible to detect the change before misclassifications occur.

Decentralized and Collaborative Subspace Pursuit: A Communication-Efficient Algorithm for Joint Sparsity Pattern Recovery With Sensor Networks

In this paper, we consider the problem of joint sparsity pattern recovery in a distributed sensor network. The sparse multiple measurement vector signals (MMVs) observed by all the nodes are assumed to have a common (but unknown) sparsity pattern. To accurately recover the common sparsity pattern in a decentralized manner with a low communication overhead of the network, we develop an algorithm named decentralized and collaborative subspace pursuit (DCSP). In DCSP, each node is required to perform three kinds of operations per iteration: 1) estimate the local sparsity pattern by finding the subspace that its measurement vector most probably lies in; 2) share its local sparsity pattern estimate with one-hop neighboring nodes; and 3) update the final sparsity pattern estimate by majority vote based fusion of all the local sparsity pattern estimates obtained in its neighborhood. The convergence of DCSP is proved and its communication overhead is quantitatively analyzed. We also propose another decentralized algorithm named generalized DCSP (GDCSP) by allowing more information exchange among neighboring nodes to further improve the accuracy of sparsity pattern recovery at the cost of increased communication overhead. Experimental results show that, 1) compared with existing decentralized algorithms, DCSP provides much better accuracy of sparsity pattern recovery at a comparable communication cost; and 2) the accuracy of GDCSP is very close to that of centralized processing.

Communication-optimal eventually perfect failure detection in partially synchronous systems

Since Chandra and Toueg introduced the failure detector abstraction for crash-prone systems, several algorithms implementing failure detectors in partially synchronous systems have been proposed. Their performance can be measured by their Communication efficiency, defined as the number of links used forever. In this regard, in a communication-efficient algorithm only n links are used forever, n being the number of processes in the system. In this paper, we present communication optimality, a communication efficiency degree reached when only c links are used forever, c being the number of correct processes. We show that c is the minimum number of links used forever required to implement ◇P and that c is also optimal for ◇S and Ω when c<n. Finally, we propose two communication-optimal ◇P algorithms following respectively one-to-all and one-to-one communication patterns to manage suspicions, showing that there is a trade-off between detection latency and sporadic communication overhead.

Top- [formula omitted] vectorial aggregation queries in a distributed environment

Given a large set of objects in a distributed database, the goal of a top- k query is to determine the top- k scoring objects and return them to the user. Efficient top- k ranking over distributed databases has been the focus of recent research, with most current algorithms operating on the assumption that each node holds a single or small subset of each object’s numerical attributes. However, in many important setups each node might hold instead a full d-dimensional vector of numerical attributes for each object. Examples include website activity in distributed servers, sales statistics for a retail chain, or share price information in different stock markets. For these setups, we define a novel ranking problem, top- k vectorial aggregation queries, where each object’s score is determined by first aggregating the attribute vectors held for it and then applying the scoring function over the aggregated vector. Our communication-efficient algorithm uses a blend of geometric and skyline related machinery, some of which is newly developed, as well as an algorithmic framework for defining generic local constraints. Whereas previous algorithms have reduced data sharing by defining local thresholds for each attribute, such tailored solutions might perform poorly. Experimental results on real-world data demonstrate that our algorithm maintains low latency, with a communication cost up to four orders of magnitude lower than that of existing solutions.

A communication efficient probabilistic algorithm for mining frequent itemsets from a peer‐to‐peer network

AbstractData intensive large‐scale distributed systems like peer‐to‐peer (P2P) networks are becoming increasingly popular where centralization of data is impossible for mining and analysis. Unfortunately, most of the existing data mining algorithms work only when data can be accessed in its entirety. Finding all the network‐wide frequent itemsets is computationally difficult and usually has large communication overhead in such environment. This paper focuses on developing a communication efficient algorithm for discovering frequent itemsets from a P2P network. A sampling‐based approach is adopted to find approximate solution instead of an exact solution with probabilistic guarantee. The benefit of approximation technique is reflected in the low communication overhead in discovering majority of frequent itemsets with probabilistic guarantee. The main principal followed by the algorithm assumes that an independent and identically distributed (iid) sample of the entire data is available at one location to generate a set of candidate itemsets. Collecting iid sample from a P2P network is a challenging problem because of varying degrees of connectivity and sizes of data shared. The paper first addresses this issue and shows how an iid sample of nodes and data can be collected from a P2P network using random walk. It applies the proposed sampling technique to identify most of the frequent itemsets from a P2P network. Theoretical analysis shows how to decide about optimum sample size and minimize communication to compute the results. Experimental results show that the proposed algorithm discovers all of the network‐wide frequent itemsets using communication that scales sublinearly with network and datasize. © 2009 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 2: 48‐69, 2009