Decentralized Learning Algorithm Research Articles

DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations

Decentralized learning has emerged as an alternative method to the popular parameter-server framework which suffers from high communication burden, single-point failure and scalability issues due to the need of a central server. However, most existing works focus on a single shared model for all workers regardless of the data heterogeneity problem, rendering the resulting model performing poorly on individual workers. In this work, we propose a novel personalized decentralized learning algorithm named DePRL via shared representations. Our algorithm relies on ideas from representation learning theory to learn a low-dimensional global representation collaboratively among all workers in a fully decentralized manner, as well as a user-specific low-dimensional local head leading to a personalized solution for each worker. We show that DePRL achieves, for the first time, a provable \textit{linear speedup for convergence} with general non-linear representations (i.e., the convergence rate is improved linearly with respect to the number of workers). Experimental results support our theoretical findings showing the superiority of our method in data heterogeneous environments.

Decentralized graph-based multi-agent reinforcement learning using reward machines

In multi-agent reinforcement learning (MARL), it is challenging for a collection of agents to learn complex temporally extended tasks. The difficulties lie in computational complexity and how to learn the high-level ideas behind reward functions. We study the graph-based Markov Decision Process (MDP), where the dynamics of neighboring agents are coupled. To learn complex temporally extended tasks, we use a reward machine (RM) to encode each agent’s task and expose reward function internal structures. RM has the capacity to describe high-level knowledge and encode non-Markovian reward functions. We propose a decentralized learning algorithm to tackle computational complexity, called decentralized graph-based reinforcement learning using reward machines (DGRM), that equips each agent with a localized policy, allowing agents to make decisions independently based on the information available to the agents. DGRM uses the actor-critic structure, and we introduce the tabular Q-function for discrete state problems. We show that the dependency of the Q-function on other agents decreases exponentially as the distance between them increases. To further improve efficiency, we also propose the deep DGRM algorithm, using deep neural networks to approximate the Q-function and policy function to solve large-scale or continuous state problems. The effectiveness of the proposed DGRM algorithm is evaluated by three case studies, two wireless communication case studies with independent and dependent reward functions, respectively, and COVID-19 pandemic mitigation. Experimental results show that local information is sufficient for DGRM and agents can accomplish complex tasks with the help of RM. DGRM improves the global accumulated reward by 119% compared to the baseline in the case of COVID-19 pandemic mitigation.

Efficient Decentralized Multi-agent Learning in Asymmetric Bipartite Queueing Systems

New Algorithm Enables Efficient Decentralized Learning in Bipartite Queueing Systems Bipartite queueing systems, where agents with individual job queues request service from a pool of heterogeneous servers, are standard models for service applications like data networks and call centers. Traditionally, a central controller schedules agent requests with full knowledge of system parameters. However, emerging applications require decentralized operation without this central coordination or complete system information. This presents challenges as agents lack the global knowledge needed to efficiently route jobs. Recent research into efficient decentralized learning algorithms for such systems faces limitations in nonoptimal throughput, demanding computations, or degrading efficiency with exponential queue growth. In contrast, this paper introduces an algorithm that enables queues to efficiently learn decentralized scheduling policies while ensuring throughput optimality. The approach is computationally lightweight, achieving queue length bounds that scale polynomially rather than exponentially in system size. Experiments demonstrate faster convergence and robustness of our algorithm compared with prior decentralized algorithms.

Robust communication-efficient decentralized learning with heterogeneity

In this paper, we propose a robust communication-efficient decentralized learning algorithm, named RCEDL, to address data heterogeneity, communication heterogeneity and communication efficiency simultaneously in real-world scenarios. To the best of our knowledge, this is the first work to address the above challenges in a united framework. In detail, we design a compressed cross-gradient aggregation mechanism with delay to resolve the Non-IID issues, a blocking-resilient mechanism which allows receiving delayed parameters and gradients, and a communication-efficient mechanism including parameters compression and adaptive neighbors selection methods to reduce the communication cost as much as possible. In addition, we also provide convergence analysis of RCEDL and prove its convergence rate O(1NK) same with the state-of-the-art decentralized learning algorithms. Finally, we conduct extensive experiments to evaluate RCEDL algorithm on two widely used datasets CIFAR-10 and MNIST under different experimental settings. Compared with the state-of-the-art baseline methods, the proposed RCEDL is much more robust with higher accuracy and at least 3.4 × communication cost reduction under the heterogeneous environment.

Robust decentralized stochastic gradient descent over unstable networks

Decentralized learning is essential for large-scale deep learning due to its great advantage in breaking the communication bottleneck. Most decentralized learning algorithms focus on reducing the communication overhead without taking into account the possibility of a shaky network connection, and existing analyses over unstable networks have various limitations such as centralized settings, strong unrealistic assumptions, etc. Hence, in this work, we study a non-convex optimization problem over unstable networks that fully consider unstable factors including unstable network connections, communication and artificially injected noise. Specifically, we focus on the most commonly used Stochastic Gradient Descent (SGD) algorithm in a mild decentralized setting and propose a robust algorithm to handle unstable networks. It is shown that our algorithm can attain a convergence rate which has the same order as decentralized algorithms over stable networks, and achieves linear speedup comparing with centralized ones. Moreover, the proposed algorithm also applies to the general case that the data are not independently and identically distributed. Extensive experiments on image classification demonstrate that the practical performance of our algorithm is comparable with the state-of-art decentralized algorithms in stable networks with only a little accuracy loss.

A decentralized learning strategy to restore connectivity during multi-agent formation control

In this paper, we propose a decentralized learning algorithm to restore communication connectivity during multi-agent formation control. The time-varying connectivity profile of a mobile multi-agent system represents the dynamic information exchange capabilities among agents. While connected to the neighbors, each mobile agent in the proposed scheme learns to raise the team connectivity. When the inter-agent communication is lost, the associated trained neural network generates appropriate control actions to restore connectivity. The proposed learning technique leverages an adaptive control formalism, wherein a neural network tries to mimic the negative gradient of a value that relies on the agent-to-neighbor distances. All agents use the conventional consensus protocol during the connected multi-agent dynamics, and under communication loss, only the lost agent executes the neural network predicted actions to come back to the fleet. Simulation results demonstrate the effectiveness of our proposed approach for single/multiple agent loss even in the presence of velocity disturbances.

A Graph Neural Network Based Decentralized Learning Scheme.

As an emerging paradigm considering data privacy and transmission efficiency, decentralized learning aims to acquire a global model using the training data distributed over many user devices. It is a challenging problem since link loss, partial device participation, and non-independent and identically distributed (non-iid) data distribution would all deteriorate the performance of decentralized learning algorithms. Existing work may restrict to linear models or show poor performance over non-iid data. Therefore, in this paper, we propose a decentralized learning scheme based on distributed parallel stochastic gradient descent (DPSGD) and graph neural network (GNN) to deal with the above challenges. Specifically, each user device participating in the learning task utilizes local training data to compute local stochastic gradients and updates its own local model. Then, each device utilizes the GNN model and exchanges the model parameters with its neighbors to reach the average of resultant global models. The iteration repeats until the algorithm converges. Extensive simulation results over both iid and non-iid data validate the algorithm’s convergence to near optimal results and robustness to both link loss and partial device participation.

3-D Deployment of UAV Swarm for Massive MIMO Communications

We consider the uplink transmission between a multi-antenna ground station and an unmanned aerial vehicle (UAV) swarm. The UAVs are assumed as intelligent agents, which can explore their optimal three dimensional (3-D) deployment to maximize the channel capacity of the multiple input multiple output (MIMO) system. Specifically, considering the limitations of each UAV in accessing the global information of the network, we focus on a decentralized control strategy by noting that each UAV in the swarm can only utilize the local information to achieve the optimal 3-D deployment. In this case, the optimization problem can be divided into several optimization sub-problems with respect to the rank function. Due to the non-convex nature of the rank function and the fact that the optimization sub-problems are coupled, the original problem is NP-hard and, thus, cannot be solved with standard convex optimization solvers. Interestingly, we can relax the constraint condition of each sub-problem and solve the optimization problem by a formulated UAVs channel capacity maximization game. We analyze such game according to the designed reward function and the potential function. Then, we discuss the existence of the pure Nash equilibrium in the game. To achieve the best Nash equilibrium of the MIMO system, we develop a decentralized learning algorithm, namely decentralized UAVs channel capacity learning. The details of the algorithm are provided, and then, the convergence, the effectiveness and the computational complexity are analyzed, respectively. Moreover, we give some insightful remarks based on the proofs and the theoretical analysis. Also, extensive simulations illustrate that the developed learning algorithm can achieve a high MIMO channel capacity by optimizing the 3-D UAV swarm deployment with the local information.

Decentralized Dynamic Rate and Channel Selection Over a Shared Spectrum

We consider the problem of distributed dynamic rate and channel selection in a multi-user network, in which each user selects a wireless channel and a modulation and coding scheme (corresponds to a transmission rate) in order to maximize the network throughput. We assume that the users are cooperative, however, there is no coordination and communication among them, and the number of users in the system is unknown. We formulate this problem as a multi-player multi-armed bandit problem and propose a decentralized learning algorithm that performs almost optimal exploration of the transmission rates to learn fast. We prove that the regret of our learning algorithm with respect to the optimal allocation increases logarithmically over rounds with a leading term that is logarithmic in the number of transmission rates. Finally, we compare the performance of our learning algorithm with the state-of-the-art via simulations and show that it substantially improves the throughput and minimizes the number of collisions.

Stochastic Congestion Game for Load Balancing in Mobile-Edge Computing

Mobile-edge computing can reduce task execution delay and improve the Quality of Experience (QoE) for the network edge users. However, when there are multiple independent cloudlets in the network with the mobile users offloading tasks randomly, how to maintain the load balancing of the independent cloudlets, how to improve the quality of service and users' QoE are still issues need to be solved. To this end, we study these issues from the perspective of game theory and propose decentralized learning algorithms. First, we turn the cloudlets load-balancing issue into a competition that each user minimizes its task execution time, and then a stochastic congestion game with incomplete information is proposed. Second, based on the existence proof of the Nash equilibria by using potential game theory, we propose a multiuser decentralized learning algorithm to obtain the pure Nash equilibrium strategy of each user. Then, an ordinary differential equation is derived to prove the convergence of the algorithm. Finally, we propose two application scenarios, one is for static users and the other is for dynamic users, and then the performances of the algorithm in a static scenario is tested. In order to adapt to dynamic scenarios, and further improve the performance and reduce communication costs, we propose a decentralized learning algorithm with termination condition. The experiments show that this algorithm can improve the load balancing of the multicloudlet system, and enhance the quality of service.

Performance Analysis of Trial and Error Algorithms

Model-free decentralized optimizations and learning are receiving increasing attention from theoretical and practical perspectives. In particular, two fully decentralized learning algorithms, namely Trial and Error Learning (TEL) and Optimal Dynamical Learning (ODL), are very appealing for a broad class of games. Indeed, ODL has the property to spend a high proportion of time in an optimum state that maximizes the sum of the utilities of all players, whereas, TEL has the property to spend a high proportion of time in an optimum state that maximizes the sum of the utilities of all players if there is a pure Nash equilibrium, otherwise, it spends a high proportion of time in a state that maximizes a trade-off between the sum of the utilities of the players and a predefined stability function. On the other hand, estimating the mean fraction of time spent in the optimum state (as well as the mean time duration to reach it) is challenging due to the high complexity and dimension of the inherent Markov chains. In this article, under some specific system model, an evaluation of the above performance metrics is provided by proposing an approximation of the considered Markov chains, which allows overcoming the problem of high dimensionality. A comparison between the two algorithms is then performed which allows a better understanding of their performance.

Asynchronous broadcast-based decentralized learning in sensor networks

In this paper, we study the problem of decentralized learning in sensor networks in which local learners estimate and reach consensus to the quantity of interest inferred globally while communicating only with their immediate neighbours. The main challenge lies in reducing the communication cost in the network, which involves inter-node synchronisation and data exchange. To address this issue, a novel asynchronous broadcast-based decentralized learning algorithm is proposed. Furthermore, we prove that the iterates generated by the developed decentralized method converge to a consensual optimal solution (model). Numerical results demonstrate that it is a promising approach for decentralized learning in sensor networks.

Performance Optimization for Cooperative Multiuser Cognitive Radio Networks with RF Energy Harvesting Capability

We study the performance optimization problem for a cognitive radio network with radio frequency (RF) energy harvesting capability for secondary users. In such networks, the secondary users are able to not only transmit packets on a channel licensed to a primary user when the channel is idle, but also harvest RF energy from the primary users' transmissions when the channel is busy. Specifically, we propose a system model where the secondary users are able to cooperate to maximize the overall network throughput through sensing a set of common channels. We first consider the case where the secondary users cooperate in a TDMA fashion and propose a novel solution based on a learning algorithm to find optimal channel access policies for the secondary users. Then, we examine the case where the secondary users cooperate in a decentralized manner and we formulate the cooperative decentralized optimization problem as a decentralized partially observable Markov decision process (DEC-POMDP). To solve the cooperative decentralized stochastic optimization problem, we apply a decentralized learning algorithm based on the policy gradient and the Lagrange multiplier method to obtain optimal channel access policies. Extensive performance evaluation is conducted and it shows the efficiency and the convergence of the learning algorithms.

Partially decentralized reinforcement learning in finite, multi-agent Markov decision processes

In this paper, we propose a novel, partially decentralized learning algorithm for the control of finite, multi-agent Markov Decision Process with unknown transition probabilities and reward values. One learning automaton is associated with each agent acting in a state and the automata acting within a state may communicate with each other. However, there is no communication between the automata present in different states, thus making the system partially decentralized. We propose novel algorithms so that the entire automata team converges to the policy that maximizes the long-term expected reward per step. Simulation results are presented to demonstrate the usefulness of the proposed algorithms.

User strategy learning when pricing a RED buffer

We study a buffer that implements the Random Early Detect/Discard (RED) mechanism to cope with congestion, and offers service differentiation by proposing a finite number of slopes at different prices for the RED probability. As a characteristic, the smaller the slope, the better the resulting QoS. Users are sensitive to their average throughput and to the price they pay. Since the study of the noncooperative game played is rendered difficult by the discrete nature of the strategy sets, and since it is not likely that users have a perfect knowledge of the game but only know their experienced utility, we introduce a decentralized learning algorithm to progressively reach a Nash equilibrium over time. We examine the effect of prices on the final game outcomes.