Communication Bottleneck Research Articles

Model Parameter Prediction Method for Accelerating Distributed DNN Training

As the size of deep neural network (DNN) models and datasets increases, distributed training becomes popular to reduce the training time. However, a severe communication bottleneck in distributed training limits its scalability. Many methods aim to solve this communication bottleneck by reducing communication traffic, such as gradient sparsification and quantification. However, these methods either are at the expense of losing model accuracy or introducing lots of computing overhead. We observe that the data distribution between layers of neural network models is similar. Thus, we propose a model parameter prediction method (MP2) to accelerate distributed DNN training under parameter server (PS) framework, where workers push only a subset of model parameters to PS, and residual model parameters are locally predicted by an already-trained deep neural network model on PS. Some key challenges are addressed. First, we build a hierarchical parameters dataset by randomly sampling a subset of model from normal distributed trainings. Second, we build a neural network model with the structure of “convolution + channel attention + Max pooling” for predicting model parameters by using a prediction result-based evaluation method. For VGGNet, ResNet, and AlexNet models on CIFAR10 and CIFAR100 datasets, compared with Baseline, Top-k, deep gradient compression (DGC), and weight nowcaster network (WNN), MP2 can reduce traffic by up to 88.98%; and accelerates the training by up to 47.32% while not losing the model accuracy. MP2 has shown good generalization.

Federated learning-based natural language processing: a systematic literature review

Federated learning (FL) is a decentralized machine learning (ML) framework that allows models to be trained without sharing the participants’ local data. FL thus preserves privacy better than centralized machine learning. Since textual data (such as clinical records, posts in social networks, or search queries) often contain personal information, many natural language processing (NLP) tasks dealing with such data have shifted from the centralized to the FL setting. However, FL is not free from issues, including convergence and security vulnerabilities (due to unreliable or poisoned data introduced into the model), communication and computation bottlenecks, and even privacy attacks orchestrated by honest-but-curious servers. In this paper, we present a systematic literature review (SLR) of NLP applications in FL with a special focus on FL issues and the solutions proposed so far. Our review surveys 36 recent papers published in relevant venues, which are systematically analyzed and compared from multiple perspectives. As a result of the survey, we also identify the most outstanding challenges in the area.

Stabilizing and Accelerating Federated Learning on Heterogeneous Data With Partial Client Participation.

Federated learning (FL) commonly encourages the clients to perform multiple local updates before the global aggregation, thus avoiding frequent model exchanges and relieving the communication bottleneck between the server and clients. Though empirically effective, the negative impact of multiple local updates on the stability of FL is not thoroughly studied, which may result in a globally unstable and slow convergence. Based on sensitivity analysis, we define in this paper a local-update stability index for the general FL, as measured by the maximum inter-client model discrepancy after the multiple local updates that mainly stems from the data heterogeneity. It enables to determine how much the variation of client's models with multiple local updates may influence the global model, and can also be linked with the convergence and generalization. We theoretically derive the proposed local-update stability for current state-of-the-art FL methods, providing possible insight to understanding their motivation and limitation from a new perspective of stability. For example, naively executing the parallel acceleration locally at clients would harm the local-update stability. Motivated by this, we then propose a novel accelerated yet stabilized FL algorithm (named FedANAG) based on the server- and client-level Nesterov accelerated gradient (NAG). In FedANAG, the global and local momenta are elaborately designed and alternatively updated, while the stability of local update is enhanced with help of the global momentum. We prove the convergence of FedANAG for strongly convex, general convex and non-convex settings. We then conduct evaluations on both the synthetic and real-world datasets to first validate our proposed local-update stability. The results further show that across various data heterogeneity and client participation ratios, FedANAG not only accelerates the global convergence by reducing the required number of communication rounds to a target accuracy, but converges to an eventually higher accuracy.

Investigation of Optical Interconnects for nano-scale VLSI applications

The relentless trend toward miniaturization has brought interconnects to the brink of communication bottlenecks, operating at or near their ampacity (current carrying capacity) limits. This degradation in performance necessitates novel technologies with increased ampacity. This study explores optical interconnect (OI) as a potential future technology, offering high-bandwidth communication and low latency. This study models and simulates OI, integrating recent optical device advancements with modest modulator and detector capacitance (50 fF). Performance metrics including signal delay, power dissipation, and power-delay-product (PDP) are compared across copper (Cu) and single-wall carbon nanotube bundle (SWCNT-B) interconnects at 22 nm and 14 nm technology nodes. Results consistently show OI, with an active voltage current feedback (AVCF) based regulated gain cascode (RGC) transimpedance amplifier (TIA) receiver, outperforms Cu and SWCNT-B interconnects at global lengths and beyond. For instance, at 1000 μm and 22 nm, OI exhibits 88.47 % and 62.15 % delay improvements over Cu and SWCNT-B, respectively. This trend persists at 14 nm, with OI showing 93.68 % and 84.29 % delay improvements, respectively. Additionally, OI surpasses Cu and SWCNT-B in power efficiency beyond a critical length. As interconnects extend to global scales and beyond, the differences in delay, power, and PDP between OI and electrical interconnect increases, highlighting OI's advantages for future VLSI IC applications.

An adaptive asynchronous federated learning framework for heterogeneous Internet of things

Federated learning (FL) is a distributed machine learning framework that enables the training of shared models without the need to share local data. However, FL faces challenges in heterogeneous Internet of Things (IoT) environments, including communication bottleneck, staleness, and non- independent and identically distributed (non-IID) data. To tackle these challenges, we present ASAFL, an Asynchronous Federated Learning framework with an Adaptive Scheduling Strategy. Firstly, we quantify the potential contribution of client models relative to the server model. Secondly, we design a client upload strategy to reduce the uploading of redundant models with low contribution. Finally, we propose a server model update method based on the contributions to address model divergence caused by staleness and non-IID data. Furthermore, our theoretical analysis guarantees ASAFL's convergence, and experiments show it reduces communication overhead by over 70% compared to traditional asynchronous FL.

An intelligent fuzzy reinforcement learning-based routing algorithm with guaranteed latency and bandwidth in SDN: Application of video conferencing services

Online video conferencing services connection by multipoint control units leads to communication bottleneck and high latency. Due to the dynamic traffic in routing connection requests to video conferences, the existing routing algorithms face problems such as communication bottlenecks, high latency, congestion, path redundancy, and local optimum. In recent years, the implementation of video conferencing systems using Software-Defined Networking (SDN) provides new opportunities to improve Quality of Service (QoS) with low latency. In this paper, an Intelligent Fuzzy reinforcement learning-based Routing Algorithm is proposed, which simultaneously Guarantees Latency and Bandwidth for online video conferencing services in SDN (named IFRA-GLB). Fuzzy logic mainly performs online routing between an ingress–egress pair. Meanwhile, through ongoing training, reinforcement learning lowers the average number of hops on the path determined by fuzzy model. Enhancements to the convergence capability of IFRA and reduced reliance on network topology are achieved by adjusting link weights using a weighted shortest path algorithm focused on critical nodes. When the network encounters congestion, a deferral module is applied to assign higher priority to requests with lower resource demands. Experimental results demonstrate that IFRA-GLB significantly improves performance and convergence compared to existing solutions, enhancing the QoS for video conferencing services. Specifically, IFRA-GLB increases the average admission rate by 1.96% for MIRA topology and 2.71% for ANSNET topology with 3.5% lower latency.

Adaptive task migration strategy with delay risk control and reinforcement learning for emergency monitoring

The timely and reliable handling of post-disaster emergency monitoring tasks is crucial for effective rescue operations. UAV-assisted edge computing plays a pivotal role in the rapid deployment of such systems. However, challenges persist due to communication and computation resource bottlenecks when dealing with delay-sensitive monitoring tasks. In dynamic post-disaster environments, effective task scheduling and resource allocation decisions directly impact the system’s ability to process tasks. Therefore, this paper proposes an adaptive task migration decision-making system for emergency monitoring tasks in UAV-assisted edge computing. Firstly, We decomposed the optimization objectives based on the task processing workflow, then devised a stepwise delay risk control and resource recovery mechanism based on early discarding. Secondly, by integrating multi-agent reinforcement learning (MARL), optimal strategies for task offloading, UAV queue scheduling, and communication resource allocation are learned to enhance the decision system’s environmental awareness and maximize the successful completion of emergency monitoring tasks. Simulation experiments demonstrate that the algorithm significantly improves the success rate of migration tasks and data processing capacity, thereby validating its convergence and effectiveness.

Robust and Privacy-Preserving Decentralized Deep Federated Learning Training: Focusing on Digital Healthcare Applications.

Federated learning of deep neural networks has emerged as an evolving paradigm for distributed machine learning, gaining widespread attention due to its ability to update parameters without collecting raw data from users, especially in digital healthcare applications. However, the traditional centralized architecture of federated learning suffers from several problems (e.g., single point of failure, communication bottlenecks, etc.), especially malicious servers inferring gradients and causing gradient leakage. To tackle the above issues, we propose a robust and privacy-preserving decentralized deep federated learning (RPDFL) training scheme. Specifically, we design a novel ring FL structure and a Ring-Allreduce-based data sharing scheme to improve the communication efficiency in RPDFL training. Furthermore, we improve the process of distributing parameters of the Chinese residual theorem to update the execution process of the threshold secret sharing, supporting healthcare edge to drop out during the training process without causing data leakage, and ensuring the robustness of the RPDFL training under the Ring-Allreduce-based data sharing scheme. Security analysis indicates that RPDFL is provable secure. Experiment results show that RPDFL is significantly superior to standard FL methods in terms of model accuracy and convergence, and is suitable for digital healthcare applications.

Large-Eddy Simulations of a Supersonic Impinging Jet Using OpenFOAM

Supersonic impinging jets are a versatile configuration that can model the compressible flows of cold-spray manufacturing and vertical take-off-and landing strategy. In this work, rhoCentralFoam, solver of the OpenFOAM framework, and a large-eddy simulation formulation were used to simulate an underexpanded supersonic jet of Mach 1.45 and nozzle pressure ratio of 4, impinging on a flat wall situated at 1.5 nozzle diameters away from the jet outlet. Care was taken in the mesh construction to properly capture the characteristic standoff shock and vortical structures. The grid convergence index was evaluated with three meshes of increasing spatial resolution. All meshes can generally be considered as sufficient in terms of results focused on time-averaged values and mean physical properties such as centerline Mach number profile. However, the highest resolution mesh was found to capture fine shear vortical structures and behaviors that are absent in the coarser cases. Therefore, the notion of adequate grid convergence may differ between analyses of time-averaged and transient information, and so should be determined by the user’s intention for conducting the simulations. To guide the selection of mesh resolution, scaling analyses were performed, for which the current rhoCentralFoam solver displays a good weak scaling performance and maintains a linear strong scaling up to 4096 cores (32 nodes) for an approximately 40 million-cell mesh. Due to the internode communication bottlenecks of OpenFOAM and improvements in central processing units, this work recommends, for future scaling analyses, adopting a “cells-per-node” basis over the conventional “cells-per-core” basis, with particular attention to the interconnect speed and architecture used.

Reliable incentive mechanism in hierarchical federated learning based on two-way reputation and contract theory

Hierarchical federated learning (HFL) can effectively alleviate the communication bottleneck of traditional federated learning. However, the long-term healthy development of federated learning needs to continue to attract reliable participants with high-quality data. Therefore, we propose a novel reliable incentive mechanism based on two-way reputation and contract theory. Firstly, a two-way reputation mechanism named TWRM is introduced to evaluate the reliability of cluster heads and mobile devices simultaneously. Secondly, a reputation blockchain is designed to prevent malicious nodes from tampering with reputation scores, and effectively deal with malicious edge servers colluding with mobile devices. Finally, contract theory is used to reward customers based on data quality and encourage them to provide high-quality training data. Experimental results can verify that the scheme proposed in this paper effectively ensures the reliability and data quality of the participants, which can help to attract more reliable participants with high-quality data to join in federated learning.

Towards Robust Multi-Agent Reinforcement Learning

Stochastic gradient descent (SGD) is at the heart of large-scale distributed machine learning paradigms such as federated learning (FL). In these applications, the task of training high-dimensional weight vectors is distributed among several workers that exchange information over networks of limited bandwidth. While parallelization at such an immense scale helps to reduce the computational burden, it creates several other challenges: delays, asynchrony, and most importantly, a significant communication bottleneck. The popularity and success of SGD can be attributed in no small part to the fact that it is extremely robust to such deviations from ideal operating conditions. Inspired by these findings, we ask: Are common reinforcement learning (RL) algorithms also robust to similarly structured perturbations? Perhaps surprisingly, despite the recent surge of interest in multi-agent/federated RL, almost nothing is known about the above question. This paper collects some of our recent results in filling this void.

A Study on the International Communication of Heluo Culture from the Perspective of Digital Humanities

Heluo culture is the submission of material and spiritual culture centered in Luoyang where the Yellow River and Luo River meet. It occupies an important position in the 5, 000-year history of Chinese civilization. With the increasing cultural exchanges and communications at home and abroad, the research of Heluo culture is breaking through regional restrictions and gradually going to the whole country and overseas. Digital humanities are a new research method that applies digital technology to humanities studies by using computer technology as the path. From the perspective of digital humanities, this paper analyzes the bottleneck of international communication of Heluo culture, studies the new way of international communication of Heluo culture based on Internet +, and makes contributions to Chinese culture going out.

Generative artificial intelligence for distributed learning to enhance smart grid communication

Machine learning models are the backbone of smart grid optimization, but their effectiveness hinges on access to vast amounts of training data. However, smart grids face critical communication bottlenecks due to the ever-increasing volume of data from distributed sensors. This paper introduces a novel approach leveraging Generative Artificial Intelligence (GenAI), specifically a type of pre-trained Foundation Model (FM) architecture suitable for time series data due to its efficiency and privacy-preserving properties. These GenAI models are distributed to agents, or data holders, empowering them to fine-tune the foundation model with their local datasets. By fine-tuning the foundation model, the updated model can produce synthetic data that mirrors real-world grid conditions. The server aggregates fine-tuned model from all agents and then generates synthetic data which considers all data collected in the grid. This synthetic data can be used to train global machine learning models for specific tasks like anomaly detection and energy optimization. Then, the trained task models are distributed to agents in the grid to leverage them. The paper highlights the advantages of GenAI for smart grid communication, including reduced communication burden, enhanced privacy through anonymized data transmission, and improved efficiency and scalability. By enabling a distributed and intelligent communication architecture, GenAI introduces a novel way for a more secure, efficient, and sustainable energy future.

Accurate implementation of two-level asynchronous domain decomposition solvers

Recently, asynchronous coarse-grid correction has been achieved within additive Schwarz-type and primal Schur domain decomposition frameworks. Both additive and multiplicative coarse-grid corrections were discussed, however, the implemented asynchronous Schwarz-type solver with additive correction relies on the specific design of the restricted additive Schwarz (RAS) method, and also requires an overlap between the subsets of unknowns. In this paper, we first highlight a gap between the theoretical analysis from the literature and the associated RAS implementation. It turns out that communications delays would actually need to be bounded in order to fit the theory. This has to be stressed since, despite the asynchronous nature of the solver, the coarse-grid correction requires non-blocking global synchronization, which is subject to communications bottleneck. Second, we propose an implementation approach which applies to a wider class of additive Schwarz-type methods while still coping with the bounded delays requirement.

Research on Communication Strategy of Automobile Culture under the Background of Media Convergence

Instantly, as an important tool that people can't live without, cars are gaining more attention and attraction in the context of media integration. At the same time, with the continuous development of new media technology, automobile cultural communication, as a new type of cultural communication with automobiles as the main communication content, has also ushered in a huge development opportunity. In the context of media integration, this paper will focus on the analysis of the characteristics, communication bottlenecks and communication strategies of automobile culture communication, taking the "two races and one exhibition" of China automotive information technology as an example. I hope it can provide some suggestions and thoughts for other related research.

Tucker Tensor Approach for Accelerating Fock Exchange Computations in a Real-Space Finite-Element Discretization of Generalized Kohn-Sham Density Functional Theory.

The evaluation of Fock exchange is often the computationally most expensive part of hybrid functional density functional theory calculations in a systematically improvable, complete basis. In this work, we employ a Tucker tensor based approach that substantially accelerates the evaluation of the action of Fock exchange by transforming three-dimensional convolutional integrals into a tensor product of one-dimensional convolution integrals. Our numerical implementation uses a parallelization strategy that balances the memory and communication bottlenecks, alongside overlapping compute and communication operations to enhance computational efficiency and parallel scalability. The accuracy and computational efficiency are demonstrated on various systems, including Pt clusters of various sizes and a TiO2 cluster with 3684 electrons.

A Communication-Efficient Federated Learning Framework for Sustainable Development Using Lemurs Optimizer

The pressing need for sustainable development solutions necessitates innovative data-driven tools. Machine learning (ML) offers significant potential, but faces challenges in centralized approaches, particularly concerning data privacy and resource constraints in geographically dispersed settings. Federated learning (FL) emerges as a transformative paradigm for sustainable development by decentralizing ML training to edge devices. However, communication bottlenecks hinder its scalability and sustainability. This paper introduces an innovative FL framework that enhances communication efficiency. The proposed framework addresses the communication bottleneck by harnessing the power of the Lemurs optimizer (LO), a nature-inspired metaheuristic algorithm. Inspired by the cooperative foraging behavior of lemurs, the LO strategically selects the most relevant model updates for communication, significantly reducing communication overhead. The framework was rigorously evaluated on CIFAR-10, MNIST, rice leaf disease, and waste recycling plant datasets representing various areas of sustainable development. Experimental results demonstrate that the proposed framework reduces communication overhead by over 15% on average compared to baseline FL approaches, while maintaining high model accuracy. This breakthrough extends the applicability of FL to resource-constrained environments, paving the way for more scalable and sustainable solutions for real-world initiatives.

FOLD: Fog-dew infrastructure-aided optimal workload distribution for cloud robotic operations

In our fast-paced, technology-driven world, multi-robot systems have emerged as crucial solutions to tackle contemporary challenges, from industrial automation to disaster response, especially where the scope of human interventions is significantly constrained. In such scenarios, a notable number of event-driven operations trigger robots to perform a substantial amount of tasks. Nonetheless, completion of the tasks proves challenging due to the limited computational capabilities inherent to many robotic systems. Although cloud computing solutions can be integrated to address these limitations by distributing the workload to clouds, ensuring optimized performance remains a formidable challenge due to the communication bottlenecks encountered by the robots. Moreover, the presence of robots’ energy constraints and stringent real-time service requirements further exacerbate this workload distribution problem. In response to the aforementioned challenges, this paper introduces a fog-dew-enabled robotic system designed to mitigate latency and energy consumption while orchestrating crucial workload distribution decisions among robots. The execution of decision-making tasks is conceptualized as a multi-objective optimization problem. Due to the NP-hardness of the multi-objective optimization, we propose an innovative solution based on a meta-heuristic Binary Particle Swarm Optimization algorithm. Through rigorous experimentation conducted via simulation within the iFogSim2 simulator, our results demonstrate that the proposed algorithm surpasses the current state-of-the-art in simultaneously optimizing latency and energy consumption.

An efficient bandwidth-adaptive gradient compression algorithm for distributed training of deep neural networks

In distributed deep learning with data parallelism, communication bottleneck throttles the efficiency of model training. Recent studies adopt versatile gradient compression techniques, with communication sparsification standing out as an effective approach for reducing the number of gradients to be transmitted. However, the deployment of gradient sparsification is adversely influenced by the change of network environment in real systems, and existing methods either neglect bandwidth dynamics during training or experience drastic fluctuation of compression ratios. In this paper, we propose ACE, a novel adaptive gradient compression mechanism with high communication efficiency under bandwidth variation. ACE adapts the sparsification ratio to the average bandwidth in a time window, other than following its dynamics exactly. To accurately compute the compression ratio, we first profile the compression time and model a single iteration time consisting of communication, computation and compression operations. We then present a practical model to fit the needed training rounds till convergence, and formulate an optimization problem to compute the optimal sparsification ratio. We conduct experiments on different DNN models in different network environments and compare various methods in terms of convergence and model quality. The experimental results show that ACE achieves up to 9.39× and 1.28× training speedups over fixed and state-of-the-art adaptive compression methods.

Communication-efficient distributed cubic Newton with compressed lazy Hessian

Recently, second-order distributed optimization algorithms have been becoming a research hot in distributed learning, due to their faster convergence rate than the first-order algorithms. However, second-order algorithms always suffer from serious communication bottleneck. To conquer such challenge, we propose communication-efficient second-order distributed optimization algorithms in the parameter-server framework, by incorporating cubic Newton methods with compressed lazy Hessian. Specifically, our algorithms require each worker communicate compressed Hessians with the server only at some particular iterations, which can save both communication bits and communication rounds. For non-convex problems, we theoretically prove that our algorithms can reduce the communication cost comparing to the state-of-the-art second-order algorithms, while maintaining the same iteration complexity order O(ϵ−3/2) as the centralized cubic Newton methods. By further using gradient regularization technique, our algorithms can achieve global convergence for convex problems. Moreover, for strongly convex problems, our algorithms can achieve local superlinear convergence rate without any requirement on initial conditions. Finally, numerical experiments are conducted to show the high efficiency of the proposed algorithms.