Distributed Learning System Research Articles

Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks.

In distributed learning systems, robustness threat may arise from two major sources. On the one hand, due to distributional shifts between training data and test data, the trained model could exhibit poor out-of-sample performance. On the other hand, a portion of working nodes might be subject to Byzantine attacks, which could invalidate the learning result. In this article, we propose a new research direction that jointly considers distributional shifts and Byzantine attacks. We illuminate the major challenges in addressing these two issues simultaneously. Accordingly, we design a new algorithm that equips distributed learning with both distributional robustness and Byzantine robustness. Our algorithm is built on recent advances in distributionally robust optimization (DRO) as well as norm-based screening (NBS), a robust aggregation scheme against Byzantine attacks. We provide convergence proofs in three cases of the learning model being nonconvex, convex, and strongly convex for the proposed algorithm, shedding light on its convergence behaviors and endurability against Byzantine attacks. In particular, we deduce that any algorithm employing NBS (including ours) cannot converge when the percentage of Byzantine nodes is [Formula: see text] or higher, instead of [Formula: see text] , which is the common belief in current literature. The experimental results verify our theoretical findings (on the breakpoint of NBS and others) and also demonstrate the effectiveness of our algorithm against both robustness issues, justifying our choice of NBS over other widely used robust aggregation schemes. To the best of our knowledge, this is the first work to address distributional shifts and Byzantine attacks simultaneously.

Learning with Asynchronous Labels

Learning with data streams has attracted much attention in recent decades. Conventional approaches typically assume that the feature and label of a data item can be timely observed at each round. In many real-world tasks, however, it often occurs that either the feature or the label is observed firstly while the other arrives with delay. For instance, in distributed learning systems, a central processor collects training data from different sub-processors to train a learning model, whereas the feature and label of certain data items can arrive asynchronously due to network latency. The problem of learning with asynchronous feature or label in streams encompasses many applications but still lacks sound solutions. In this article, we formulate the problem and propose a new approach to alleviate the negative effect of asynchronicity and mining asynchronous data streams. Our approach carefully exploits the timely arrived information and builds an online ensemble structure to adaptively reuse historical models and instances. We provide the theoretical guarantees of our approach and conduct extensive experiments to validate its effectiveness.

Self-adaptive asynchronous federated optimizer with adversarial sharpness-aware minimization

The past years have witnessed the success of a distributed learning system called Federated Learning (FL). Recently, asynchronous FL (AFL) has demonstrated its potential in concurrency compared to mainstream synchronous FL. However, the inherent systematic and statistical heterogeneity has presented several impediments to AFL: On the client side, the discrepancies in trips and local model drift impede global performance enhancement; On the server side, dynamic communication leads to significant fluctuations in gradient arrival time, while asynchronous arrival gradients with ambiguous value are not fully leveraged. In this paper, we propose an adaptive AFL framework, ARDAGH, which systematically addresses the aforementioned challenges: Firstly, to address the discrepancies in client trips, ARDAGH ensures their convergence by incorporating only 1-bit feedback information into the downlink. Secondly, to counter the drift of clients, ARDAGH generalizes the local models by employing our novel adversarial sharpness-aware minimization, which does not necessitate reliance on additional global variables. Thirdly, in the face of gradient latency issues, ARDAGH employs a communication-aware dropout strategy to adaptively compress gradients to ensure similar transmission times. Finally, to fully unleash the potential of each gradient, we establish a consistent optimal direction by conceptualizing the aggregation as an optimizer with successive momentum. In light of the comprehensive solution provided by ARDAGH, an algorithm named FedAMO is derived, and its superiority is confirmed by experimental results obtained under challenging prototype and simulation settings. Particularly in typical sentiment analysis tasks, FedAMO demonstrates an improvement of up to 5.351% with a 20.056-fold acceleration compared to conventional asynchronous methods.

Ontological description method for building service-oriented distributed learning systems

The subject of the research is the analysis and justification of the use of software engineering processes based on ontologies of data access and representation of access to knowledge bases and knowledge reuse in the language of algebra of finite predicates, which is a class of descriptive logics. This approach allows you to use the advantages of ontologies and logic programming in the refactoring of distributed distance learning systems. The purpose of the work is to create a formalism for describing the interaction of services and algorithms for building web service interfaces for the implementation of an effective SOA system using the paradigm of access to data based on ontologies. To achieve the goal, the following tasks were solved: an analysis of the main types of interaction structures of SOA services, an overview of implementations of systems working with a family of descriptive logic languages, namely, algebra of finite predicates, and a description of the structure of the ontology necessary for the operation of service-oriented systems, which has the markup of the Semantic Web standard using finite predicate algebra equations. The task is an urgent task of ensuring the organization of interaction of Internet resources in the development of distributed virtual learning systems, and such interaction is enabled by the service-oriented architecture of software development. The article examines modern technologies of the Semantic Web and their role in ensuring the creation of applied software systems of distributed learning resources built using distributed interactive services. The considered methods and materials include methods of algebra of finite predicates, theory of algorithms, object-oriented design, theory of unification. The results. The semantics of the SOA system is obtained, which makes it possible to describe SOA systems in the languages of descriptive logic. The effectiveness of the mathematical formalism of the algebra of finite predicates for the tasks of logical analysis of applied ontologies, the use of knowledge reuse methods, and the description of service-oriented systems is shown. The necessity of using algorithms for automatic construction of web interfaces is shown. The conclusions of the work highlight the advantages of the researched solution, namely, algorithms for the automated construction of web service interfaces for SOA architecture, which make it possible to automatically build web service interfaces. Algorithms differ in a more general model and speed of work and require an assessment of complexity.

Federated Learning for Edge Computing in REST APIs: Investigate the Feasibility of Employing Federated Learning Techniques to Train Machine Learning Models Directly on Edge Devices Interacting with REST APIs, Minimizing Data Transfer and Enhancing Privacy

This research explores the feasibility of using federated learning techniques to train machine learning models directly on edge devices interacting through REST APIs. This article designs an architecture that facilitates federated learning using REST APIs for communication between a central server and edge nodes. The architecture ensures efficient model distribution, update aggregation, and iterative global model refinement. This research implements this system in a simulated environment to evaluate its performance. Initial results show that federated learning significantly reduces data transfer and enhances privacy without compromising model accuracy. The REST API-based communication model effectively coordinates updates between the central server and edge devices, and privacy-preserving techniques maintain data security throughout the process. This study confirms that federated learning is a viable solution for edge computing, addressing key challenges in data privacy and communication efficiency, and opens pathways for scalable and effective distributed learning systems in real-world IoT applications. Federated learning offers a decentralized alternative, enabling local model training on edge devices with only model updates shared with a central server. This approach minimizes data transfer and enhances privacy.

Enhancing Earth data analysis in 5G satellite networks: A novel lightweight approach integrating improved deep learning

Efficiently handling huge data amounts and enabling processing-intensive applications to run in faraway areas simultaneously is the ultimate objective of 5G networks. Currently, in order to distribute computing tasks, ongoing studies are exploring the incorporation of fog-cloud servers onto satellites, presenting a promising solution to enhance connectivity in remote areas. Nevertheless, analyzing the copious amounts of data produced by scattered sensors remains a challenging endeavor. The conventional strategy of transmitting this data to a central server for analysis can be costly. In contrast to centralized learning methods, distributed machine learning (ML) provides an alternative approach, albeit with notable drawbacks. This paper addresses the comparative learning expenses of centralized and distributed learning systems to tackle these challenges directly. It proposes the creation of an integrated system that harmoniously merges cloud servers with satellite network structures, leveraging the strengths of each system. This integration could represent a major breakthrough in satellite-based networking technology by streamlining data processing from remote nodes and cutting down on expenses. The core of this approach lies in the adaptive tailoring of learning techniques for individual entities based on their specific contextual nuances. The experimental findings underscore the prowess of the innovative lightweight strategy, LMAED2L (Enhanced Deep Learning for Earth Data Analysis), across a spectrum of machine learning assignments, showcasing remarkable and consistent performance under diverse operational conditions. Through a strategic fusion of centralized and distributed learning frameworks, the LMAED2L method emerges as a dynamic and effective remedy for the intricate data analysis challenges encountered within satellite networks interfaced with cloud servers. The empirical findings reveal a significant performance boost of our novel approach over traditional methods, with an average increase in reward (4.1 %), task completion rate (3.9 %), and delivered packets (3.4 %). This report suggests that these advancements will catalyze the integration of cutting-edge machine learning algorithms within future networks, elevating responsiveness, efficiency, and resource utilization to new heights.

Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers

Byzantine machine learning has garnered considerable attention in light of the unpredictable faults that can occur in large-scale distributed learning systems. The key to secure resilience against Byzantine machines in distributed learning is resilient aggregation mechanisms. Although abundant resilient aggregation rules have been proposed, they are designed in ad-hoc manners, imposing extra barriers on comparing, analyzing, and improving the rules across performance criteria. This paper studies near-optimal aggregation rules using clustering in the presence of outliers. Our outlier-robust clustering approach utilizes geometric properties of the update vectors provided by workers. Our analysis show that constant approximations to the 1-center and 1-mean clustering problems with outliers provide near-optimal resilient aggregators for metric-based criteria, which have been proven to be crucial in the homogeneous and heterogeneous cases respectively. In addition, we discuss two contradicting types of attacks under which no single aggregation rule is guaranteed to improve upon the naive average. Based on the discussion, we propose a two-phase resilient aggregation framework. We run experiments for image classification using a non-convex loss function. The proposed algorithms outperform previously known aggregation rules by a large margin with both homogeneous and heterogeneous data distributions among non-faulty workers. Code and appendix are available at https://github.com/jerry907/AAAI24-RASHB.

Attack-model-agnostic defense against model poisonings in distributed learning

The distributed nature of distributed learning renders the learning process susceptible to model poisoning attacks. Most existing countermeasures are designed based on a presumed attack model, and can only perform under the presumed attack model. However, in reality a distributed learning system typically does not have the luxury of knowing the attack model it is going to be actually facing in its operation when the learning system is deployed, thus constituting a zero-day vulnerability of the system that has been largely overlooked so far. In this paper, we study the attack-model-agnostic defense mechanisms for distributed learning, which are capable of countering a wide-spectrum of model poisoning attacks without relying on assumptions of the specific attack model, and hence alleviating the zero-day vulnerability of the system. Extensive experiments are performed to verify the effectiveness of the proposed defense.

Privacy preserving vertical distributed learning for health data

Federated learning has become a pivotal tool in healthcare, enabling valuable insights to be gleaned from disparate datasets held by cautious data owners concerned about data privacy. This method involves the analysis of data from diverse locations, which is subsequently aggregated and trained on a central server. Data distribution can occur vertically or horizontally in this decentralized setup. In our approach, we employ a unique vertical partition learning process, segmenting data by characteristics or columns for each record across all local sites, known as Vertical Distributed Learning or features distributed machine learning. Our collaborative learning approach utilizes Stochastic Gradient Descent to collectively learn from each local site and compute the final result on a central server. Notably, during the training phase, no raw data or model parameters are exchanged; only local prediction results are shared and aggregated. Yet, sharing local prediction results raises privacy concerns, which we mitigate by introducing noise into the local results using a Differential Privacy algorithm. This paper introduces a robust vertical distributed learning system that emphasizes user privacy for healthcare data. To assess our approach, we conducted experiments using the sensitive healthcare data in the Medical Information Mart for Intensive Care-Ⅲ dataset and the publicly available Adult dataset. Our experimental results demonstrate that our approach achieves an accuracy level similar to that of a fully centralized model, significantly surpassing training based solely on local features. Consequently, our solution offers an effective federated learning approach for healthcare, preserving data locality and privacy while efficiently harnessing vertically partitioned data.

Self-Organizing Democratized Learning: Toward Large-Scale Distributed Learning Systems.

Emerging cross-device artificial intelligence (AI) applications require a transition from conventional centralized learning systems toward large-scale distributed AI systems that can collaboratively perform complex learning tasks. In this regard, democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. The outlined principles are meant to study a generalization in distributed learning systems that go beyond existing mechanisms such as federated learning (FL). Moreover, such learning systems rely on hierarchical self-organization of well-connected distributed learning agents who have limited and highly personalized data and can evolve and regulate themselves based on the underlying duality of specialized and generalized processes. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this article. The approach consists of a self-organizing hierarchical structuring mechanism based on agglomerative clustering, hierarchical generalization, and corresponding learning mechanism. Subsequently, hierarchical generalized learning problems in recursive forms are formulated and shown to be approximately solved using the solutions of distributed personalized learning problems and hierarchical update mechanisms. To that end, a distributed learning algorithm, namely DemLearn, is proposed. Extensive experiments on benchmark MNIST, Fashion-MNIST, FE-MNIST, and CIFAR-10 datasets show that the proposed algorithm demonstrates better results in the generalization performance of learning models in agents compared to the conventional FL algorithms. The detailed analysis provides useful observations to further handle both the generalization and specialization performance of the learning models in Dem-AI systems.

Privacy and Efficiency of Communications in Federated Split Learning

Everyday, large amounts of sensitive data is distributed across mobile phones, wearable devices, and other sensors.Traditionally, these enormous datasets have been processed on a single system, with complex models being trained to make valuable predictions. Distributed machine learning techniques such as Federated and Split Learning have recently been developed to protect user data and privacy better while ensuring high performance. Both of these distributed learning architectures have advantages and disadvantages. In this paper, we examine these tradeoffs and suggest a new hybrid Federated Split Learning architecture that combines the efficiency and privacy benefits of both. Our evaluation demonstrates how our hybrid Federated Split Learning approach can lower the amount of processing power required by each client running a distributed learning system, reduce training and inference time while keeping a similar accuracy. We also discuss the resiliency of our approach to deep learning privacy inference attacks and compare our solution to other recently proposed benchmarks.

Weighted aggregation through probability based ranking: An optimized federated learning architecture to classify respiratory diseases

Background and ObjectiveRespiratory Diseases are one of the leading chronic illnesses in the world according to the reports by World Health Organization. Diagnosing these respiratory diseases is done through auscultation where a medical professional listens to sounds of air in the lungs for anomalies through a stethoscope. This method necessitates extensive experience and can also be misinterpreted by the medical professional. To address this issue, we introduce an AI-based solution that listens to the lung sounds and classifies the respiratory disease detected. Since the research work deals with medical data that is tightly under wraps due to privacy concerns in the medical field, we introduce a Deep learning solution to classify the diseases and a custom Federated learning (FL) approach to further improve the accuracy of the deep learning model and simultaneously maintain data privacy. Federated Learning architecture maintains data privacy and facilitates a distributed learning system for medical infrastructures.MethodsThe approach utilizes Generative Adversarial Networks (GAN) based Federated learning approach to ensure data privacy. Generative Adversarial Networks generate new data by synthesizing new lung sounds. This new synthesized data is then converted to spectrograms and trained on a neural network to classify four lung diseases, Heart Attack and Normal breathing patterns. Furthermore, to address performance loss during FL, we also propose a new “Weighted Aggregation through Probability-based Ranking (FedWAPR)” algorithm for optimizing the FL aggregation process. The FedWAPR aggregation takes inspiration from exponential distribution function and ranks better performing clients according to it.Results and ConclusionA test accuracy of about 92% was achieved by the trained model while classifying various respiratory diseases and heart failure. Additionally, we developed a novel FedWAPR approach that significantly outperformed the FedAVG approach for the FL aggregate function. A patient can be checked for respiratory diseases using this improved learning approach without the need for extensive sensitive data recording or for making sure the data sample obtained is secure. In a decentralized training runtime, the trained model successfully classifies various respiratory diseases and heart failure using lung sounds with a test accuracy on par with a centralized model.

Over-the-Air Computation for Distributed Systems: Something Old and Something New

Facing the upcoming era of Internet-of-Things and connected intelligence, efficient information processing, computation, and communication design becomes a key challenge in large-scale intelligent systems. Recently, Over-the-Air (OtA) computation has been proposed for data aggregation and distributed computation of functions over a large set of network nodes. Theoretical foundations for this concept exist for a long time, but it was mainly investigated within the context of wireless sensor networks. There are still many open questions when applying OtA computation in different types of distributed systems where modern wireless communication technology is applied. In this article, we provide a comprehensive overview of the OtA computation principle and its applications in distributed learning, control, and inference systems, for both server-coordinated and fully decentralized architectures. Particularly, we highlight the importance of the statistical heterogeneity of data and wireless channels, the temporal evolution of model updates, and the choice of performance metrics, for the communication design in OtA federated learning (FL) systems. Several key challenges in privacy, security, and robustness aspects of OtA FL are also identified for further investigation.

EIDLS: An Edge-Intelligence-Based Distributed Learning System Over Internet of Things

With the rapid development of wireless sensor networks (WSNs) and the Internet of Things (IoT), increasing computing tasks are sinking to mobile edge networks, such as distributed learning systems. These systems benefit from the massive amounts of data and computing power on mobile devices and can learn qualified models on the premise of protecting user privacy. In fact, coordinating mobile devices to participate in computing is challenging. On the one hand, the heterogeneous performance of devices makes it difficult to guarantee computing efficiency. On the other hand, there are unreliable factors in the mobile network, which will destroy the stability of the distributed learning. Therefore, we design a three-layer framework called an edge-intelligence-based distributed learning system (EIDLS). Specifically, a novel multilayer perceptron-based device availability evaluation model is proposed to select devices with good performance. The evaluation model performs online learning and optimization according to the resources (CPU, battery, etc.) of devices. Meanwhile, we propose a dynamic trust evaluation algorithm to reduce the side effects of unreliable devices. The experimental results of some commonly used datasets validate that the proposed EIDLS dramatically minimizes the energy consumption and communication cost and improves the calculation accuracy and the stability of the system.

Modeling the Training Iteration Time for Heterogeneous Distributed Deep Learning Systems

Distributed deep learning systems effectively respond to the increasing demand for large-scale data processing in recent years. However, the significant investment in building distributed learning systems with powerful computing nodes places a huge financial burden on developers and researchers. It will be good to predict the precise benefit, i.e., how many times of speedup it can get compared with training on single machine (or a few), before actually building such big learning systems. To address this problem, this paper presents a novel performance model on training iteration time for heterogeneous distributed deep learning systems based on the characteristics of the parameter server (PS) system with bulk synchronous parallel (BSP) synchronization style. The accuracy of our performance model is demonstrated by comparing real measurement results on TensorFlow when training different neural networks with various kinds of hardware testbeds: the prediction accuracy is higher than 90% in most cases.

Predicting Model Training Time to Optimize Distributed Machine Learning Applications

Despite major advances in recent years, the field of Machine Learning continues to face research and technical challenges. Mostly, these stem from big data and streaming data, which require models to be frequently updated or re-trained, at the expense of significant computational resources. One solution is the use of distributed learning algorithms, which can learn in a distributed manner, from distributed datasets. In this paper, we describe CEDEs—a distributed learning system in which models are heterogeneous distributed Ensembles, i.e., complex models constituted by different base models, trained with different and distributed subsets of data. Specifically, we address the issue of predicting the training time of a given model, given its characteristics and the characteristics of the data. Given that the creation of an Ensemble may imply the training of hundreds of base models, information about the predicted duration of each of these individual tasks is paramount for an efficient management of the cluster’s computational resources and for minimizing makespan, i.e., the time it takes to train the whole Ensemble. Results show that the proposed approach is able to predict the training time of Decision Trees with an average error of 0.103 s, and the training time of Neural Networks with an average error of 21.263 s. We also show how results depend significantly on the hyperparameters of the model and on the characteristics of the input data.

Byzantine-Robust Distributed Learning With Compression

Communication between workers and the master node to collect local stochastic gradients is a key bottleneck in a large-scale distributed learning system. Various recent works have proposed to compress the local stochastic gradients to mitigate the communication overhead. However, robustness to malicious attacks is rarely considered in such a setting. In this work, we investigate the problem of Byzantine-robust compressed distributed learning, where the attacks from Byzantine workers can be arbitrarily malicious. We theoretically point out that different to the attacks-free compressed stochastic gradient descent (SGD), its vanilla combination with geometric median-based robust aggregation seriously suffers from the compression noise in the presence of Byzantine attacks. In light of this observation, we propose to reduce the compression noise with gradient difference compression so as to improve the Byzantine-robustness. We also observe the impact of the intrinsic stochastic noise caused by selecting random samples, and adopt the stochastic average gradient algorithm (SAGA) to gradually eliminate the inner variations of regular workers. We theoretically prove that the proposed algorithm reaches a neighborhood of the optimal solution at a linear convergence rate, and the asymptotic learning error is in the same order as that of the state-of-the-art uncompressed method. Finally, numerical experiments demonstrate the effectiveness of the proposed method.

Communication-efficient federated continual learning for distributed learning system with Non-IID data

Communication-efficient federated continual learning for distributed learning system with Non-IID data

Differential Privacy Meets Federated Learning Under Communication Constraints

The performance of federated learning systems is bottlenecked by communication costs and training variance. The communication overhead problem is usually addressed by three communication-reduction techniques, namely, model compression, partial device participation, and periodic aggregation, at the cost of increased training variance. Different from traditional distributed learning systems, federated learning suffers from data heterogeneity (since the devices sample their data from possibly different distributions), which induces additional variance among devices during training. Various variance-reduced training algorithms have been introduced to combat the effects of data heterogeneity, while they usually cost additional communication resources to deliver necessary control information. Additionally, data privacy remains a critical issue in FL and, thus, there have been attempts at bringing Differential Privacy to this framework as a mediator between utility and privacy requirements. This article investigates the tradeoffs between communication costs and training variance under a resource-constrained federated system theoretically and experimentally, and studies how communication reduction techniques interplay in a differentially private setting. The results provide important insights into designing practical privacy-aware federated learning systems.

Common belief multi-agent reinforcement learning based on variational recurrent models

The tacit cooperation among human teams benefits from the fact that consensus can be reached on a task through common belief. Similar to human social groups, agents in distributed learning systems can also rely on common belief to achieve cooperation under the condition of limited communication. In this paper, we show the role of common belief among agents in completing cooperative tasks, by proposing the Common Belief Multi-Agent (CBMA) reinforcement learning method. CBMA is a novel value-based method that infers the belief between agents with a variational model and models the environment with a variational recurrent neural network. We validate CBMA on two grid-world games as well as the StarCraft II micromanagement benchmark. Experimental results show that the learned common belief by CBMA can improve performance in both discrete and continuous state settings.