Distributed Machine Learning Algorithms Research Articles

FMDL: Federated Mutual Distillation Learning for Defending Backdoor Attacks

Federated learning is a distributed machine learning algorithm that enables collaborative training among multiple clients without sharing sensitive information. Unlike centralized learning, it emphasizes the distinctive benefits of safeguarding data privacy. However, two challenging issues, namely heterogeneity and backdoor attacks, pose severe challenges to standardizing federated learning algorithms. Data heterogeneity affects model accuracy, target heterogeneity fragments model applicability, and model heterogeneity compromises model individuality. Backdoor attacks inject trigger patterns into data to deceive the model during training, thereby undermining the performance of federated learning. In this work, we propose an advanced federated learning paradigm called Federated Mutual Distillation Learning (FMDL). FMDL allows clients to collaboratively train a global model while independently training their private models, subject to server requirements. Continuous bidirectional knowledge transfer is performed between local models and private models to achieve model personalization. FMDL utilizes the technique of attention distillation, conducting mutual distillation during the local update phase and fine-tuning on clean data subsets to effectively erase the backdoor triggers. Our experiments demonstrate that FMDL benefits clients from different data, tasks, and models, effectively defends against six types of backdoor attacks, and validates the effectiveness and efficiency of our proposed approach.

Research on distributed coding computation based on matrix multiplication

Matrix multiplication is used in machine learning to train larger, more accurate models with massive data more effectively. The distributed machine learning paradigm necessitates the distributed computing of many matrix multiplications once it is adopted. The execution time of distributed machine learning algorithms increases on dropout nodes, which are computational nodes in distributed clusters that randomly slow down computation due to a variety of factors (e.g., node failure, system failure, communication bottlenecks, etc.), becoming a significant bottleneck in distributed computing systems. It has been discovered that coded computation is less expensive than replica methods and can more effectively reduce the effects of dropped nodes. In this study, we present theoretical insights on how encoded solutions might produce significant gains over unencoded solutions.

From distributed machine to distributed deep learning: a comprehensive survey

Artificial intelligence has made remarkable progress in handling complex tasks, thanks to advances in hardware acceleration and machine learning algorithms. However, to acquire more accurate outcomes and solve more complex issues, algorithms should be trained with more data. Processing this huge amount of data could be time-consuming and require a great deal of computation. To address these issues, distributed machine learning has been proposed, which involves distributing the data and algorithm across several machines. There has been considerable effort put into developing distributed machine learning algorithms, and different methods have been proposed so far. We divide these algorithms in classification and clustering (traditional machine learning), deep learning and deep reinforcement learning groups. Distributed deep learning has gained more attention in recent years and most of the studies have focused on this approach. Therefore, we mostly concentrate on this category. Based on the investigation of the mentioned algorithms, we highlighted the limitations that should be addressed in future research.

A hybrid parallelization approach based on workers grouping algorithm

As the volume of model data increases, traditional machine learning is not able to train models efficiently, so distributed machine learning is gradually used in large-scale data training. Currently, commonly used distributed machine learning algorithms are based on data parallelism, and often use an overall synchronous parallel strategy when passing data, but using this strategy makes the overall training speed limited by the computation speed of the slower workers in the cluster. While the asynchronous parallel strategy maximizes the computational speed of the cluster, there is a delay in updating the parameters of the global model, which may lead to excessive computational errors or non-convergence of the model. In this paper, the author combines these two data delivery methods by grouping workers together and using synchronous parallelism for the workers in the group and asynchronous parallelism for the components for training. The experiment shows that the hybrid parallelism strategy can reduce the training time with guaranteed correctness.

Energy-Efficient Resource Allocation for Federated Learning in NOMA-Enabled and Relay-Assisted Internet of Things Networks

Distributed machine learning (ML) algorithms are imperative for the next-generation Internet of Things (IoT) networks, thanks to preserving the privacy of users’ data and efficient usage of the communication resources. Federated learning (FL) is a promising distributed ML algorithm where the models are trained at the edge devices over the local data sets, and only the model parameters are shared with the cloud server (CS) to generate global model parameters. Nevertheless, due to the limited battery life of the edge devices, improving the energy-efficiency is a prime concern for FL. In this work, we investigate a resource allocation scheme to reduce the overall energy consumption of FL in the relay-assisted IoT networks. We aim at minimizing the overall energy consumption of IoT devices subject to the FL time constraint. FL time consists of model training computation time and wireless transmission latency. Toward this goal, a joint optimization problem, considering scheduling the IoT devices with the relays, transmit power allocation, and computation frequency allocation, is formulated. Due to the NP-hardness of the joint optimization problem, a global optimal solution is intractable. Therefore, leveraging graph theory, joint near-optimal, and low-complexity suboptimal solutions are proposed. Efficiency of our proposed solutions over several benchmark schemes is verified via extensive simulations. Simulation results show that the proposed near-optimal scheme achieves 6, 4, and 2 times lower energy consumption, respectively, compared to the considered fixed, computation adaptation, and power adaptation schemes. Such an appealing energy efficiency comes at the cost of slightly increased FL time compared to the fixed and computation only adaptation schemes.

Byzantine Resilient Distributed Learning in Multirobot Systems

Distributed machine learning algorithms are increasingly used in multirobot systems and are prone to Byzantine attacks. In this article, we consider a distributed implementation of the stochastic gradient descent (SGD) algorithm in a cooperative network, where networked agents optimize a global loss function using SGD on the local data and aggregation of the estimates of immediate neighbors. Byzantine agents can send arbitrary estimates to their neighbors, which may disrupt the convergence of normal agents to the optimum state. We show that if every normal agent combines its neighbors’ estimates (states) such that the aggregated state is in the convex hull of its normal neighbors’ states, then the resilient convergence is guaranteed. To assure this sufficient condition, we propose a resilient aggregation rule based on the notion of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">centerpoint</i> , which is a generalization of the median in the higher-dimensional Euclidean space. We evaluate our results using examples of target pursuit and pattern recognition in multirobot systems. The evaluation results demonstrate that distributed learning with average, coordinate-wise median, and geometric median-based aggregation rules fail to converge to the optimum state, whereas the centerpoint-based aggregation rule is resilient in the same scenario.

Role of Blockchain Technology and Machine Learning in Design of Smart and Secure Warehouse Management System: A Survey

Blockchain network is defined as interconnection of many computers, and each and every computer holds the copy of the ledger. It can be observed as continuously budding chain of blocks, and blocks are interconnected with the support of hash function. Validating of new blocks is followed by a set of protocols and consensus mechanism from every node in the network. The records are kept and arranged in linear fashion chain. The main feature of the Blockchain technology is that it allows secure communication between untrusted parties without the involvement of any third party authority. Artificial intelligence, which emulates the human intelligence, is impacting heavily on the business and social media applications nowadays. Machine learning which is the subset AI, automatically learns and improve based on input data. Whereas deep learning which is subset of machine learning uses networks to identify complex patterns in data. The basic approach of machine learning is to collect and analyze the data at central location like server. But in today’s scenario the data is decentralized and emerges from multiple sources. Hence the need of distributed machine learning algorithms in many applications is required. ML can be used to make chain smarter than before. By making use of decentralized data architecture of Blockchain we can build good models of machine learning. This paper investigates the possibility of integrating Blockchain Technology and Machine learning for optimization and improvement of Warehouse operations at data and transactions levels by providing security processes needed for smart and secure warehouse system.

CON: A Computation-Oriented Network for Efficient Edge Intelligence

Edge intelligence enables edge devices to collaboratively perform computations and generate knowledge where distributed computations act as the main innovative technologies. The existing solutions for distributed computations mainly focus on the designs of computations, such as distributed machine learning algorithms. By contrast, edge intelligence poses the design requirements for efficient networking technology, including one request multiple reply in a time Interval (ORMRI), computation aggregation, and chained computation, which have not been well addressed. To satisfy those requirements, we design a Computation-Oriented Network (CON) to enable the efficient discovery, task allocation, and collaborative computations for edge devices in this article. In CON, model and datasets are afforded with the names for direct resource discovery, and a hybrid computation oriented routing (HCOR) mechanism has been proposed, which combines proactive and reactive routing approaches. Through CON, computation requests and replies can be sent to and collected from multiple suitable edge devices, respectively; computation aggregation is enabled during the computation result replying process; and chained computation is supported through sequentially popping out model names. We examine the performance of CON through analysis and implementation, which shows that the proposed CON can greatly reduce the communication bandwidth consumption compared to the existing end-to-end approach.

Spatiotemporal data partitioning for distributed random forest algorithm: Air quality prediction using imbalanced big spatiotemporal data on spark distributed framework

Spatiotemporal air quality datasets are typically collected hourly in monitoring stations deployed non-uniformly across a metropolitan city. These datasets are not only big, which poses challenges on the storage and processing capacity of centralized computing systems but also imbalanced and spatially heterogeneous, which may result in biased air quality prediction. To address these challenges, we designed and developed a parallel air quality prediction system equipped with a spatiotemporal data partitioning method, a distributed machine learning algorithm, Hadoop’s distributed data storage platform and its resource scheduler/manager, and Spark’s efficient and in-memory execution environment, which is suitable for running iterative algorithms, e.g., machine learning. Our proposed spatiotemporal partitioning method accounted for imbalance and spatial heterogeneity features of big air quality data in predictive models, which comply with the load-balancing requirement of distributed computing systems. Distributed Random Forest algorithm in the H2O library of the Spark framework was selected as the distributed machine learning algorithm to develop the air quality predictive model. This algorithm is an ensemble forest with algorithm-level adjustments to perform as efficiently as possible for big imbalanced datasets. An application of the parallel quality prediction system for Tehran, Iran showed that the parallel prediction system had considerable speedup gain and improved both the overall accuracy and class precision of air quality prediction when working with imbalanced big spatiotemporal air quality datasets. A future research direction is to add data streaming and visualization functions to the system to provide rapid and reliable air quality prediction for supporting environmental health management. • A parallel air quality prediction system based on Spark and Hadoop frameworks. • A spatiotemporal big data partitioning method based on location and year of data. • Implemented H2O Distributed Random Forest Algorithm using Sparkling Water library. • Achieved overall accuracy of 68% and precision improvements for minority classes. • Observed an increasing trend in speedup gain when adding more parallel processors.

Overview of Distributed Machine Learning Techniques for 6G Networks

The main goal of this paper is to survey the influential research of distributed learning technologies playing a key role in the 6G world. Upcoming 6G technology is expected to create an intelligent, highly scalable, dynamic, and programable wireless communication network able to serve many heterogeneous wireless devices. Various machine learning (ML) techniques are expected to be deployed over the intelligent 6G wireless network that provide solutions to highly complex networking problems. In order to do this, various 6G nodes and devices are expected to generate tons of data through external sensors, and data analysis will be needed. With such massive and distributed data, and various innovations in computing hardware, distributed ML techniques are expected to play an important role in 6G. Though they have several advantages over the centralized ML techniques, implementing the distributed ML algorithms over resource-constrained wireless environments can be challenging. Therefore, it is important to select a proper ML algorithm based upon the characteristics of the wireless environment and the resource requirements of the learning process. In this work, we survey the recently introduced distributed ML techniques with their characteristics and possible benefits by focusing our attention on the most influential papers in the area. We finally give our perspective on the main challenges and advantages for telecommunication networks, along with the main scenarios that could eventuate.

Byzantine Fault Tolerant Distributed Stochastic Gradient Descent Based on Over-the-Air Computation

Wireless distributed machine learning is envisaged to facilitate advanced learning services and applications in wireless networks consisting of devices with limited computing capability. Distributed machine learning algorithms are more vulnerable in wireless systems since information exchange in learning is limited by wireless resources and channel conditions. Moreover, their performance can be significantly degraded by attacks of the Byzantine devices, and information distorted by channel fading can be treated as Byzantine attacks. Consequently, protection of wireless distributed machine learning from Byzantine devices is paramount. Leveraging over-the-air computation, we put forth a novel wireless distributed stochastic gradient descent system which is resilient to Byzantine attacks. The proposed learning system is underpinned by two novel and distinct features which enable more accurate and faster distributed machine learning resilient to Byzantine attacks: <xref ref-type="disp-formula" rid="deqn1-deqn2" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(1)</xref> collecting training data in the PS to obtain its own training results and <xref ref-type="disp-formula" rid="deqn1-deqn2" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(2)</xref> grouping the distributed devices. We derive upper bounds of the mean square error of the global parameter when the proposed algorithms are used in the cases with and without Byzantine devices, and prove the convergence of the proposed algorithms with the derived bounds. The effectiveness of the algorithms is validated by showing the accuracy and convergence speed.

Robust Decentralized Learning Using ADMM With Unreliable Agents

Many signal processing and machine learning problems can be formulated as consensus optimization problems which can be solved efficiently via a cooperative multi-agent system. However, the agents in the system can be unreliable due to a variety of reasons: noise, faults and attacks. Providing erroneous updates leads the optimization process in a wrong direction, and degrades the performance of distributed machine learning algorithms. This paper considers the problem of decentralized learning using ADMM in the presence of unreliable agents. First, we rigorously analyze the effect of erroneous updates (in ADMM learning iterations) on the convergence behavior of multi-agent system. We show that the algorithm linearly converges to a neighborhood of the optimal solution under certain conditions and characterize the neighborhood size analytically. Next, we provide guidelines for network design to achieve a faster convergence to the neighborhood. We also provide conditions on the erroneous updates for exact convergence to the optimal solution. Finally, to mitigate the influence of unreliable agents, we propose ROAD, a robust variant of ADMM, and show its resilience to unreliable agents with an exact convergence to the optimum.

Screening hardware and volume factors in distributed machine learning algorithms on spark

Screening hardware and volume factors in distributed machine learning algorithms on spark

Design and Evaluation of a New Machine Learning Framework for IoT and Embedded Devices

Low-cost, high-performance embedded devices are proliferating and a plethora of new platforms are available on the market. Some of them either have embedded GPUs or the possibility to be connected to external Machine Learning (ML) algorithm hardware accelerators. These enhanced hardware features enable new applications in which AI-powered smart objects can effectively and pervasively run in real-time distributed ML algorithms, shifting part of the raw data analysis and processing from cloud or edge to the device itself. In such context, Artificial Intelligence (AI) can be considered as the backbone of the next generation of Internet of the Things (IoT) devices, which will no longer merely be data collectors and forwarders, but really “smart” devices with built-in data wrangling and data analysis features that leverage lightweight machine learning algorithms to make autonomous decisions on the field. This work thoroughly reviews and analyses the most popular ML algorithms, with particular emphasis on those that are more suitable to run on resource-constrained embedded devices. In addition, several machine learning algorithms have been built on top of a custom multi-dimensional array library. The designed framework has been evaluated and its performance stressed on Raspberry Pi III- and IV-embedded computers.

Optimal Task Allocation and Coding Design for Secure Edge Computing With Heterogeneous Edge Devices

In recent years, edge computing has attracted significant attention because it can effectively support many delay-sensitive applications. Despite such a salient feature, edge computing also faces many challenges, especially for efficiency and security, because edge devices are usually heterogeneous and may be untrustworthy. To address these challenges, we propose a unified framework to provide efficiency and confidentiality by coded distributed computing. Within the proposed framework, we use matrix multiplication, a fundamental building block of many distributed machine learning algorithms, as the representative computation task. To minimize resource consumption while achieving information-theoretic security, we investigate two highly-coupled problems, (1) task allocation that assigns data blocks in a computing task to edge devices and (2) linear code design that generates data blocks by encoding the original data with random information. Specifically, we first theoretically analyze the necessary conditions for the optimal solution. Based on the theoretical analysis, we develop an efficient <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">task allocation</i> algorithm to obtain a set of selected edge devices and the number of coded vectors allocated to them. Using the task allocation results, we then design <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">secure coded computing</i> schemes, for two cases, (1) with redundant computation and (2) without redundant computation, all of which satisfy the availability and security conditions. Moreover, we also theoretically analyze the optimization of the proposed scheme. Finally, we conduct extensive simulation experiments to demonstrate the effectiveness of the proposed schemes.

Online Supervised Learning with Distributed Features over Multiagent System

Most current online distributed machine learning algorithms have been studied in a data-parallel architecture among agents in networks. We study online distributed machine learning from a different perspective, where the features about the same samples are observed by multiple agents that wish to collaborate but do not exchange the raw data with each other. We propose a distributed feature online gradient descent algorithm and prove that local solution converges to the global minimizer with a sublinear rate O 2 T . Our algorithm does not require exchange of the primal data or even the model parameters between agents. Firstly, we design an auxiliary variable, which implies the information of the global features, and estimate at each agent by dynamic consensus method. Then, local parameters are updated by online gradient descent method based on local data stream. Simulations illustrate the performance of the proposed algorithm.

Dynamic parameter allocation in parameter servers

To keep up with increasing dataset sizes and model complexity, distributed training has become a necessity for large machine learning tasks. Parameter servers ease the implementation of distributed parameter management---a key concern in distributed training---, but can induce severe communication overhead. To reduce communication overhead, distributed machine learning algorithms use techniques to increase parameter access locality (PAL), achieving up to linear speed-ups. We found that existing parameter servers provide only limited support for PAL techniques, however, and therefore prevent efficient training. In this paper, we explore whether and to what extent PAL techniques can be supported, and whether such support is beneficial. We propose to integrate dynamic parameter allocation into parameter servers, describe an efficient implementation of such a parameter server called Lapse, and experimentally compare its performance to existing parameter servers across a number of machine learning tasks. We found that Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.

Distributed Graph Computation Meets Machine Learning

TuX2 is a new distributed graph engine that bridges graph computation and distributed machine learning. TuX2 inherits the benefits of elegant graph computation model, efficient graph layout, and balanced parallelism to scale to billion-edge graphs, while extended and optimized for distributed machine learning to support heterogeneity in data model, Stale Synchronous Parallel in scheduling, and a new Mini-batch, Exchange, GlobalSync, and Apply ( MEGA ) model for programming. TuX2 further introduces a hybrid vertex-cut graph optimization and supports various consistency models in fault tolerance for machine learning. We have developed a set of representative distributed machine learning algorithms in TuX2 , covering both supervised and unsupervised learning. Compared to the implementations on distributed machine learning platforms, writing those algorithms in TuX2 takes only about 25 percent of the code: our graph computation model hides the detailed management of data layout, partitioning, and parallelism from developers. The extensive evaluation of TuX2 , using large datasets with up to 64 billion of edges, shows that TuX2 outperforms PowerGraph/PowerLyra, the state-of-the-art distributed graph engines, by an order of magnitude, while beating two state-of-the-art distributed machine learning systems by at least 60 percent.

Distributed learning on 20 000+ lung cancer patients – The Personal Health Train

Background and purposeAccess to healthcare data is indispensable for scientific progress and innovation. Sharing healthcare data is time-consuming and notoriously difficult due to privacy and regulatory concerns. The Personal Health Train (PHT) provides a privacy-by-design infrastructure connecting FAIR (Findable, Accessible, Interoperable, Reusable) data sources and allows distributed data analysis and machine learning. Patient data never leaves a healthcare institute. Materials and methodsLung cancer patient-specific databases (tumor staging and post-treatment survival information) of oncology departments were translated according to a FAIR data model and stored locally in a graph database. Software was installed locally to enable deployment of distributed machine learning algorithms via a central server. Algorithms (MATLAB, code and documentation publicly available) are patient privacy-preserving as only summary statistics and regression coefficients are exchanged with the central server. A logistic regression model to predict post-treatment two-year survival was trained and evaluated by receiver operating characteristic curves (ROC), root mean square prediction error (RMSE) and calibration plots. ResultsIn 4 months, we connected databases with 23 203 patient cases across 8 healthcare institutes in 5 countries (Amsterdam, Cardiff, Maastricht, Manchester, Nijmegen, Rome, Rotterdam, Shanghai) using the PHT. Summary statistics were computed across databases. A distributed logistic regression model predicting post-treatment two-year survival was trained on 14 810 patients treated between 1978 and 2011 and validated on 8 393 patients treated between 2012 and 2015. ConclusionThe PHT infrastructure demonstrably overcomes patient privacy barriers to healthcare data sharing and enables fast data analyses across multiple institutes from different countries with different regulatory regimens. This infrastructure promotes global evidence-based medicine while prioritizing patient privacy.

Distributed Approximate Newton's Method Robust to Byzantine Attackers

There is a recent surge of interest in the design of the first-order and the second-order distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified information to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers' choice. Some recent works have proposed algorithms that can defend against Byzantine attackers for the first-order methods. In this paper, we design two algorithms that can deal with Byzantine attackers for the second-order methods. The main idea of the first algorithm, named median-based approximate Newton's method (MNM), is to ask the parameter server to aggregate gradient information and approximate Newton's direction from all workers by geometric median. We show that MNM can converge when up to half of the workers are Byzantine attackers. To deal with the case with an arbitrary number of attackers, we then propose a comparison-based approximate Newton's method (CNM). The main idea of CNM is to ask the server to randomly select a small clean dataset and compute noisy gradient and Newton's direction using this small dataset. These noisy information will then be used as an approximation of the ground truth to filter out bad information from Byzantine attackers. We show that CNM can converge to the neighborhood of the population minimizer even when more than half of the workers are Byzantine workers. We further provide numerical examples to illustrate the performance of the proposed algorithms.