Distributed Learning Techniques Research Articles

Federated learning for biometric recognition: a survey

Recently, deep learning (DL) has achieved great success in biometric recognition. The application of DL has also led to a high demand for biometric data. However, as people attach more importance to privacy protection, biometric data have become increasingly difficult to obtain and access, leading to significant limitations in the development and application of DL-based biometric recognition. Federated learning (FL), a distributed learning technique with privacy protection, provides a potential solution to this problem. Several researchers have attempted to integrate FL into biometric recognition. These studies have shown that the introduction of FL not only solves the conflict between privacy and accessibility of biometric data but also improves the accuracy and generalizability of local recognition systems. Therefore, the combination of FL and biometric recognition techniques has become a new research hotspot. In this survey, we comprehensively review the latest advances regarding the application of FL to biometric recognition, biometric presentation attack detection and the related fields to provide new researchers with a quick and systematic overview of this emerging cross-disciplinary field. This paper also summarizes the future opportunities and challenges of this field. To our knowledge, this is the first survey that systematically organizes and analyses federated biometric recognition and related fields to provide suggestions and references for future research.

Proposal and investigation of a distributed learning strategy in Orbital Edge Computing-endowed satellite networks for Earth Observation applications

One of the key enabling solutions to in-orbit extract information from Earth Observation images is given by deep learning techniques. However, the accuracy of these algorithms is strictly related to the availability of large datasets of satellite images for training purposes. Limitations on the available transmission bandwidth in the orbital context may prevent the possibility to downlink all acquired images to a node where centralized training happens. Instead, Federated Learning (FL) could be fruitfully leveraged in this scenario, since it provides for each satellite to train a local model only with its own dataset, and then to share its trained model with a central server, which receives models trained by the different satellites and aggregates them into a new global model being eventually shared with all the satellites, and this repeats until convergence is reached. However, because communication with a node acting as a central parameter server may be still limited by short visibility time, the described process may need a long time because of limited communication windows, negatively impacting the time needed to reach model convergence. For this reason, we propose a communication strategy to support a completely distributed learning technique to train a deep learning model in-orbit, by leveraging the fact that satellites may form a network thanks to the potential availability of Inter-Satellite Links (ISLs) within and between orbital planes. Our proposal is different from a FL approach since we provide for each satellite to receive all the information needed to calculate an updated global model by itself, without leaning on a central parameter server. Numerical results show that distributed learning outperforms FL in number of learning rounds completed in the unit time, allowing for reaching validation accuracy convergence in a shorter time, as it has been verified on a land coverage classification task based on the EuroSAT dataset.

SFL-TUM: Energy efficient SFRL method for large scale AI model's task offloading in UAV-assisted MEC networks

The convergence of mobile edge computing (MEC) network with unmanned aerial vehicles (UAVs) presents an auspicious opportunity to revolutionize wireless communication and facilitate high-speed internet access in remote regions for mobile devices (MDs) as well as large scale artificial intelligence (AI) models. However, the substantial amount of data produced by the UAVs-assisted MEC network necessitates the integration of efficient distributed learning techniques in AI models. In recent times, distributed learning algorithms, including federated reinforcement learning (FRL) and split learning (SL), have been explored for the purpose of learning machine learning (ML) models that are distributed by sharing model parameters, as opposed to large raw data-sets as seen in traditional centralized learning algorithms. To implement the hybrid method, the model is first trained locally on each UAV-assisted MEC network using SL. Subsequently, the model parameters that have been encrypted are sent to a central server for federated averaging. Finally, after the model has been updated, it is distributed to each UAV-assisted MEC network for local fine-tuning. Our simulations indicate that the proposed split and federated reinforcement learning (SFRL) framework yields comparable high-test accuracy performance while consuming less energy compared to extant distributed learning algorithms. Furthermore, the SFRL algorithm efficiently realizes energy-efficient selection between the SL and FRL methods under different distributions. Numerical results shows that the proposed scheme improves the accuracy by 29.31% and reduced the energy consumption by around 67.34% and time delay by about 7.37%. as compared to the existing baseline schemes.

Federated Learning with Pareto Optimality for Resource Efficiency and Fast Model Convergence in Mobile Environments.

Federated learning (FL) is an emerging distributed learning technique through which models can be trained using the data collected by user devices in resource-constrained situations while protecting user privacy. However, FL has three main limitations: First, the parameter server (PS), which aggregates the local models that are trained using local user data, is typically far from users. The large distance may burden the path links between the PS and local nodes, thereby increasing the consumption of the network and computing resources. Second, user device resources are limited, but this aspect is not considered in the training of the local model and transmission of the model parameters. Third, the PS-side links tend to become highly loaded as the number of participating clients increases. The links become congested owing to the large size of model parameters. In this study, we propose a resource-efficient FL scheme. We follow the Pareto optimality concept with the biased client selection to limit client participation, thereby ensuring efficient resource consumption and rapid model convergence. In addition, we propose a hierarchical structure with location-based clustering for device-to-device communication using k-means clustering. Simulation results show that with prate at 0.75, the proposed scheme effectively reduced transmitted and received network traffic by 75.89% and 78.77%, respectively, compared to the FedAvg method. It also achieves faster model convergence compared to other FL mechanisms, such as FedAvg and D2D-FedAvg.

Federated learning on low-power Arduino Nano33 BLE Sense to predict the length of stay using a linear regression model

Federated learning (FL) is a collaborative learning paradigm where multiple clients are used to build the model without sharing data and preserving privacy. An FL-based linear regression model is designed to predict the length of stay for patients at hospitals using the low-power Arduino Nano 33 BLE Sense microcontroller unit (MCU). FL uses a distributed learning technique that allows model building from decentralized data sources. The Arduino Nano 33 BLE Sense is a compact and energy-efficient MCU providing an ideal platform for implementing FL in resource-constrained environments. FL algorithms aggregate model parameters from multiple Arduino clients and collectively train and build a predictive model to estimate the length of stay at the hospital by patients. Experiments were conducted to understand the performance of FL on clients with data of equal and varying sizes and heterogeneous data from multiple sources. The performance of the algorithm is evaluated based on Mean Absolute Error (MAE), Percentage Decrease in Training error (PDTE), and Percentage Difference with Optimal Testing (PDOT) value. Experimental results show that the number of local epochs and FL rounds affects the convergence of clients to the optimal value. The experimental results demonstrate the applicability of FL on low-power MCUs, preserving privacy which is a core requirement for healthcare solutions.

Implementation and Evaluation of a Federated Learning Framework on Raspberry PI Platforms for IoT 6G Applications

With the advent of 6G technology, the proliferation of interconnected devices necessitates a robust, fully connected intelligence network. Federated Learning (FL) stands as a key distributed learning technique, showing promise in recent advancements. However, the integration of novel Internet of Things (IoT) applications and virtualization technologies has introduced diverse and heterogeneous devices into wireless networks. This diversity encompasses variations in computation, communication, storage resources, training data, and communication modes among connected nodes. In this context, our study presents a pivotal contribution by analyzing and implementing FL processes tailored for 6G standards. Our work defines a practical FL platform, employing Raspberry Pi devices and virtual machines as client nodes, with a Windows PC serving as a parameter server. We tackle the image classification challenge, implementing the FL model via PyTorch, augmented by the specialized FL library, Flower. Notably, our analysis delves into the impact of computational resources, data availability, and heating issues across heterogeneous device sets. Additionally, we address knowledge transfer and employ pre-trained networks in our FL performance evaluation. This research underscores the indispensable role of artificial intelligence in IoT scenarios within the 6G landscape, providing a comprehensive framework for FL implementation across diverse and heterogeneous devices.

Distributed One-Pass Online AUC Maximization

AUC (Area Under ROC curve) is a widely used metric for measuring classification performance. Recently, due to the efficiency and scalability of online learning in handling large-scale problems, it attracts lots of interests to maximize AUC in online setting. Traditional online learning algorithms require a computation node to process all the data sequentially. However, due to the explosion in volume and velocity of modern data generated from distributed sources such as sensor network and Internet, it is not efficient to send all the data to one node for processing and updating model sequentially. To address this challenge, we explore the distributed learning techniques for maximizing AUC in online setting with a linear model. Specifically, we propose two different algorithms for solving the distributed online AUC maximization problem: (i) the Centralized Distributed One-Pass Online AUC Maximization (C-DOPOAM) algorithm in which the server updates the global model with the gradients computed from workers; and (ii) the Decentralized Distributed One-Pass Online AUC Maximization (D-DOPOAM) algorithm in which each worker updates its local model and exchanges information with its neighbors to achieve consensus. Moreover, we establish the regret bounds of the two proposed algorithms. They achieve the similar convergence rate with the state-of-the-art non-distributed online AUC maximization algorithms. Finally, we conduct extensive experiments to validate our theoretical result on different computation platforms and network configurations. We also make a comparison between C-DOPOAM and D-DOPOAM in poor network condition. The results show D-DOPOAM is more robust than C-DOPOAM against low bandwidth or high latency.

Performance-Oriented Design for Intelligent Reflecting Surface-Assisted Federated Learning

-1To efficiently exploit the massive amounts of raw data that are increasingly being generated in mobile edge networks, federated learning (FL) has emerged as a promising distributed learning technique by collaboratively training a shared learning model on edge devices. The number of resource blocks when using traditional orthogonal transmission strategies for FL linearly scales with the number of participating devices, which conflicts with the scarcity of communication resources. To tackle this issue, over-the-air computation (AirComp) has emerged recently which leverages the inherent superposition property of wireless channels to perform <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">one-shot</i> model aggregation. However, the aggregation accuracy in AirComp suffers from the unfavorable wireless propagation environment. In this paper, we consider the use of intelligent reflecting surfaces (IRSs) to mitigate this problem and improve FL performance with AirComp. Specifically, a novel performance-oriented long-term design scheme that integrated design multiple communication rounds to minimize the optimality gap of the loss function is proposed. We first analyze the convergence behavior of the FL procedure with the absence of channel fading and noise. Based on the obtained optimality gap which characterizes the impact of channel fading and noise in different communication rounds on the ultimate performance of FL, we propose both online and offline schemes to tackle the resulting design problem. Simulation results demonstrate that such a long-term design strategy can achieve higher test accuracy than the conventional isolated design approach in FL. Both the theoretical analysis and numerical results exhibit a “later-is-better” principle, which demonstrates the later rounds in the FL procedure are more sensitive to aggregation error, and hence more resources are required over time.

Data-Driven Participant Selection and Bandwidth Allocation for Heterogeneous Federated Edge Learning

Federated edge learning (FEEL) is a rapidly growing distributed learning technique for next-generation wireless edge systems. Smart systems across various application domains face challenges, such as data heterogeneity, limited wireless resources, and device heterogeneity, which necessitate intelligent participant selection schemes that accelerate convergence rates. Consequently, this article presents joint participant selection and bandwidth allocation schemes to address these challenges. First, we formulate an optimization problem that considers communication and computation latencies, as well as imbalanced data distribution, while meeting round deadlines and bandwidth constraints. To address the combinatorial problems of participant selection, we employ a relaxation method followed by a proposed priority selection algorithm to select near-optimal participants. The proposed algorithm initially prioritizes participants with larger datasets, effective channel states, and better CPU speeds. To address data heterogeneity, we propose a randomized deadline-controlling algorithm that diversifies updates by allowing the edge server to include different participants with fewer data samples in training rounds. The proposed algorithms offer near-optimal performance compared to the brute-force method. Experiments demonstrate that our proposed scheme accelerates the convergence rate by up to 55% under extensive non-IID settings compared to benchmarks. Furthermore, the deadline-controlling algorithm improves performance at high levels of data heterogeneity, resulting in faster FEEL systems.

EHR privacy preservation using federated learning with DQRE-Scnet for healthcare application domains

A distributed learning technique named Federated Learning (FL) is utilized by mobile devices, clinical research labs, and hospitals for secure healthcare data sharing. FL has been utilized by several researchers for preserving the privacy of Electronic Health Record (EHR) information while updating the global server. Still, the privacy and higher number of communication rounds limit the performance of the model. Hence, this research introduces the privacy-preserving EHR information sharing technique, FL, based on deep Q reinforcement learning with spectral clustering (FL+DQRE-SCnet). In this, the local healthcare information is trained using the DCNN in the respective hospitals based on FL criteria. After learning the local model, the information is shared with the global model for updating the EHR to enhance the accuracy of disease diagnosis. The global model aggregates the information from the local models over several communication rounds by randomly selecting the local models, increasing the need for more communication rounds. Therefore, DQRE-SCnet is introduced before data aggregation to minimize the communication rounds. In addition, including a homomorphic encryption technique ensures privacy protection for FL data sharing. The analysis of FL+DQRE-SCnet based on the assessment measures of accuracy, precision, recall, and F-measure acquired maximal values of 95%, 94.9%, 94.94%, and 93.94%, respectively. Also, the error rate of 0.46 and the execution time of 1400 sec are accomplished by the FL+DQRE-SCnet method. Here, for comparing the performance of the model, the baseline methods like FedAvg, LoAdaBoost, BOFRF, PP-FedAvg and AdaFed are considered, wherein the proposed method acquired superior performance.

Distributed Intelligence in Wireless Networks

The cloud-based solutions are becoming inefficient due to considerably large time delays, high power consumption, and security and privacy concerns caused by billions of connected wireless devices and typically zillions of bytes of data they produce at the network edge. A blend of edge computing and Artificial Intelligence (AI) techniques could optimally shift the resourceful computation servers closer to the network edge, which provides the support for advanced AI applications (e.g., video/audio surveillance and personal recommendation system) by enabling intelligent decision making on computing at the point of data generation as and when it is needed, and distributed Machine Learning (ML) with its potential to avoid the transmission of the large dataset and possible compromise of privacy that may exist in cloud-based centralized learning. Besides, the deployment of AI techniques to redesign end-to-end communication is attracting attention to improve communication performance. Therefore, the interaction of AI and wireless communications generates a new concept, named native AI wireless networks. In this paper, we conduct a comprehensive overview of recent advances in distributed intelligence in wireless networks under the umbrella of native AI wireless networks, with a focus on the design of distributed learning architectures for heterogeneous networks, on AI-enabled edge computing, on the communication-efficient technologies to support distributed learning, and on the AI-empowered end-to-end communications. We highlight the advantages of hybrid distributed learning architectures compared to state-of-the-art distributed learning techniques. We summarize the challenges of existing research contributions in distributed intelligence in wireless networks and identify potential future opportunities.

Federated learning for energy constrained devices: a systematic mapping study

Federated Machine Learning (Fed ML) is a new distributed machine learning technique applied to collaboratively train a global model using clients local data without transmitting it. Nodes only send parameter updates (e.g., weight updates in the case of neural networks), which are fused together by the server to build the global model. By not divulging node data, Fed ML guarantees its confidentiality, a crucial aspect of network security, which enables it to be used in the context of data-sensitive Internet of Things (IoT) and mobile applications, such as smart Geo-location and the smart grid. However, most IoT devices are particularly energy constrained, which raises the need to optimize the Fed ML process for efficient training tasks and optimized power consumption. In this paper, we conduct, to the best of our knowledge, the first Systematic Mapping Study (SMS) on Fed ML optimization techniques for energy-constrained IoT devices. From a total of more than 800 papers, we select 67 that satisfy our criteria and give a structured overview of the field using a set of carefully chosen research questions. Finally, we attempt to provide an analysis of the energy-constrained Fed ML state of the art and try to outline some potential recommendations for the research community.

Fog-supported Low-latency Monitoring of System Disruptions in Industry 4.0: A Federated Learning Approach

Industry 4.0 is based on machine learning and advanced digital technologies, such as Industrial-Internet-of-Things and Cyber-Physical-Production-Systems, to collect and process data coming from manufacturing systems. Thus, several industrial issues may be further investigated including, flows disruptions, machines’ breakdowns, quality crisis, and so on. In this context, traditional machine learning techniques require the data to be stored and processed in a central entity, e.g., a cloud server. However, these techniques are not suitable for all manufacturing use cases, due to the inaccessibility of private data such as resources’ localization in real time, which cannot be shared at the cloud level as they contain personal and sensitive information. Therefore, there is a critical need to go toward decentralized learning solutions to handle efficiently distributed private sub-datasets of manufacturing systems. In this article, we design a new monitoring tool for system disruption related to the localization of mobile resources. Our tool may identify mobile resources (human operators) that are in unexpected locations, and hence has a high probability to disturb production planning. To do so, we use federated deep learning, as distributed learning technique, to build a prediction model of resources locations in manufacturing systems. Our prediction model is generated based on resources locations defined in the initial tasks schedule. Thus, system disruptions are detected, in real time, when comparing predicted locations to the real ones, that is collected through the IoT network. In addition, our monitoring tool is deployed at Fog computing level that provides local data processing support with low latency. Furthermore, once a system disruption is detected, we develop a dynamic rescheduling module that assigns each task to the nearest available resource while improving the execution accuracy and reducing the execution delay. Therefore, we formulate an optimization problem of tasks rescheduling, before solving it using the meta-heuristic Tabu search. The numerical results show the efficiency of our schemes in terms of prediction accuracy when compared to other machine learning algorithms, in addition to their ability to detect and resolve system disruption in real time.

Distributed Training of Deep Neural Network Acoustic Models for Automatic Speech Recognition: A comparison of current training strategies

The past decade has witnessed great progress in automatic speech recognition (ASR) due to advances in deep learning. The improvements in performance can be attributed to both improved models and large-scale training data. The key to training such models is the employment of efficient distributed learning techniques. In this article, we provide an overview of distributed training techniques for deep neural network (DNN) acoustic models used for ASR. Starting with the fundamentals of data parallel stochastic gradient descent (SGD) and ASR acoustic modeling, we investigate various distributed training strategies and their realizations in high-performance computing (HPC) environments with an emphasis on striking a balance between communication and computation. Experiments are carried out on a popular public benchmark to study the convergence, speedup, and recognition performance of the investigated strategies.

Computational Intelligence for Solving Difficult Transportation Problems

Intelligent Transportation Systems (ITSs) are providing a broad range of services to all the actors involved in any type of transportation activities. Path planning is a common transportation problem, which can be extremely difficult when an exact solution is sought. The quality of heuristic approaches to this problem is therefore important, as they are now implemented as planning modules into almost all the ITSs on the market. The goal of this paper is to analyze several distributed reinforcement learning techniques involving the collective experience of artificial ants when approaching a transportation problem. Traveling Salesman Problem (TSP), a well-known NP-hard transportation problem, was chosen for computational experiments. In order to assess the behavior of the investigated solving methods, two medium-size TSP instances, covering large areas, are proposed. The results are compared with those provided by a state-of-the-art technique, namely the highly effective Lin-Kernighan(LK) heuristic procedure for generating optimum and near-optimum solutions for the TSP. This paper connects the academic Operational Research with societal needs in Transportation and Logistics.

Big Data in radiation therapy: challenges and opportunities.

Data collected and generated by radiation oncology can be classified by the Volume, Variety, Velocity and Veracity (4Vs) of Big Data because they are spread across different care providers and not easily shared owing to patient privacy protection. The magnitude of the 4Vs is substantial in oncology, especially owing to imaging modalities and unclear data definitions. To create useful models ideally all data of all care providers are understood and learned from; however, this presents challenges in the guise of poor data quality, patient privacy concerns, geographical spread, interoperability and large volume. In radiation oncology, there are many efforts to collect data for research and innovation purposes. Clinical trials are the gold standard when proving any hypothesis that directly affects the patient. Collecting data in registries with strict predefined rules is also a common approach to find answers. A third approach is to develop data stores that can be used by modern machine learning techniques to provide new insights or answer hypotheses. We believe all three approaches have their strengths and weaknesses, but they should all strive to create Findable, Accessible, Interoperable, Reusable (FAIR) data. To learn from these data, we need distributed learning techniques, sending machine learning algorithms to FAIR data stores around the world, learning from trial data, registries and routine clinical data rather than trying to centralize all data. To improve and personalize medicine, rapid learning platforms must be able to process FAIR “Big Data” to evaluate current clinical practice and to guide further innovation.

A vlHMM approach to context-aware search

Capturing the context of a user's query from the previous queries and clicks in the same session leads to a better understanding of the user's information need. A context-aware approach to document reranking, URL recommendation, and query suggestion may substantially improve users' search experience. In this article, we propose a general approach to context-aware search by learning a variable length hidden Markov model ( vlHMM ) from search sessions extracted from log data. While the mathematical model is powerful, the huge amounts of log data present great challenges. We develop several distributed learning techniques to learn a very large vlHMM under the map-reduce framework. Moreover, we construct feature vectors for each state of the vlHMM model to handle users' novel queries not covered by the training data. We test our approach on a raw dataset consisting of 1.9 billion queries, 2.9 billion clicks, and 1.2 billion search sessions before filtering, and evaluate the effectiveness of the vlHMM learned from the real data on three search applications: document reranking, query suggestion, and URL recommendation. The experiment results validate the effectiveness of vlHMM in the applications of document reranking, URL recommendation, and query suggestion.

The Development of Distributed Learning Techniques in Bhutan and Nepal

This paper discusses research and development work currently being conducted with universities in Bhutan and Nepal to design appropriate systems for distance and distributed learning courses among a network of campus sites. Although working from a high level of awareness of pedagogic skills, staff in the region face two significant impediments in the adoption of a more open culture of learning. Firstly, Internet access is improving rapidly, but is still generally too weak and inconsistent to allow any reliance on net-based learning solutions. Secondly, the academic culture is resistant to the recognition of the value of open-learning degrees, with subsequent difficulties in re-designing course materials for a more educationally flexible, student-centred learning environment. Some current pilot initiatives in distributed learning are described. Methods for addressing these two impediments are discussed.

Autonomous Agents that Learn to Better Coordinate

A fundamental difficulty faced by groups of agents that work together is how to efficiently coordinate their efforts. This coordination problem is both ubiquitous and challenging, especially in environments where autonomous agents are motivated by personal goals.Previous AI research on coordination has developed techniques that allow agents to act efficiently from the outset based on common built-in knowledge or to learn to act efficiently when the agents are not autonomous. The research described in this paper builds on those efforts by developing distributed learning techniques that improve coordination among autonomous agents.The techniques presented in this work encompass agents who are heterogeneous, who do not have complete built-in common knowledge, and who cannot coordinate solely by observation. An agent learns from her experiences so that her future behavior more accurately reflects what works (or does not work) in practice. Each agent stores past successes (both planned and unplanned) in their individual casebase. Entries in a casebase are represented as coordinated procedures and are organized around learned expectations about other agents.It is a novel approach for individuals to learn procedures as a means for the group to coordinate more efficiently. Empirical results validate the utility of this approach. Whether or not the agents have initial expertise in solving coordination problems, the distributed learning of the individual agents significantly improves the overall performance of the community, including reducing planning and communication costs.