Distributed Learning Approach Research Articles

Federated Incremental Learning algorithm based on Topological Data Analysis

Federated learning is a distributed learning approach aimed at preserving user’s data privacy, while incremental learning is an adaptive machine learning method that enables continuous learning of new data. The combination of these two approaches into Federated Incremental Learning (FIL) algorithms brings together their respective advantages. However, the existing federated incremental learning still faces two main challenges: (1) catastrophic forgetting of previous knowledge by local models when adapting to incremental tasks, and (2) limited capability of the server to capture critical features during federated aggregation. To address these challenges, this paper proposes a federated incremental learning algorithm based on Topological Data Analysis (TDA). Firstly, the algorithm extracts topological features from the input information and designs a Topological Stability Loss (TSL) to mitigate catastrophic forgetting of previous knowledge by local models. Secondly, a feature attention mechanism is utilized to select appropriate attention weights for each local model, enhancing the recognition performance of the global model. The experimental results on publicly available datasets, namely CIFAR10, CIFAR100, and ImageNet, demonstrate that the proposed approach in this paper achieves global accuracies of 67.23%, 65.75%, and 62.41% respectively for the federated incremental learning model. These accuracies surpass the existing Icarl incremental algorithm by improvements of 3.21%, 2.87%, and 1.34% respectively. Therefore, the algorithm presented in this paper achieves better performance, indicating its superiority over existing methods.

Mobility-aware federated self-supervised learning in vehicular network

The development of the Internet of Things has led to a significant increase in the number of devices, consequently generating a vast amount of data and resulting in an influx of unlabeled data. Collecting these data enables the training of robust models to support a broader range of applications. However, labeling these data can be costly, and the models dependent on labeled data are often unsuitable for rapidly evolving fields like vehicular networks and mobile Internet of Things, where new data continuously emerge. To address this challenge, Self-Supervised Learning (SSL) offers a way to train models without the need for labels. Nevertheless, the data stored locally in vehicles are considered private, and vehicles are reluctant to share data with others. Federated Learning (FL) is an advanced distributed machine learning approach that protects each vehicle’s privacy by allowing models to be trained locally and the model parameters to be exchanged across multiple devices simultaneously. Additionally, vehicles capture images while driving through cameras mounted on their rooftops. If a vehicle’s velocity is too high, the captured images, donated as local data, may be blurred. Simple aggregation of such data can negatively impact the accuracy of the aggregated model and slow down the convergence speed of FL. This paper proposes a FL algorithm for aggregation based on image blur levels, which is called FLSimCo. This algorithm does not require labels and serves as a pre-training stage for SSL in vehicular networks. Simulation results demonstrate that the proposed algorithm achieves fast and stable convergence.

HarmonyTM: multi-center data harmonization applied to distributed learning for Parkinson's disease classification.

Distributed learning is widely used to comply with data-sharing regulations and access diverse datasets for training machine learning (ML) models. The traveling model (TM) is a distributed learning approach that sequentially trains with data from one center at a time, which is especially advantageous when dealing with limited local datasets. However, a critical concern emerges when centers utilize different scanners for data acquisition, which could potentially lead models to exploit these differences as shortcuts. Although data harmonization can mitigate this issue, current methods typically rely on large or paired datasets, which can be impractical to obtain in distributed setups. We introduced HarmonyTM, a data harmonization method tailored for the TM. HarmonyTM effectively mitigates bias in the model's feature representation while retaining crucial disease-related information, all without requiring extensive datasets. Specifically, we employed adversarial training to "unlearn" bias from the features used in the model for classifying Parkinson's disease (PD). We evaluated HarmonyTM using multi-center three-dimensional (3D) neuroimaging datasets from 83 centers using 23 different scanners. Our results show that HarmonyTM improved PD classification accuracy from 72% to 76% and reduced (unwanted) scanner classification accuracy from 53% to 30% in the TM setup. HarmonyTM is a method tailored for harmonizing 3D neuroimaging data within the TM approach, aiming to minimize shortcut learning in distributed setups. This prevents the disease classifier from leveraging scanner-specific details to classify patients with or without PD-a key aspect for deploying ML models for clinical applications.

Robustness evaluation of large-scale machine learning-based reduced order models for reproducing flow fields

The robustness of an artificial neural network that performs model order reduction for flow field data is studied. The network is trained with a large-scale distributed learning approach using up to 6259 nodes of the supercomputer Fugaku. Flow around two square cylinders with a varying distance between their centers is investigated. The network is trained and tested with data from numerical simulations. First, the capability to reproduce flow fields with 2, 12, and 24 modes is investigated by comparing the reconstructed flow data to simulated data. It is shown, that reconstructions based on 2 modes cannot capture both, low- and high-frequency flow structures correctly, whereas predictions based on 12 and 24 modes yield improved flow fields, especially in the case of high-frequency waves in the vicinity of the square cylinders. Reconstructions with 24 modes provide smooth velocity fields that reproduce all relevant low- and high-frequency waves for all variations of the distance between the two square cylinders. Second, the performance of the machine learning-based reconstructions are compared to proper orthogonal decomposition, which is a commonly used reduced order model technique. The comparison only includes flow fields based on 24 modes. For all geometric variations, the mean squared errors of the reconstructions by the conventional method are higher than those of the machine learning model. This underlines the advantage of artificial neural networks over linear methods like proper orthogonal decomposition for tasks like reconstructing flow fields that are characterized by non-linear governing equations.

A multi-center distributed learning approach for Parkinson's disease classification using the traveling model paradigm.

Distributed learning is a promising alternative to central learning for machine learning (ML) model training, overcoming data-sharing problems in healthcare. Previous studies exploring federated learning (FL) or the traveling model (TM) setup for medical image-based disease classification often relied on large databases with a limited number of centers or simulated artificial centers, raising doubts about real-world applicability. This study develops and evaluates a convolution neural network (CNN) for Parkinson's disease classification using data acquired by 83 diverse real centers around the world, mostly contributing small training samples. Our approach specifically makes use of the TM setup, which has proven effective in scenarios with limited data availability but has never been used for image-based disease classification. Our findings reveal that TM is effective for training CNN models, even in complex real-world scenarios with variable data distributions. After sufficient training cycles, the TM-trained CNN matches or slightly surpasses the performance of the centrally trained counterpart (AUROC of 83% vs. 80%). Our study highlights, for the first time, the effectiveness of TM in 3D medical image classification, especially in scenarios with limited training samples and heterogeneous distributed data. These insights are relevant for situations where ML models are supposed to be trained using data from small or remote medical centers, and rare diseases with sparse cases. The simplicity of this approach enables a broad application to many deep learning tasks, enhancing its clinical utility across various contexts and medical facilities.

Distributed Learning in the IoT–Edge–Cloud Continuum

The goal of the IoT–Edge–Cloud Continuum approach is to distribute computation and data loads across multiple types of devices taking advantage of the different strengths of each, such as proximity to the data source, data access, or computing power, while mitigating potential weaknesses. Most current machine learning operations are currently concentrated on remote high-performance computing devices, such as the cloud, which leads to challenges related to latency, privacy, and other inefficiencies. Distributed learning approaches can address these issues by enabling the distribution of machine learning operations throughout the IoT–Edge–Cloud Continuum by incorporating Edge and even IoT layers into machine learning operations more directly. Approaches like transfer learning could help to transfer the knowledge from more performant IoT–Edge–Cloud Continuum layers to more resource-constrained devices, e.g., IoT. The implementation of these methods in machine learning operations, including the related data handling security and privacy approaches, is challenging and actively being researched. In this article the distributed learning and transfer learning domains are researched, focusing on security, robustness, and privacy aspects, and their potential usage in the IoT–Edge–Cloud Continuum, including research on tools to use for implementing these methods. To achieve this, we have reviewed 145 sources and described the relevant methods as well as their relevant attack vectors and provided suggestions on mitigation.

Self-Organizing Democratized Learning: Toward Large-Scale Distributed Learning Systems.

Emerging cross-device artificial intelligence (AI) applications require a transition from conventional centralized learning systems toward large-scale distributed AI systems that can collaboratively perform complex learning tasks. In this regard, democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. The outlined principles are meant to study a generalization in distributed learning systems that go beyond existing mechanisms such as federated learning (FL). Moreover, such learning systems rely on hierarchical self-organization of well-connected distributed learning agents who have limited and highly personalized data and can evolve and regulate themselves based on the underlying duality of specialized and generalized processes. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this article. The approach consists of a self-organizing hierarchical structuring mechanism based on agglomerative clustering, hierarchical generalization, and corresponding learning mechanism. Subsequently, hierarchical generalized learning problems in recursive forms are formulated and shown to be approximately solved using the solutions of distributed personalized learning problems and hierarchical update mechanisms. To that end, a distributed learning algorithm, namely DemLearn, is proposed. Extensive experiments on benchmark MNIST, Fashion-MNIST, FE-MNIST, and CIFAR-10 datasets show that the proposed algorithm demonstrates better results in the generalization performance of learning models in agents compared to the conventional FL algorithms. The detailed analysis provides useful observations to further handle both the generalization and specialization performance of the learning models in Dem-AI systems.

Distributed Computing and Inference for Big Data

Data are distributed across different sites due to computing facility limitations or data privacy considerations. Conventional centralized methods—those in which all datasets are stored and processed in a central computing facility—are not applicable in practice. Therefore, it has become necessary to develop distributed learning approaches that have good inference or predictive accuracy while remaining free of individual data or obeying policies and regulations to protect privacy. In this article, we introduce the basic idea of distributed learning and conduct a selected review on various distributed learning methods, which are categorized by their statistical accuracy, computational efficiency, heterogeneity, and privacy. This categorization can help evaluate newly proposed methods from different aspects. Moreover, we provide up-to-date descriptions of the existing theoretical results that cover statistical equivalency and computational efficiency under different statistical learning frameworks. Finally, we provide existing software implementations and benchmark datasets, and we discuss future research opportunities.

An Incentive Auction for Heterogeneous Client Selection in Federated Learning

Federated Learning (FL) is a new distributed machine learning (ML) approach which enables thousands of mobile devices to collaboratively train artificial intelligence (AI) models using local data without compromising user privacy. Although FL represents a promising computing paradigm, such training process can not be fully realized without an appropriate economic mechanism that incentivizes the participation of heterogeneous clients. This work targets social cost minimization, and studies the incentive mechanism design in FL through a procurement auction. Different from existing literature, we consider a practical scenario of FL where clients are selected and scheduled at different global iterations to guarantee the completion of the FL job, and capture the distinct feature of FL that the number of global iterations is determined by the local accuracy of all participants to balance between computation and communication. Our auction framework <inline-formula><tex-math notation="LaTeX">$A_{FL}$</tex-math></inline-formula> first decomposes the social cost minimization problem into a series of winner determination problems (WDPs) based on the number of global iterations. To solve each WDP, <inline-formula><tex-math notation="LaTeX">$A_{FL}$</tex-math></inline-formula> invokes a greedy algorithm to determine the winners, and a payment algorithm for computing remuneration to winners. Finally, <inline-formula><tex-math notation="LaTeX">$A_{FL}$</tex-math></inline-formula> returns the best solution among all WDPs. We carried out theoretical analysis to prove that <inline-formula><tex-math notation="LaTeX">$A_{FL}$</tex-math></inline-formula> is truthful, individual rational, computationally efficient, and achieves a near-optimal social cost. We further extend our model to consider multiple FL jobs with corresponding budgets and propose another efficient algorithm <inline-formula><tex-math notation="LaTeX">$A_{FL-M}$</tex-math></inline-formula> to solve the extended problem. We conduct large-scale simulations based on the real-world data and testbed experiments by adopting FL frameworks FAVOR and CoCoA. Simulation and experiment results show that both <inline-formula><tex-math notation="LaTeX">$A_{FL}$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$A_{FL-M}$</tex-math></inline-formula> can reduce the social cost by up to 55&#x0025; compared with state-of-the-art algorithms.

Analysis and Performance Evaluation of Transfer Learning Algorithms for 6G Wireless Networks

The development of the 5G network and the transition to 6G has given rise to multiple challenges for ensuring high-quality and reliable network services. One of these main challenges is the emergent intelligent defined networks (IDN), designed to provide highly efficient connectivity, by merging artificial intelligence (AI) and networking concepts, to ensure distributed intelligence over the entire network. To this end, it will be necessary to develop and implement proper machine learning (ML) algorithms that take into account this new distributed nature of the network to represent increasingly dynamic, adaptable, scalable, and efficient systems. To be able to cope with more stringent service requirements, it is necessary to renew the ML approaches to make them more efficient and faster. Distributed learning (DL) approaches are shown to be effective in enabling the possibility of deploying intelligent nodes in a distributed network. Among several DL approaches, transfer learning (TL) is a valid technique to achieve the new objectives required by emerging networks. Through TL, it is possible to reuse ML models to solve new problems without having to recreate a learning model from scratch. TL, combined with distributed network scenarios, turns out to be one of the key technologies for the advent of this new era of distributed intelligence. The goal of this paper is to analyze TL performance in different networking scenarios through a proper MATLAB implementation.

Enabling Long-Term Cooperation in Cross-Silo Federated Learning: A Repeated Game Perspective

Cross-silo federated learning (FL) is a distributed learning approach where clients of the same interest train a global model cooperatively while keeping their local data private. While there has been some work on incentivizing clients to join FL, the analysis of clients long-term selfish participation behaviors in cross-silo FL remains largely unexplored. In this paper, we model clients long-term selfish participation behaviors as an infinitely repeated game, with the stage game being a selfish participation game in one cross-silo FL process (SPFL). For the stage game SPFL, we derive the unique Nash equilibrium (NE), and propose a distributed algorithm for each client to calculate its equilibrium participation strategy. We show that at the NE, some clients may choose to be free riders. The existence of free riders has a detrimental effect on sustaining other clients long-term participation. For the long-term interactions among clients, we derive the optimal SPNE which minimizes the number of free riders while maximizing the amount of local data for model training. Simulation results show that our derived optimal SPNE can effectively reduce the number of free riders by up to 99.3% and increase the amount of local data for model training by up to 82.3%.

A DQN-Based Multi-Objective Participant Selection for Efficient Federated Learning

As a new distributed machine learning (ML) approach, federated learning (FL) shows great potential to preserve data privacy by enabling distributed data owners to collaboratively build a global model without sharing their raw data. However, the heterogeneity in terms of data distribution and hardware configurations make it hard to select participants from the thousands of nodes. In this paper, we propose a multi-objective node selection approach to improve time-to-accuracy performance while resisting malicious nodes. We firstly design a deep reinforcement learning-assisted FL framework. Then, the problem of multi-objective node selection under this framework is formulated as a Markov decision process (MDP), which aims to reduce the training time and improve model accuracy simultaneously. Finally, a Deep Q-Network (DQN)-based algorithm is proposed to efficiently solve the optimal set of participants for each iteration. Simulation results show that the proposed method not only significantly improves the accuracy and training speed of FL, but also has stronger robustness to resist malicious nodes.

End-to-End Deep Learning for Multipair Two-Way Massive MIMO with PA Impairments

Multipair (MP) massive multiple-input–multiple-output (MIMO) two-way relaying system is a promising solution that is able to provide excellent spectral efficiency performance. It consists in multiple pairs ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2K$</tex-math></inline-formula> ) of single-antenna user terminals that exchange information through of a relay using a large number of antennas <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$M_{t}\gg 2K$</tex-math></inline-formula> . The relay is equipped with a large number of energy-efficient power amplifiers (PAs) and ensure the multipair two-way forwarding. In this article, we first present an analytical study focused on the effects of PAs on the considered system over block fading Rayleigh channel. We derive closed-form expressions for the symbol error rate. Then, we develop a new efficient distributed learning approach for MP two-way massive MIMO. Specifically, we design the MP two-way massive MIMO system by leveraging the concept of the autoencoder, whereas the end-to-end communication system is designed using one deep neural network (NN). Interestingly, the proposed scheme, referred to as AE-MP-mMIMO, is suited for varying block fading Rayleigh channel scenarios. Indeed, we propose to adopt one stage precoder/decoder for each UE and two-stage precoding scheme for the relay: 1) an NN-Tx is used to address the PA nonlinearity and 2) a linear zero-forcing precoder is adopted to remove the multiuser interference. The NN-Tx and the UE's precoder/decoder are trained off-line and when the channel varies, only the zero-forcing precoder changes on-line. Numerical simulations show the capability of the proposed approach to achieve competitive performance with a significantly lower complexity compared to existing literature.

Distributed Learning Meets 6G: A Communication and Computing Perspective

With the ever improving computing capabilities and storage capacities of mobile devices in line with evolving telecommunication network paradigms, there has been an explosion of research interest toward exploring distributed learning (DL) frameworks to realize stringent key performance indicators (KPIs) that are expected in next-generation/6G cellular networks. In conjunction with edge computing, federated learning (FL) has emerged as the DL architecture of choice in prominent wireless applications. This article provides an outline of how DL in general and FL-based strategies specifically can contribute toward realizing part of the 6G vision and strike a balance between communication and computing constraints. As a practical use case, we apply multi-agent reinforcement learning within the FL framework to the dynamic spectrum access (DSA) problem and present preliminary evaluation results. Top contemporary challenges in applying DL approaches to 6G networks are also highlighted.

A federated learning scheme meets dynamic differential privacy

AbstractFederated learning is a widely used distributed learning approach in recent years, however, despite model training from collecting data become to gathering parameters, privacy violations may occur when publishing and sharing models. A dynamic approach is proposed to add Gaussian noise more effectively and apply differential privacy to federal deep learning. Concretely, it is abandoning the traditional way of equally distributing the privacy budget and adjusting the privacy budget to accommodate gradient descent federation learning dynamically, where the parameters depend on computation derived to avoid the impact on the algorithm that hyperparameters are created manually. It also incorporates adaptive threshold cropping to control the sensitivity, and finally, moments accountant is used to counting the consumed on the privacy‐preserving, and learning is stopped only if the by clients setting is reached, this allows the privacy budget to be adequately explored for model training. The experimental results on real datasets show that the method training has almost the same effect as the model learning of non‐privacy, which is significantly better than the differential privacy method used by TensorFlow.

STAR-RISs Assisted NOMA Networks: A Distributed Learning Approach

A novel simultaneously transmitting and reflecting reconfigurable intelligent surfaces (STAR-RISs) aided downlink non-orthogonal multiple access (NOMA) communication framework is proposed. Two STAR-RIS protocols are investigated, namely the energy splitting (ES) and the mode switching (MS). However, since the STAR-RIS has a massive number of reconfigurable elements, the passive beamforming problem has enormous action dimensions and extremely high complexity, resulting in an increased training time and performance degradation for the artificial intelligent agent. To resolve this predicament, a partitioning approach is proposed to divide the STAR-RIS into several tiles. A distributed learning approach is conceived for the partitioning and the corresponding tile-based passive beamforming of each STAR-RIS, as well as the power allocation for users to maximize the average throughput achieved by multiple base stations. In particular, the deep reinforcement learning (DRL) agent is employed at each BS, and an access-free federated learning (AFFL) model, which gathers the features of federated learning (FL) and transfer learning (TL) is proposed to accelerate or even exempt the training process for agents. Simulation results indicate that 1) the ES protocol is preferred for being employed in the NOMA networks compared with the MS protocol; 2) the tile-based passive beamforming approach outperforms benchmarks while the STAR-RIS has a large size; 3) the proposed AFFL framework effectively reduces or exempts the training overhead.

Task-Agnostic Vision Transformer for Distributed Learning of Image Processing.

Recently, distributed learning approaches have been studied for using data from multiple sources without sharing them, but they are not usually suitable in applications where each client carries out different tasks. Meanwhile, Transformer has been widely explored in computer vision area due to its capability to learn the common representation through global attention. By leveraging the advantages of Transformer, here we present a new distributed learning framework for multiple image processing tasks, allowing clients to learn distinct tasks with their local data. This arises from a disentangled representation of local and non-local features using a task-specific head/tail and a task-agnostic Vision Transformer. Each client learns a translation from its own task to a common representation using the task-specific networks, while the Transformer body on the server learns global attention between the features embedded in the representation. To enable decomposition between the task-specific and common representations, we propose an alternating training strategy between clients and server. Experimental results on distributed learning for various tasks show that our method synergistically improves the performance of each client with its own data.

Monitoring of NOx sensor drift in automotive fleets in a cloud/edge framework

Nitrogen oxides (NOx) are one of the main pollutants from both SI and CI engines. In recent years, regulation focus has moved from type-approval certification over known driving cycles to real-life verification of the emission level. In this paper, a system for NOx in-service emission monitoring based on ASSIST-IoT cloud/edge architecture is presented, and the effect of sensor bias on the emission level estimate discussed. The use of an additional high-fidelity sensor in a sample of the fleet is proposed as a method for estimating series sensor drift through a distributed learning approach, able to provide local and global models for the sensor.

Artificial Intelligence Meets Autonomy in Wireless Networks: A Distributed Learning Approach

Future wireless networks entail autonomous controls of mobile devices in a variety of applications, such as internet of things and mobile edge computing. A distributed nature of wireless networks poses a fundamental challenge in decentralized management of separate devices. In particular, the coordination among devices possessing heterogeneous levels of cooperation policy and computation power emerges as a critical implementation issue for practical wireless networks. This challenge turns out to handle dynamic link topologies with node interconnections prone to arbitrary configurations and random changes. This article presents a structural learning solution called Platform for Learning Autonomy in Distributed Optimization (PLAyDO), with an objective of modeling dynamic interactions among wireless devices through a mesh of deep neural networks (NNs). This framework identifies distributed mechanisms among autonomous devices, such as decentralized computations and communication protocols, to handle arbitrary interaction topologies. The autonomous cooperation involves the design of a novel NN architecture that learns network-wide cooperation policies. To this end, device interaction mechanisms are classified according to interaction levels. NN unit groups called interaction slices are dedicated to configure heterogeneous cooperation abilities so that a multi-hop coordination of individual NN-empowered devices collectively characterizes a self-organizing management of the network. The feasibility and effectiveness of this NN-oriented formalism are investigated for decentralized power management of wireless ad-hoc networks.

Coverage Control for UAV Swarm Communication Networks: A Distributed Learning Approach

Recently, unmanned aerial vehicle (UAV) swarm communication has drawn much attention in search and rescue (SAR) missions owing to its wide wireless coverage and increasing autonomy in navigation. In this article, we consider maximizing the downlink wireless coverage of a UAV swarm in an unknown mission area by controlling the quasistationary deployments of UAVs. Particularly, the stochastic wireless link failures caused by channel fading and noise in UAV-to-UAV communication links are considered in coverage control. Specifically, due to delay sensitivity and onboard energy limitation of UAV-enabled SAR networks, we study a distributed control strategy where swarm UAVs can address the coverage problem by exchanging only the local information. In this case, the wireless coverage problem is divided into several distributed optimization subproblems. However, due to the integer variable and nonlinear constraints, each subproblem is nonconvex and mutually coupling, which makes it difficult to solve via standard convex optimization solvers. Thus, we model the UAV swarm network as an undirected random graph and then solve the optimization subproblems by formulating a UAV swarm wireless coverage game. As per the designed utility function and potential function of the formulated game, existence of the pure Nash equilibrium is discussed and a distributed algorithm is developed to achieve the best Nash equilibrium. We analyze the convergence property and computational complexity of the proposed algorithm. Meanwhile, we analyze effects of the initial learning rate and step size on algorithm performance from both theoretical and simulation results. Simulation results show that the proposed algorithm improves the coverage by around 58% when compared with initial performance.