Distributed Learning Approach Research Articles

Representing Camera Response Function by a Single Latent Variable and Fully Connected Neural Network

Modelling the mapping from scene irradiance to image intensity is essential for many computer vision tasks. Such mapping is known as the camera response. Most digital cameras use a nonlinear function to map irradiance, as measured by the sensor to an image intensity used to record the photograph. Modelling of the response is necessary for the nonlinear calibration. In this paper, a new high-performance camera response model that uses a single latent variable and fully connected neural network is proposed. The model is produced using unsupervised learning with an autoencoder on real-world (example) camera responses. Neural architecture searching is then used to find the optimal neural network architecture. A latent distribution learning approach was introduced to constrain the latent distribution. The proposed model achieved state-of-the-art CRF representation accuracy in a number of benchmark tests, but is over twice as fast as the best current models when performing the maximum likelihood estimation during camera response calibration due to the simple yet efficient model representation.

Supervised Approach to Identify Autism Spectrum Neurological Disorder via Label Distribution Learning.

Autism Spectrum Disorder (ASD) is a complicated collection of neurodevelopmental illnesses characterized by a variety of developmental defects. It is a binary classification system that cannot cope with reality. Furthermore, ASD, data label noise, high dimension, and data distribution imbalance have all hampered the existing classification algorithms. As a result, a new ASD was proposed. This strategy employs label distribution learning (LDL) to deal with label noise and uses support vector regression (SVR) to deal with sample imbalance. The experimental results show that the proposed method balances the effects of majority and minority classes on outcomes. It can effectively deal with imbalanced data in ASD diagnosis, and it can help with ASD diagnosis. This study presents a cost-sensitive approach to correct sample imbalance and uses a support vector regression (SVR)-based method to remove label noise. The label distribution learning approach overcomes high-dimensional feature classification issues by mapping samples to the feature space and then diagnosing multiclass ASD. This technique outperforms previous methods in terms of classification performance and accuracy, as well as resolving the issue of unbalanced data in ASD diagnosis.

A communication-efficient method for generalized linear regression with ℓ0 regularization

In this paper, based on the Karush–Kuhn–Tucker (KKT) conditions of regularization, we propose a communication-efficient distributed learning approach for high-dimensional and sparse generalized linear models with massive data sets stored across different machines. This proposed method is a support detection and root finding method for generalized linear models in a distributed form. In each round of the proposed method, the support set is first determined by the primal and dual information reduced to the master machine, then the reduced maximum likelihood estimator is obtained by the gradient descent method, among which it only suffices to calculate the gradient vectors on each machine and communicate them instead of the data. We give the optimal -norm error bound for the sequences generated by the proposed algorithm and show that this -norm error bound decays exponentially to the optimal order. Moreover, we show that the oracle estimator can be recovered if the target signal is not less than the detectable level. In addition, an adaptive version of the proposed algorithm is developed to estimate the sparsity level. Simulation studies illustrate the superior performance of the proposed methods.

Communication-Efficient Vertical Federated Learning

Federated learning (FL) is a privacy-preserving distributed learning approach that allows multiple parties to jointly build machine learning models without disclosing sensitive data. Although FL has solved the problem of collaboration without compromising privacy, it has a significant communication overhead due to the repetitive updating of models during training. Several studies have proposed communication-efficient FL approaches to address this issue, but adequate solutions are still lacking in cases where parties must deal with different data features, also referred to as vertical federated learning (VFL). In this paper, we propose a communication-efficient approach for VFL that compresses the local data of clients, and then aggregates the compressed data from all clients to build an ML model. Since local data are shared in compressed form, the privacy of these data is preserved. Experiments on publicly available benchmark datasets using our proposed method show that the final model obtained by aggregation of compressed data from clients outperforms the performance of the local models of the clients.

Federated edge learning for the wireless physical layer: Opportunities and challenges

Deep learning (DL) has been applied to the physical layer of wireless communication systems, which directly extracts environment knowledge from data and outperforms conventional methods either in accuracy or computation complexity. However, most related research works employ centralized training that inevitably involves collecting training data from edge devices. The data uploading process usually results in excessive communication overhead and privacy disclosure. Alternatively, a distributed learning approach named federated edge learning (FEEL) is introduced to physical layer designs. In FEEL, all devices collaborate to train a global model only by exchanging parameters with a nearby access point. Because all datasets are kept local, data privacy is better protected and data transmission overhead can be reduced. This paper reviews the studies on applying FEEL to the wireless physical layer including channel state information acquisition, transmitter, and receiver design, which represent a paradigm shift of the DL-based physical layer design. In the meantime they also reveal several limitations inherent in FEEL, particularly when applied to the wireless physical layer, thus motivating further research efforts in the field.

SplitFed: When Federated Learning Meets Split Learning

Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. Both follow a model-to-data scenario; clients train and test machine learning models without sharing raw data. SL provides better model privacy than FL due to the machine learning model architecture split between clients and the server. Moreover, the split model makes SL a better option for resource-constrained environments. However, SL performs slower than FL due to the relay-based training across multiple clients. In this regard, this paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches eliminating their inherent drawbacks, along with a refined architectural configuration incorporating differential privacy and PixelDP to enhance data privacy and model robustness. Our analysis and empirical results demonstrate that (pure) SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients. Furthermore, as in SL, its communication efficiency over FL improves with the number of clients. Besides, the performance of SFL with privacy and robustness measures is further evaluated under extended experimental settings.

On Predictive Modeling Using a New Flexible Weibull Distribution and Machine Learning Approach: Analyzing the COVID-19 Data

Predicting and modeling time-to-events data is a crucial and interesting research area. For modeling and predicting such types of data, numerous statistical models have been suggested and implemented. This study introduces a new statistical model, namely, a new modified flexible Weibull extension (NMFWE) distribution for modeling the mortality rate of COVID-19 patients. The introduced model is obtained by modifying the flexible Weibull extension model. The maximum likelihood estimators of the NMFWE model are obtained. The evaluation of the estimators of the NMFWE model is assessed in a simulation study. The flexibility and applicability of the NMFWE model are established by taking two datasets representing the mortality rates of COVID-19-infected persons in Mexico and Canada. For predictive modeling, we consider two pure statistical models and two machine learning (ML) algorithms. The pure statistical models include the autoregressive moving average (ARMA) and non-parametric autoregressive moving average (NP-ARMA), and the ML algorithms include neural network autoregression (NNAR) and support vector regression (SVR). To evaluate their forecasting performance, three standard measures of accuracy, namely, root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE) are calculated. The findings demonstrate that ML algorithms are very effective at predicting the mortality rate data.

Scalable Malware Detection System Using Distributed Deep Learning

The number of complex and novel malware attacks is increasing exponentially in the cyberworld. Malware detection systems are facing new challenges due to the volume, velocity, and complexity of malware. The current malware detection system relies on a time-consuming, resource-intensive, and knowledge-intensive classification approach. Most of the existing malware detection system is ineffective in detecting novel malware attacks. A deep learning approach can be used to build a malware detection system that can effectively detect novel malware attacks without much human intervention. The current circumstance necessitates not just a malware system with excellent accuracy, but also one that can serve a large volume of demand in near real-time. A scalable malware detection system capable of detecting complex attacks is the need of time. This article discusses a scalable and distributed deep learning approach for malware detection using convolutional neural network and bidirectional long short-term memory (CNN-BiLSTM). The deep learning approach has been used to make the system learn and make predictive decisions without human intervention. The performance of the deep learning approach depends on various parameters and training data sets. Hence, different combinations of deep learning algorithms have been used to design and test the models to achieve the desired result. The experimental results show that the double layer of CNN and BiLSTM has better performance than single-layer CNN.

Distributed learning of fully connected neural networks using independent subnet training

Distributed machine learning (ML) can bring more computational resources to bear than single-machine learning, thus enabling reductions in training time. Distributed learning partitions models and data over many machines, allowing model and dataset sizes beyond the available compute power and memory of a single machine. In practice though, distributed ML is challenging when distribution is mandatory, rather than chosen by the practitioner. In such scenarios, data could unavoidably be separated among workers due to limited memory capacity per worker or even because of data privacy issues. There, existing distributed methods will utterly fail due to dominant transfer costs across workers, or do not even apply. We propose a new approach to distributed fully connected neural network learning, called independent subnet training (IST), to handle these cases. In IST, the original network is decomposed into a set of narrow subnetworks with the same depth. These subnetworks are then trained locally before parameters are exchanged to produce new subnets and the training cycle repeats. Such a naturally "model parallel" approach limits memory usage by storing only a portion of network parameters on each device. Additionally, no requirements exist for sharing data between workers (i.e., subnet training is local and independent) and communication volume and frequency are reduced by decomposing the original network into independent subnets. These properties of IST can cope with issues due to distributed data, slow interconnects, or limited device memory, making IST a suitable approach for cases of mandatory distribution. We show experimentally that IST results in training times that are much lower than common distributed learning approaches.

Semi-asynchronous personalized federated learning for short-term photovoltaic power forecasting

Accurate forecasting for photovoltaic power generation is one of the key enablers for the integration of solar photovoltaic systems into power grids. Existing deep-learning-based methods can perform well if there are sufficient training data and enough computational resources. However, there are challenges in building models through centralized shared data due to data privacy concerns and industry competition. Federated learning is a new distributed machine learning approach which enables training models across edge devices while data reside locally. In this paper, we propose an efficient semi-asynchronous federated learning framework for short-term solar power forecasting and evaluate the framework performance using a CNN-LSTM model. We design a personalization technique and a semi-asynchronous aggregation strategy to improve the efficiency of the proposed federated forecasting approach. Thorough evaluations using a real-world dataset demonstrate that the federated models can achieve significantly higher forecasting performance than fully local models while protecting data privacy, and the proposed semi-asynchronous aggregation and the personalization technique can make the forecasting framework more robust in real-world scenarios.

Efficient-Grad: Efficient Training Deep Convolutional Neural Networks on Edge Devices with Grad ient Optimizations

With the prospering of mobile devices, the distributed learning approach, enabling model training with decentralized data, has attracted great interest from researchers. However, the lack of training capability for edge devices significantly limits the energy efficiency of distributed learning in real life. This article describes Efficient-Grad, an algorithm-hardware co-design approach for training deep convolutional neural networks, which improves both throughput and energy saving during model training, with negligible validation accuracy loss. The key to Efficient-Grad is its exploitation of two observations. Firstly, the sparsity has potential for not only activation and weight, but gradients and the asymmetry residing in the gradients for the conventional back propagation (BP). Secondly, a dedicated hardware architecture for sparsity utilization and efficient data movement can be optimized to support the Efficient-Grad algorithm in a scalable manner. To the best of our knowledge, Efficient-Grad is the first approach that successfully adopts a feedback-alignment (FA)-based gradient optimization scheme for deep convolutional neural network training, which leads to its superiority in terms of energy efficiency. We present case studies to demonstrate that the Efficient-Grad design outperforms the prior arts by 3.72x in terms of energy efficiency.

Joint Coding and Scheduling Optimization for Distributed Learning Over Wireless Edge Networks

Unlike theoretical analysis of distributed learning (DL) in the literature, DL over wireless edge networks faces the inherent dynamics/uncertainty of wireless connections and edge nodes, making DL less efficient or even inapplicable under the highly dynamic wireless edge networks. This article addresses these problems by leveraging recent advances in coded computing and the deep dueling neural network architecture. By introducing coded structures/redundancy, a distributed learning task can be completed without waiting for straggling nodes. Unlike conventional coded computing that only optimizes the code structure, coded distributed learning over the wireless edge also requires to optimize the selection/scheduling of wireless edge nodes with heterogeneous connections, computing capability, and straggling effects. However, even neglecting the aforementioned dynamics/uncertainty, the resulting joint optimization of coding and scheduling to minimize the distributed learning time turns out to be NP-hard. To tackle this and to account for the dynamics and uncertainty of wireless connections and edge nodes, we reformulate the problem as a Markov Decision Process and design a novel deep reinforcement learning algorithm that employs the deep dueling neural network architecture to find the jointly optimal coding scheme and the best set of edge nodes for different learning tasks without explicit information about the wireless environment and edge nodes’ straggling parameters. Simulations show that the proposed framework reduces the average learning delay in wireless edge computing up to 66% compared with other DL approaches. The jointly optimal framework in this article is also applicable to any distributed learning scheme with heterogeneous and uncertain computing nodes.

On the implementation of a new version of the Weibull distribution and machine learning approach to model the COVID-19 data.

Statistical methodologies have broader applications in almost every sector of life including education, hydrology, reliability, management, and healthcare sciences. Among these sectors, statistical modeling and predicting data in the healthcare sector is very crucial. In this paper, we introduce a new method, namely, a new extended exponential family to update the distributional flexibility of the existing models. Based on this approach, a new version of the Weibull model, namely, a new extended exponential Weibull model is introduced. The applicability of the new extended exponential Weibull model is shown by considering two data sets taken from the health sciences. The first data set represents the mortality rate of the patients infected by the coronavirus disease 2019 (COVID-19) in Mexico. Whereas, the second set represents the mortality rate of COVID-19 patients in Holland. Utilizing the same data sets, we carry out forecasting using three machine learning (ML) methods including support vector regression (SVR), random forest (RF), and neural network autoregression (NNAR). To assess their forecasting performances, two statistical accuracy measures, namely, root mean square error (RMSE) and mean absolute error (MAE) are considered. Based on our findings, it is observed that the RF algorithm is very effective in predicting the death rate of the COVID-19 data in Mexico. Whereas, for the second data, the SVR performs better as compared to the other methods.

Accelerating privacy-preserving momentum federated learning for industrial cyber-physical systems

Federated learning (FL) is a distributed learning approach, which allows the distributed computing nodes to collaboratively develop a global model while keeping their data locally. However, the issues of privacy-preserving and performance improvement hinder the applications of the FL in the industrial cyber-physical systems (ICPSs). In this work, we propose a privacy-preserving momentum FL approach, named PMFL, which uses the momentum term to accelerate the model convergence rate during the training process. Furthermore, a fully homomorphic encryption scheme CKKS is adopted to encrypt the gradient parameters of the industrial agents’ models for preserving their local privacy information. In particular, the cloud server calculates the global encrypted momentum term by utilizing the encrypted gradients based on the momentum gradient descent optimization algorithm (MGD). The performance of the proposed PMFL is evaluated on two common deep learning datasets, i.e., MNIST and Fashion-MNIST. Theoretical analysis and experiment results confirm that the proposed approach can improve the convergence rate while preserving the privacy information of the industrial agents.

FedParking: A Federated Learning Based Parking Space Estimation With Parked Vehicle Assisted Edge Computing

As a distributed learning approach, federated learning trains a shared learning model over distributed datasets while preserving the training data privacy. We extend the application of federated learning to parking management and introduce FedParking in which Parking Lot Operators (PLOs) collaborate to train a long short-term memory model for parking space estimation without exchanging the raw data. Furthermore, we investigate the management of Parked Vehicle assisted Edge Computing (PVEC) by FedParking. In PVEC, different PLOs recruit PVs as edge computing nodes for offloading services through an incentive mechanism, which is designed according to the computation demand and parking capacity constraints derived from FedParking. We formulate the interactions among the PLOs and vehicles as a multi-lead multi-follower Stackelberg game. Considering the dynamic arrivals of the vehicles and time-varying parking capacity constraints, we present a multi-agent deep reinforcement learning approach to gradually reach the Stackelberg equilibrium in a distributed yet privacy-preserving manner. Finally, numerical results are provided to demonstrate the effectiveness and efficiency of our scheme.

Implementation of the Australian Computer-Assisted Theragnostics (AusCAT) network for radiation oncology dataextraction, reporting and distributed learning.

There is significant potential to analyse and model routinely collected data for radiotherapy patients to provide evidence to support clinical decisions, particularly where clinical trials evidence is limited or non-existent. However, in practice there are administrative, ethical, technical, logistical and legislative barriers to having coordinated data analysis platforms across radiation oncology centres. A distributed learning network of computer systems is presented, with software tools to extract and report on oncology data and to enable statistical model development. A distributed or federated learning approach keeps data in the local centre, but models are developed from the entire cohort. The feasibility of this approach is demonstrated across six Australian oncology centres, using routinely collected lung cancer data from oncology information systems. The infrastructure was used to validate and develop machine learning for model-based clinical decision support and for one centre to assess patient eligibility criteria for two major lung cancer radiotherapy clinical trials (RTOG-9410, RTOG-0617). External validation of a 2-year overall survival model for non-small cell lung cancer (NSCLC) gave an AUC of 0.65 and C-index of 0.62 across the network. For one centre, 65% of Stage III NSCLC patients did not meet eligibility criteria for either of the two practice-changing clinical trials, and these patients had poorer survival than eligible patients (10.6m vs. 15.8m, P=0.024). Population-based studies on routine data are possible using a distributed learning approach. This has the potential for decision support models for patients for whom supporting clinical trial evidence is not applicable.

Continuous learning of emergent behavior in robotic matter

One of the main challenges in robotics is the development of systems that can adapt to their environment and achieve autonomous behavior. Current approaches typically aim to achieve this by increasing the complexity of the centralized controller by, e.g., direct modeling of their behavior, or implementing machine learning. In contrast, we simplify the controller using a decentralized and modular approach, with the aim of finding specific requirements needed for a robust and scalable learning strategy in robots. To achieve this, we conducted experiments and simulations on a specific robotic platform assembled from identical autonomous units that continuously sense their environment and react to it. By letting each unit adapt its behavior independently using a basic Monte Carlo scheme, the assembled system is able to learn and maintain optimal behavior in a dynamic environment as long as its memory is representative of the current environment, even when incurring damage. We show that the physical connection between the units is enough to achieve learning, and no additional communication or centralized information is required. As a result, such a distributed learning approach can be easily scaled to larger assemblies, blurring the boundaries between materials and robots, paving the way for a new class of modular "robotic matter" that can autonomously learn to thrive in dynamic or unfamiliar situations, for example, encountered by soft robots or self-assembled (micro)robots in various environments spanning from the medical realm to space explorations.

Distributed Spectrum Sensing for IoT Networks: Architecture, Challenges, and Learning

Spectrum sensing is believed to be a prominent solution to spectrum scarcity caused by the presence of a large number of devices, particularly in Internet of Things (IoT) applications. Providing spectrum access to all of these devices is one of the paramount issues for I systems. Nevertheless, IoT poses several challenges for spectrum sensing that have yet to be overcome. Conventional spectrum sensing techniques have to be carefully modified to be applied to sophisticated and scalable IoT systems. In this paper, an analysis of spectrum sensing for IoT and its possible architecture configurations are presented. We provide an extensive list of challenges associated with spectrum sensing for IoT systems. Focus is given to distributed learning approaches known as incremental, consensus, and diffusion learning in the context of IoT. We further present a case study on cooperative spectrum sensing for IoT systems, where we propose an optimized distributed solution based on diffusion learning. Finally, simulation results demonstrate that the proposed solution improves detection performance and aggregate secondary IoT network throughput, and can minimize hardware complexity for secondary IoT users.

Resource Management and Model Personalization for Federated Learning over Wireless Edge Networks

Client and Internet of Things devices are increasingly equipped with the ability to sense, process, and communicate data with high efficiency. This is resulting in a major shift in machine learning (ML) computation at the network edge. Distributed learning approaches such as federated learning that move ML training to end devices have emerged, promising lower latency and bandwidth costs and enhanced privacy of end users’ data. However, new challenges that arise from the heterogeneous nature of the devices’ communication rates, compute capabilities, and the limited observability of the training data at each device must be addressed. All these factors can significantly affect the training performance in terms of overall accuracy, model fairness, and convergence time. We present compute-communication and data importance-aware resource management schemes optimizing these metrics and evaluate the training performance on benchmark datasets. We also develop a federated meta-learning solution, based on task similarity, that serves as a sample efficient initialization for federated learning, as well as improves model personalization and generalization across non-IID (independent, identically distributed) data. We present experimental results on benchmark federated learning datasets to highlight the performance gains of the proposed methods in comparison to the well-known federated averaging algorithm and its variants.

Adaptive Load Balanced Routing in IOT Networks: A Distributed Learning Approach

Facilitating large-scale load-efficient Internet of things (IoT) connectivity is a vital step toward realizing the networked society. Although legacy wide-area wireless systems are heavily based on network-side coordination, such centralized methods will become infeasible in the future, by the unbalanced signaling level and the expected increment in the number of IoT devices. In the present work, this problem is represented through self-coordinating for IoT networks and learning from past communications. In this regard, first, we assessed low-complexity distributed learning methods that can be applied to IoT communications. We presented a learning solution then, for adapting devices’ communication parameters to the environment to maximize the reliability and load balancing efficiency in data transmissions. Moreover, we used leveraging instruments from stochastic geometry to assess the behavior of the presented distributed learning solution against centralized coordinations. Ultimately, we analyzed the interplay amongst traffic efficiency, communications’ reliability against interference and noise over data channel, as well as reliability versus adversarial interference over feedback and data channels. The presented learning approach enhanced both reliability and traffic efficiency within IoT communications considerably. By such promising findings obtained via lightweight learning, our solution becomes promising in numerous low-power low-cost IoT uses.