Distributed Deep Neural Network Research Articles

An efficient cloud-integrated distributed deep neural network framework for IoT malware classification

The proliferation of interconnected devices in the Internet of Things (IoT) landscape has introduced significant security concerns. With the integration of android devices, the potential for attackers to exploit vulnerabilities becomes a crucial issue. Timely detection of malware attacks has emerged as a critical challenge, especially in industrial IoT (IIoT), to prevent permanent damage. The slow inference speed of the current algorithms causes limited performance and a significant delay in the detection and mitigation of malware, especially in large-scale environments such as IIoT. High resource utilization can limit the size of the model (number of parameters), which could be implemented in embedded devices, which causes lower accuracy of the model. Hence, we proposed a HierarchicalCloudDNN, a distributed malware detection framework that used modified state-of-the-art deep learning algorithms in a boosting ensemble architecture. The proposed method could scale efficiently from IIoT devices up to the edge and cloud to speed up malware detection, simultaneously reduce resource utilization, and maintain a controllable level of accuracy compared to rival methods in malware detection. The objective is to minimize both run-time and resource consumption while maintaining performance characteristics. The performance analysis of the HierarchicalCloudDNN method was evaluated using standard performance measures on the frequently used BIG 2015 dataset. Based on evaluations, the proposed approach has demonstrated an impressive accuracy rate of 98.90%, outperforming many state-of-the-art models in performance, response time, and resource utilization. These findings suggest a notable decrease in both false positives and false negatives while maintaining accuracy. The proposed distributed malware analysis framework is scalable, robust, and helpful for the timely detection of malicious activity, especially in geographically distributed environments such as the IIoT.

Reliable data transmission for a VANET-IoIT architecture: A DNN approach

The challenges and resilience of vehicular ad hoc network (VANET) and deep neural network (DNN) hybrid architectures in terms of reliability in smart cities have attracted much global interest stemming from the rollout of the next generation of intelligent networks. In this paper, we propose a novel distributed DNN (D-DNN) scheme with blockchain to support the Internet of Intelligent Things (IoIT) infrastructure in the VANET environment of the future. In particular, because the communication links between edge nodes are very unstable in VANETs, a new neuro-fuzzy server that serves the dual roles of finding reliable links between edge nodes and performing optimal routing path selection is proposed. Next, a blockchain layer is employed at the edge nodes, which are initially scrutinized before establishing communication links to ensure reliability during data transfer. Then, the proposed D-DNN (PD-DNN) scheme is applied to enhance the performance of the VANET-IoIT architecture by improving the data flow and convergence rate and mitigating erratic variations in output. To address reliability concerns, the coverage probability (CP) metric is investigated as a measure of network connectivity. Furthermore, we present an analysis of the PD-DNN scheme in comparison with the traditional DNN (T-DNN) scheme. Finally, simulation results for VANET-IoIT scenarios show that, subject to data protection and privacy constraints, the CP values corresponding to different communication links are improved to a greater extent under our scheme than under the traditional scheme, demonstrating the feasibility of the proposed scheme.

Functions as a service for distributed deep neural network inference over the cloud‐to‐things continuum

AbstractThe use of serverless computing has been gaining popularity in recent years as an alternative to traditional Cloud computing. We explore the usability and potential development benefits of three popular open‐source serverless platforms in the context of IoT: OpenFaaS, Fission, and OpenWhisk. To address this we discuss our experience developing a serverless and low‐latency Distributed Deep Neural Network (DDNN) application. Our findings indicate that these serverless platforms require significant resources to operate and are not ideal for constrained devices. In addition, we archived a 55% improvement compared to Kafka‐ML's performance under load, a framework without dynamic scaling support, demonstrating the potential of serverless computing for low‐latency applications.

A Novel Approach to Improving Distributed Deep Neural Networks over Cloud Computing

In recent years, deep distributed neural networks (DDNNs) and neural networks (NN) have excelled in an extensive list of applications. For example, deep convolutional neural networks (DCNNs) are constantly gaining new features in various tasks in computer vision. At the same time, the number of end devices, including Internet of Things (IoT) devices has increased prominently. These devices are attractive targets for machine learning applications because they are often directly connected to sensors. For example (cameras, microphones, and gyroscopes) that record large amounts of input data in a stream mode. This study presents the design of a DDNN with end devices, edges, and clouds that spans computer hierarchies. The idea presented is considered one of the new ideas because it depends on two layers to distinguish, namely the convolutional layer and the pooling layer. The main objective behind using these two layers in one proposal is to provide and obtain the best results. Finally, we discovered that the proposed technique produced the best results in terms of accuracy and cost, with the precision of the definition reaching 99 % and the cost being quite affordable at 25. As a result, we conclude that these results are far superior to those achieved by the researchers in their ideas provided in previous recent literature.

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

Synchronous strategies with data parallelism are widely utilized in distributed training of Deep Neural Networks (DNNs), largely owing to their easy implementation yet promising performance. In these strategies, the workers with different computational capabilities need to wait for each other because of the essential gradient or weight synchronization. This will inevitably cause the high-performance workers to waste time waiting for the weak computational workers, which in turn results in the inefficiency of the cluster. In this paper, we propose a Dynamic Load Balance (DLB) strategy for the distributed training of DNNs to tackle this issue. Specifically, the performance of each worker is evaluated first based on the performance demonstration during the previous training epochs, and then the batch size and dataset partition are adaptively adjusted in consideration of the current performance of the workers. As a result, the waiting cost among the workers will be eliminated, thereby the utilization of the clusters is highly improved. Furthermore, the essential theoretical analysis has also been provided to justify the convergence of the proposed algorithm. Extensive experiments have been conducted on the CIFAR10 and CIFAR100 benchmark datasets with four state-of-the-art DNN models. The experimental results indicate that the proposed algorithm can significantly improve the utilization of the distributed cluster. In addition, the proposed algorithm can also prevent the load imbalance of the distributed DNN training from being affected by the disturbance and can be employed flexibly in conjunction with the other synchronous distributed DNN training methods.

WH 2 D 2 N 2 : Distributed AI-enabled OK-ASN Service for Web of Things

Model data-driven ontology and knowledge presentation for evolving semantic Asian social networks (OK-ASN) is a critical strategy for web of things (WoT) services. Meanwhile, Deep Neural Network (DNN)-based OK-ASN service in WoT is growing rapidly. However, most DNN-based services cannot utilize the potential of WoT fully, as heterogeneity exists in WoT. Therefore, this article proposes a novel framework called Web-based Heterogeneous Hierarchical Distributed Deep Neural Network ( WH 2 D 2 N 2 ) to deploy the DNNs for OK-ASN services on WoT, overcoming the heterogeneity. The architecture of the system and the designed Edge-Cloud-Joint execute scheme utilize heterogeneous devices to make DNN inference ubiquitous and output two types of results to meet various requirements. To bring robustness to OK-ASN services, a global scheduling is designed to arrange the workflow dynamically. The results of our experiments prove the efficiency of the execute scheme and the global scheduling in the system.

Filtering Out High Noise Data for Distributed Deep Neural Networks

Artificial intelligence-based cyber-physical systems (CPS) applications have been spread across various fields such as smart cities, medical services, and industrial controls. When CPS devices are connected to a cloud server, big data streams generated by CPS devices impose enormous bandwidth pressure and exert excessive compute loads to the cloud server. Due to unpredictable environments and uncertainty in reality, these issues are mainly attributed to a large amount of high noise data captured and uploaded by CPS devices. To overcome these issues, this paper proposes a cyber-physical-cloud based framework for distributed deep neural networks (DDNNs) to prevent high noise data from being uploaded to the cloud. The proposed framework features a lightweight data filtering module enabled by depthwise separable convolutions to identify and filter out the high noise data that the cloud cannot recognize. Extensive experimental results demonstrate that the proposed data filtering module can achieve an accuracy of up to 83.72% in identifying high noise data and the proposed framework can effectively save bandwidth of up to 63.42% as compared to benchmarking methods. Note to Practitioners—This paper is motivated by the problems of enormous bandwidth pressure and excessive cloud compute loads in cyber-physical-cloud distributed computing paradigms. These problems are mainly caused by high noise data generated by CPS devices, because CPS devices often work in disturbing and unstable environments and there are uncontrollable uncertainties in reality. Especially for the emerging artificial intelligence-driven cyber-physical-cloud distributed paradigms, there is no existing research to solve the unnecessary transmission and cloud compute loads caused by high noise data. To tackle the challenge, this paper develops a novel cyber-physical-cloud distributed framework with data filtering capabilities to prevent high noise data from being uploaded. The proposed framework supports two popular loosely coupled and closely coupled distributed computing paradigms. Extensive experiments confirm that the proposed cyber-physical-cloud distributed framework can efficiently filter out high noise data and alleviate unnecessary transmission and needless cloud compute loads introduced by high noise data.

DNN Partitioning for Inference Throughput Acceleration at the Edge

Deep neural network (DNN) inference on streaming data requires computing resources to satisfy inference throughput requirements. However, latency and privacy sensitive deep learning applications cannot afford to offload computation to remote clouds because of the implied transmission cost and lack of trust in third-party cloud providers. Among solutions to increase performance while keeping computation on a constrained environment, hardware acceleration can be onerous, and model optimization requires extensive design efforts while hindering accuracy. DNN partitioning is a third complementary approach, and consists of distributing the inference workload over several available edge devices, taking into account the edge network properties and the DNN structure, with the objective of maximizing the inference throughput (number of inferences per second). This paper introduces a method to predict inference and transmission latencies for multi-threaded distributed DNN deployments, and defines an optimization process to maximize the inference throughput. A branch and bound solver is then presented and analyzed to quantify the achieved performance and complexity. This analysis has led to the definition of the acceleration region, which describes deterministic conditions on the DNN and network properties under which DNN partitioning is beneficial. Finally, experimental results confirm the simulations and show inference throughput improvements in sample edge deployments.

Resource-Efficient Distributed Deep Neural Networks Empowered by Intelligent Software-Defined Networking

Contemporary machine learning methods have evolved from conventional algorithms to deep neural networks (DNNs) that are computation- and data- intensive. Thus, they are suitable to be deployed in the cloud that can offer high computational capacity and scalable resources. However, the cloud computing paradigm is not optimal for delay- and energy-sensitive applications. To mitigate these problems, a battery of distributed DNNs have been proposed to allow a fast inference with device-edge-cloud synergy. Furthermore, although distributed deployment of DNNs on real communication networks is an important research topic, the legacy network architecture cannot meet the requirements of these distributed deep neural networks due to the complicated management and manual configuration, etc. To cope with these requirements, we develop a novel and explicit Intelligent Software Defined Networking (ISDN) that aims to manage the bandwidth and computing resources across the network via the SDN paradigm. We first identify the difficulties of deploying distributed intelligent computing in the current network architecture. Then, we explain how to address these problems by introducing the ISDN architecture. Specifically, we develop a dynamic routing method to enable Quality-of-Service (QoS) communication based on the SDN paradigm and propose a Markov Decision Process (MDP) based dynamic task offloading model to achieve the optimal offloading policy of DNN tasks. We develop a simulation platform based on Mininet to measure its performance advantages over traditional architectures. Extensive experimental results show that compared with the traditional network architecture, our architecture based on the SDN paradigm can perform better in terms of both network throughput and resource utilization.

FLEE: A Hierarchical Federated Learning Framework for Distributed Deep Neural Network over Cloud, Edge, and End Device

With the development of smart devices, the computing capabilities of portable end devices such as mobile phones have been greatly enhanced. Meanwhile, traditional cloud computing faces great challenges caused by privacy-leakage and time-delay problems, there is a trend to push models down to edges and end devices. However, due to the limitation of computing resource, it is difficult for end devices to complete complex computing tasks alone. Therefore, this article divides the model into two parts and deploys them on multiple end devices and edges, respectively. Meanwhile, an early exit is set to reduce computing resource overhead, forming a hierarchical distributed architecture. In order to enable the distributed model to continuously evolve by using new data generated by end devices, we comprehensively consider various data distributions on end devices and edges, proposing a hierarchical federated learning framework FLEE , which can realize dynamical updates of models without redeploying them. Through image and sentence classification experiments, we verify that it can improve model performances under all kinds of data distributions, and prove that compared with other frameworks, the models trained by FLEE consume less global computing resource in the inference stage.

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

Effective parallelization strategies are crucial for the performance of distributed deep neural network (DNN) training. Recently, several methods have been proposed to search parallelization strategies but they all optimize a single objective (e.g., execution time, memory consumption) and produce only one strategy. We propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Frontier Tracking</i> (FT), an efficient algorithm that finds <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a set of Pareto-optimal parallelization strategies</i> to explore the best trade-off among different objectives. FT can minimize the memory consumption when the number of devices is limited and fully utilize additional resources to reduce the execution time. Based on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FT</i> , we develop a user-friendly system, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TensorOpt</i> , which allows users to run their distributed DNN training jobs without caring the details about searching and coding parallelization strategies. Experimental results show that TensorOpt is more flexible in adapting to resource availability compared with existing frameworks.

Psychological Mobilization of Innovative Teaching Methods for Students' Basic Educational Curriculum Reform Under Deep Learning.

Educational innovation reform is used as the background. In response to the need to propose innovative educational programs, the concepts of Distributed Deep Neural Network (DDNN) and deep learning under edge computing are used as the basis. A teaching program for Science Technology Engineering Mathematics (STEM) is proposed. The average training method is used to verify the performance of the model. Sampling rate means the number of samples per second taken from a continuous signal to form a discrete signal. The accuracy and sample ratio obtained are higher than 95%. The communication volume is 309 bytes, which is in a good range. On this basis, a university uses STEM teaching plans and questionnaires to influence the psychological mobilization factors of students' deep learning effects. Challenging learning tasks and learning motivation have the greatest impact on deep learning, and conclusions that both are positive effects are obtained. Therefore, STEM innovative teaching programs can be widely used. The plan provides a reference theory for improving teaching innovation in the context of the basic educational curriculum reform in China. STEM curriculum is the dual subject of teachers and students, and the learning community includes multi-stakeholders. There are hierarchical relationships among the subjects. In terms of financial support, the first two funds come from the school. Learning communities have dedicated sponsorship partners complemented by clear financial planning. There is not much difference in course resources. Still, the learning community will provide more diversified media forms and special websites, and other auxiliary resources are open to all users. They can obtain first-hand resources without applying. In terms of project form, in addition to the core classroom teaching, the latter two can provide richer activities and realize the diversity of time, space, and information exchange.

Intelligent task offloading and collaborative computation in multi-UAV-enabled mobile edge computing

This article establishes a three-tier mobile edge computing (MEC) network, which takes into account the cooperation between unmanned aerial vehicles (UAVs). In this MEC network, we aim to minimize the processing delay of tasks by jointly optimizing the deployment of UAVs and offloading decisions, while meeting the computing capacity constraint of UAVs. However, the resulting optimization problem is nonconvex, which cannot be solved by general optimization tools in an effective and efficient way. To this end, we propose a two-layer optimization algorithm to tackle the non-convexity of the problem by capitalizing on alternating optimization. In the upper level algorithm, we rely on differential evolution (DE) learning algorithm to solve the deployment of the UAVs. In the lower level algorithm, we exploit distributed deep neural network (DDNN) to generate offloading decisions. Numerical results demonstrate that the two-layer optimization algorithm can effectively obtain the near-optimal deployment of UAVs and offloading strategy with low complexity.

An open source framework based on Kafka-ML for Distributed DNN inference over the Cloud-to-Things continuum

The current dependency of Artificial Intelligence (AI) systems on Cloud computing implies higher transmission latency and bandwidth consumption. Moreover, it challenges the real-time monitoring of physical objects, e.g., the Internet of Things (IoT). Edge systems bring computing closer to end devices and support time-sensitive applications. However, Edge systems struggle with state-of-the-art Deep Neural Networks (DNN) due to computational resource limitations. This paper proposes a technology framework that combines the Edge-Cloud architecture concept with BranchyNet advantages to support fault-tolerant and low-latency AI predictions. The implementation and evaluation of this framework allow assessing the benefits of running Distributed DNN (DDNN) in the Cloud-to-Things continuum. Compared to a Cloud-only deployment, the results obtained show an improvement of 45.34% in the response time. Furthermore, this proposal presents an extension for Kafka-ML that reduces rigidness over the Cloud-to-Things continuum managing and deploying DDNN.

Accelerating distributed deep neural network training with pipelined MPI allreduce

TensorFlow (TF) is usually combined with the Horovod (HVD) workload distribution package to obtain a parallel tool to train deep neural network on clusters of computers. HVD in turn utilizes a blocking Allreduce primitive to share information among processes, combined with a communication thread to overlap communication with computation. In this work, we perform a thorough experimental analysis to expose (1) the importance of selecting the best algorithm in MPI libraries to realize the Allreduce operation; and (2) the performance acceleration that can be attained when replacing a blocking Allreduce with its non-blocking counterpart (while maintaining the blocking behaviour via the appropriate synchronization mechanism). Furthermore, (3) we explore the benefits of applying pipelining to the communication exchange, demonstrating that these improvements carry over to distributed training via TF+HVD. Finally, (4) we show that pipelining can also boost performance for applications that make heavy use of other collectives, such as Broadcast and Reduce-Scatter.

On designing the adaptive computation framework of distributed deep learning models for Internet-of-Things applications

Deep learning methods have been gradually adopted in the Internet-of-Things (IoT) applications. Nevertheless, the large demands for the computation and memory resources make these methods difficult to be deployed in the field as the resource-constrained IoT devices would be overwhelmed by the computations incurred by the inference operations of the deployed deep learning models. In this article, we propose the adaptive computation framework, which is built on top of the distributed deep neural networks (DDNNs), to facilitate the inference computations of a trained DDNN model to be collaboratively executed by the machines in a distributed computing hierarchy, e.g., the end device and the cloud server. By facilitating the trained models to be run on the actual distributed systems, the proposed framework enables the co-design of the distributed deep learning models and systems, since the delivered performance of the models on the systems, in terms of the inference time, consumed energy and model accuracy, is able to be measured and is served as the input for the next cycle for the model/system design. We have built the surveillance system for the object detection application with the prototyped framework. We use the surveillance system as a case study to demonstrate the capabilities of the proposed framework. In addition, the design considerations of developing the DDNN system are shared. With the promising results presented in the article, we believe that the framework paves a road for the automated design process of the distributed deep learning models/systems.

Deep Learning-Aided Distributed Transmit Power Control for Underlay Cognitive Radio Network

In this paper, we investigate deep learning-aided distributed transmit power control in the context of an underlay cognitive radio network (CRN). In the proposed scheme, the fully distributed transmit power control strategy of secondary users (SUs) is learned by means of a distributed deep neural network (DNN) structure in an unsupervised manner, such that the average spectral efficiency (SE) of the SUs is maximized whilst allowing the interference on primary users (PUs) to be regulated properly. Unlike previous centralized DNN-based strategies that require complete channel state information (CSI) to optimally determine the transmit power of SU transceiver pairs (TPs), in our proposed scheme, each SU TP determines its own transmit power based solely on its local CSI. Our simulation results verify that the proposed scheme can achieve a near-optimal SE comparable with a centralized DNN-based scheme, with a reduced computation time and no signaling overhead.

Training Deep Neural Network for Optimal Power Allocation in Islanded Microgrid Systems: A Distributed Learning-Based Approach.

Currently, numerical optimization methods are used to solve distributed optimal power allocation (OPA) problems for islanded microgrid (MG) systems. Most of them are developed based on rigorous mathematical derivation. However, the complexity of such optimization algorithms inevitably creates a gap between theoretical analysis and real-time implementation. In order to bridge such a gap, in this article we provide a new distributed learning-based framework to solve the real-time OPA problem. Specifically, inspired by the human-thinking scheme, distributed deep neural networks (DNNs) together with a dynamic average consensus algorithm are first employed to obtain an approximate OPA solution in a distributed manner. Then a distributed balance generation and demand algorithm is designed to fine-tune it to obtain the final optimal feasible solution. In addition, it is theoretically proved that the proposed DNN can well approximate one existing OPA algorithm (Guo et al. 2018), where quantitative numbers of at most how many hidden layers and neurons are provided. Several experimental case studies show that our proposed distributed learning framework can achieve similar optimal results to those obtained by using typical existing distributed numerical optimization methods while it is superior in terms of simplicity and real-time capability.

λDNN: Achieving Predictable Distributed DNN Training With Serverless Architectures

Serverless computing is becoming a promising paradigm for Distributed Deep Neural Network (DDNN) training in the cloud, as it allows users to decompose complex model training into a number of <i>functions</i> without managing virtual machines or servers. Though provided with a simpler resource interface (i.e., function number and memory size), inadequate function resource provisioning (either under-provisioning or over-provisioning) easily leads to <i>unpredictable</i> DDNN training performance in serverless platforms. Our empirical studies on AWS Lambda indicate that, such <i>unpredictable performance</i> of serverless DDNN training is mainly caused by the resource bottleneck of Parameter Servers (PS) and small local batch size. In this article, we design and implement <i><inline-formula><tex-math notation="LaTeX">$\lambda$</tex-math><alternatives><mml:math><mml:mi>&#x03BB;</mml:mi></mml:math><inline-graphic xlink:href="xu-ieq1-3054656.gif"/></alternatives></inline-formula>DNN</i>, a cost-efficient function resource provisioning framework to provide predictable performance for serverless DDNN training workloads, while saving the budget of provisioned functions. Leveraging the PS network bandwidth and function CPU utilization, we build a <i>lightweight</i> analytical DDNN training performance model to enable our design of <i><inline-formula><tex-math notation="LaTeX">$\lambda$</tex-math><alternatives><mml:math><mml:mi>&#x03BB;</mml:mi></mml:math><inline-graphic xlink:href="xu-ieq2-3054656.gif"/></alternatives></inline-formula>DNN</i> resource provisioning strategy, so as to guarantee DDNN training performance with serverless functions. Extensive prototype experiments on AWS Lambda and complementary trace-driven simulations demonstrate that, <i><inline-formula><tex-math notation="LaTeX">$\lambda$</tex-math><alternatives><mml:math><mml:mi>&#x03BB;</mml:mi></mml:math><inline-graphic xlink:href="xu-ieq3-3054656.gif"/></alternatives></inline-formula>DNN</i> can deliver predictable DDNN training performance and save the monetary cost of function resources by up to 66.7 percent, compared with the state-of-the-art resource provisioning strategies, yet with an acceptable runtime overhead.

DDNN based data allocation method for IIoT

Abstract With the complete application of artificial intelligence in the field of industrial production and manufacturing and the rapid development of edge computing, industrial processing sites often need to deploy machine learning tasks at edges and terminals. We propose a data allocation method based on Distributed Deep Neural Networks (DDNN) framework, which allocates data to edge servers or stays locally for processing. DDNN divides deep learning tasks and deploys pre-trained shallow neural networks and deep neural networks at local or edges, respectively. However, all data is processed locally, and the failure is sent to the edge server or the cloud. It will lead to excessive pressure on local terminal equipment and long-term idle edge servers, which cannot meet industrial production’s real-time requirements on user privacy and time-sensitive tasks. In this paper, the complexity and inference error rate of machine learning model, the data processing speed of local equipment and edge server, and the transmission time are comprehensively considered to establish the system model. A joint optimization problem is proposed to minimize the total data processing delay. The optimal solution is derived analytically, and the optimal data allocation methhod is given. Simulation experiments are designed to verify the method’s effectiveness and study the influence of key parameters on the allocation method.