Deep Learning Cluster Research Articles

CyberAIBot: Artificial Intelligence in an Intrusion Detection System for CyberSecurity in the IoT

The cyber ecosystem presents two interesting properties. Attackers and defenders are normally the same entity; a detailed knowledge of defensive strategies optimises an attack whereas a good defence, based on a layered structure also includes attacks. In addition, Artificial Intelligence (AI) provides new techniques and tools to both attackers and defenders such as Generative Adversarial Networks (GANs) based on Generators and Discriminators for impersonation or Deep Learning (DL) for exhaustive scanning. To address this dilemma, this article presents CyberAIBot: Artificial Intelligence in an Intrusion Detection System for CyberSecurity in the Internet of Things (IoT) aimed at Operational Technology (OT) and Information Technology (IT) network traffic. CyberAIBot is based on a Deep Learning management structure in a private or local edge cloud computing approach where AI makes decisions as close as possible to the source of data. CyberAIBot gradually detects, learns, and adapts to different cyber attacks. In detail, CyberAIBot uses Deep Learning (DL) technical clusters trained in specific attacks and a Deep Learning (DL) management cluster specialises in taking management decisions rather than technical evaluations. This management cluster supervises conflicting classifications from the deep learning technical clusters. CyberAIbot is trained against several datasets and its performance is evaluated between two classification algorithms, the Long Short-Term Memory (LSTM) networks and Support Vector Machines (SVMs). The SVM DL clusters learn faster (average 15x) although the LSTM DL clusters perform better (average 30%). The LSTM DL management cluster performs better than the SVM DL management cluster although it recognises fewer traffic types. The total number of data points analysed by CyberAIBot is 5.52E+08, equivalent to the distance between the planet Earth to the Moon in meters.

Sampling-Based Multi-Job Placement for Heterogeneous Deep Learning Clusters

Sampling-Based Multi-Job Placement for Heterogeneous Deep Learning Clusters

Renewable-Aware Frequency Scaling Approach for Energy-Efficient Deep Learning Clusters

Recently, renewable energy has emerged as an attractive means to reduce energy consumption costs for deep learning (DL) job processing in modern GPU-based clusters. In this paper, we propose a novel Renewable-Aware Frequency Scaling (RA-FS) approach for energy-efficient DL clusters. We have developed a real-time GPU core and memory frequency scaling method that finely tunes the training performance of DL jobs while maximizing renewable energy utilization. We introduce quantitative metrics: Deep Learning Job Requirement (DJR) and Deep Learning Job Completion per Slot (DJCS) to accurately evaluate the service quality of DL job processing. Additionally, we present a log-transformation technique to convert our non-convex optimization problem into a solvable one, ensuring the rigorous optimality of the derived solution. Through experiments involving deep neural network (DNN) model training jobs such as SqueezeNet, PreActResNet, and SEResNet on NVIDIA GPU devices like RTX3060, RTX3090, and RTX4090, we validate the superiority of our RA-FS approach. The experimental results show that our approach significantly improves performance requirement satisfaction by about 71% and renewable energy utilization by about 31% on average, compared to recent competitors.

On a Meta Learning-Based Scheduler for Deep Learning Clusters

On a Meta Learning-Based Scheduler for Deep Learning Clusters

Interference-aware opportunistic job placement for shared distributed deep learning clusters

Distributed deep learning frameworks facilitate large deep learning workloads. These frameworks support sharing one GPU device among multiple jobs to improve resource utilization. Modern deep learning training jobs consume a large amount of GPU memory. Despite that, sharing GPU memory among jobs is still possible because a training job has iterative steps that its memory usage fluctuates over time. However, resource sharing also introduces the risk of job performance degradation. Co-located jobs sharing a GPU device may suffer from different levels of interference, mainly caused by memory oversharing. How to improve resource utilization while maintaining good job performance is a novel challenge for job placement strategies. This paper studies the job placement problem. We propose an opportunistic memory sharing model to describe the time-varying job memory requirements. Based on this model, we introduce an Opportunistic Job Placement Problem (OJPP) for shared GPU clusters that seek job placement configurations using a minimum number of GPU devices and guarantee user-defined performance requirements at the same time. We propose a greedy algorithm and a heuristic algorithm with computational complexities of O(nlog⁡n) and O(n2log⁡n), respectively, to solve the problem. We also propose an online adjustment algorithm with the computational complexity of O(nlog⁡n) to perform updates to job placement configurations in runtime. A machine-learning-based interference prediction method is used to prepare accurate interference estimations. Extensive experiments are conducted on a GPU cluster to verify the correctness and effectiveness of our algorithms. Compared with standalone training jobs on dedicated clusters, the proposed approach reduces resource consumption by 46% in a shared cluster, while guaranteeing over 92.97% of the job performance, in terms of average job completion time.

SAT-171 - Deep learning cluster analysis reveals subtypes in response to antisense oligonucleotide therapy in chronic hepatitis B

SAT-171 - Deep learning cluster analysis reveals subtypes in response to antisense oligonucleotide therapy in chronic hepatitis B

Artificial Intelligence Meets Whole Slide Images: Deep Learning Model Shapes an Immune-Hot Tumor and Guides Precision Therapy in Bladder Cancer.

Background To construct and validate a deep learning cluster from whole slide images (WSI) for depicting the immunophenotypes and functional heterogeneity of the tumor microenvironment (TME) in patients with bladder cancer (BLCA) and to explore an artificial intelligence (AI) score to explore the underlying biological pathways in the developed WSI cluster. Methods In this study, the WSI cluster was constructed based on a deep learning procedure. Further rerecognition of TME features in pathological images was applied based on a neural network. Then, we integrated the TCGA cohort and several external testing cohorts to explore and validate this novel WSI cluster and a corresponding quantitative indicator, the AI score. Finally, correlations between the AI cluster (AI score) and classical BLCA molecular subtypes, immunophenotypes, functional heterogeneity, and potential therapeutic method in BLCA were assessed. Results The WSI cluster was identified associated with clinical survival (P < 0.001) and was proved as an independent predictor (P = 0.031), which could also predict the immunology and the clinical significance of BLCA. Rerecognition of pathological images established a robust 3-year survival prediction model (with an average classification accuracy of 86%, AUC of 0.95) for BLCA patients combining TME features and clinical features. In addition, an AI score was constructed to quantify the underlying logic of the WSI cluster (AUC = 0.838). Finally, we hypothesized that high AI score shapes an immune-hot TME in BLCA. Thus, treatment options including immune checkpoint blockade (ICB), chemotherapy, and ERBB therapy can be used for the treatment of BLCA patients in WSI cluster1 (high AI score subtype). Conclusions In general, we showed that deep learning can predict prognosis and may aid in the precision medicine for BLCA directly from H&E histology, which is more economical and efficient.

Cooperative Distributed GPU Power Capping for Deep Learning Clusters

The recent GPU-based clusters that handle deep learning (DL) tasks have the features of GPU device heterogeneity, a variety of deep neural network (DNN) models, and high computational complexity. Thus, the traditional power capping methods for CPU-based clusters or small-scale GPU devices cannot be applied to the GPU-based clusters handling DL tasks. This article develops a cooperative distributed GPU power capping (CD-GPC) system for GPU-based clusters, aiming to minimize the training completion time of invoked DL tasks without exceeding the limited power budget. Specifically, we first design the frequency scaling approach using the online model estimation based on the recursive least square method. This approach achieves the accurate tuning for DL task training time and power usage of GPU devices without needing offline profiling. Then, we formulate the proposed FS problem as a Lagrangian dual decomposition-based economic model predictive control problem for large-scale heterogeneous GPU clusters. We conduct both the NVIDIA GPU-based lab-scale real experiments and real job trace-based simulation experiments for performance evaluation. Experimental results validate that the proposed system improves the power capping accuracy to have a mean absolute error of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt; \!1\%$</tex-math></inline-formula> , and reduces the deadline violation ratio of invoked DL tasks by 21.5% compared with other recent counterparts.

Beamer: Stage-Aware Coflow Scheduling to Accelerate Hyper-Parameter Tuning in Deep Learning Clusters

Beamer: Stage-Aware Coflow Scheduling to Accelerate Hyper-Parameter Tuning in Deep Learning Clusters

Construction of an Industrial Structure Analysis and Evaluation Model for Oil and Gas Resource-Based Cities Based on Deep Learning Model and Cluster Analysis

In the present time of resource integration, the industrial transformation of oil and gas resource-based cities has become inevitable. Through the qualitative and quantitative analysis of the current situation of industrial structure, the overweight of industrial structure is pointed out and reasonable suggestions are put forward for the existing problems. This paper constructs an industrial structure analysis and evaluation model for oil and gas resource-based cities based on deep learning model and cluster analysis. The transformation of traditional industries is a long process, and at present the oil and petrochemical industry should still be an important support for Daqing’s economic development, so more efforts should be made to extend the service life of oil fields and to get more funds and time for successive industries. Fine exploration is being carried out, relying on continuous innovation in exploration theory, methods, technology and management, and re-measuring old exploratory wells in fields that have already been explored.

Cost Efficient GPU Cluster Management for Training and Inference of Deep Learning

Expanding the scale of GPU-based deep learning (DL) clusters would bring not only accelerated AI services but also significant energy consumption costs. In this paper, we propose a cost efficient deep learning job allocation (CE-DLA) approach minimizing the energy consumption cost for the DL cluster operation while guaranteeing the performance requirements of user requests. To do this, we first categorize the DL jobs into two classes: training jobs and inference jobs. Through the architecture-agnostic modeling, our CE-DLA approach is able to conduct the delicate mapping of heterogeneous DL jobs to GPU computing nodes. Second, we design the electricity price-aware DL job allocation so as to minimize the energy consumption cost of the cluster. We show that our approach efficiently avoids the peak-rate time slots of the GPU computing nodes by using the sophisticated mixed-integer nonlinear problem (MINLP) formulation. We additionally integrate the dynamic right-sizing (DRS) method with our CE-DLA approach, so as to minimize the energy consumption of idle nodes having no running job. In order to investigate the realistic behavior of our approach, we measure the actual output from the NVIDIA-based GPU devices with well-known deep neural network (DNN) models. Given the real trace data of the electricity price, we show that the CE-DLA approach outperforms the competitors in views of both the energy consumption cost and the performance for DL job processing.

A GPU Scheduling Framework to Accelerate Hyper-Parameter Optimization in Deep Learning Clusters

This paper proposes Hermes, a container-based preemptive GPU scheduling framework for accelerating hyper-parameter optimization in deep learning (DL) clusters. Hermes accelerates hyper-parameter optimization by time-sharing between DL jobs and prioritizing jobs with more promising hyper-parameter combinations. Hermes’s scheduling policy is grounded on the observation that good hyper-parameter combinations converge quickly in the early phases of training. By giving higher priority to fast-converging containers, Hermes’s GPU preemption mechanism can accelerate training. This enables users to find optimal hyper-parameters faster without losing the progress of a container. We have implemented Hermes over Kubernetes and compared its performance against existing scheduling frameworks. Experiments show that Hermes reduces the time for hyper-parameter optimization up to 4.04 times against previously proposed scheduling policies such as FIFO, round-robin (RR), and SLAQ, with minimal time-sharing overhead.

DL2: A Deep Learning-Driven Scheduler for Deep Learning Clusters

Efficient resource scheduling is essential for maximal utilization of expensive deep learning (DL) clusters. Existing cluster schedulers either are agnostic to machine learning (ML) workload characteristics, or use scheduling heuristics based on operators' understanding of particular ML framework and workload, which are less efficient or not general enough. In this article, we show that DL techniques can be adopted to design a generic and efficient scheduler. Specifically, we propose DL2, a DL-driven scheduler for DL clusters, targeting global training job expedition by dynamically resizing resources allocated to jobs. DL2 advocates a joint supervised learning and reinforcement learning approach: a neural network is warmed up via offline supervised learning based on job traces produced by the existing cluster scheduler; then the neural network is plugged into the live DL cluster, fine-tuned by reinforcement learning carried out throughout the training progress of the DL jobs, and used for deciding job resource allocation in an online fashion. We implement DL2 on Kubernetes and enable dynamic resource scaling in DL jobs on MXNet. Extensive evaluation shows that DL2 outperforms fairness scheduler (i.e., DRF) by 44.1 percent and expert heuristic scheduler (i.e., Optimus) by 17.5 percent in terms of average job completion time.

JPAS: Job-progress-aware flow scheduling for deep learning clusters

Deep learning (DL) is an increasingly important tool for large-scale data analytics and DL workloads are also common in today's production clusters due to the increasing number of deep-learning-driven services (e.g., online search and speech recognition). To handle ever-growing training datasets, it is common to conduct distributed DL (DDL) training to leverage multiple machines in parallel. Training DL models in parallel can incur significant bandwidth contention on shared clusters. As a result, the network is a well-known bottleneck for distributed training. Efficient network scheduling is essential for maximizing the performance of DL training. DL training is feedback-driven exploration (e.g., hyper-parameter tuning, model structure optimization), which requires multiple retrainings of deep learning models that differ in terms of their configuration. The information at the early stage of each retraining can facilitate the direct search for high-quality models. Thus, reducing the early-stage time can accelerate the exploration of DL training. In this paper, we propose JPAS, which is a flow scheduling system for DDL training jobs that aims at reducing the early-stage time. JPAS uses a simple greedy mechanism to periodically order all DDL jobs. Each host machine sets priorities for its flows using the corresponding job order and offloads the flow scheduling and rate allocation to the underlying priority-enabled network. We evaluate JPAS over a real testbed that is composed of 13 servers and a commodity switch. The evaluation results demonstrate that JPAS can reduce the time to reach 90% or 95% of the converged accuracy by up to 38%. Hence, JPAS can remarkably reduce the early-stage time and thus accelerate the search for high-quality models.

Genetic and deep learning clusters based on neural networks for management decision structures

Judgments are taken in a structured way; both human and business management decisions involve a hierarchical process that requires a level of compromise between risk, cost, reward, experience and knowledge. This article proposes a management decision structure that emulates the human brain approach based on genetic and deep learning cluster algorithms and the random neural network. Reinforcement learning takes quick and specific local decisions, deep learning clusters enables identity and memory, and deep learning management clusters make final strategic decisions. The presented genetic algorithm transmits the learned information to future generations in the network weights rather than the neurons. Because the subject’s information, a combination of memory, identity and decision data, is never lost but transmitted, the genetic algorithm provides immortality. The management decision structure has been applied and validated in a smart investment Fintech application: an intelligent banker that makes buy and sell asset decisions with an associated market and risk that entirely transmits itself to a future generation. Results are rewarding; the management decision structure with genetics and machine learning based on the random neural network algorithm that emulates the human brain and biology transmits information to future generations and learns autonomously, gradually and continuously while adapting to the environment.

Deep Learning Clusters in the Cognitive Packet Network

The Cognitive Packet Network (CPN) bases its routing decisions and flow control on the Random Neural Network (RNN) Reinforcement Learning algorithm; this paper proposes the addition of a Deep Learning (DL) Cluster management structure to the CPN for Quality of Service metrics (Delay Loss and Bandwidth), Cyber Security keys (User, Packet and Node) and Management decisions (QoS, Cyber and CEO). The RNN already models how neurons transmit information using positive and negative impulsive signals whereas the proposed additional Deep Learning structure emulates the way the brain learns and takes decisions; this paper presents a brain model as the combination of both learning algorithms, RNN and DL. The proposed model has been simulated under different network sizes and scenarios and it has been validated against the CPN itself without DL clusters. The simulation results are promising; the presented CPN with DL clusters as a mechanism to transmit, learn and make packet routing decisions is a step closer to emulate the way the brain transmits information, learns the environment and takes decisions.

The Random Neural Network with Deep Learning Clusters in Smart Search

This paper proposes a Neurocomputing application that reorders the Web results obtained from different Web Search Engines emulating the way our brain takes decisions. The proposed application is based on the Random Neural Network with Deep Learning Clusters that evaluates and adapts Web result relevance by associating independently each Deep Learning Cluster to a specific Web Search Engine. In addition, this paper presents a Deep Learning Cluster to perform as a Management Cluster that decides the final result relevance based on the inputs from each independent Deep Learning cluster. The performance of the proposed Management Cluster is evaluated when included as an additional layer to the Deep Learning Clusters. On average; the proposed Deep Learning cluster structure improves Smart Search performance.

Deep Learning Cluster Structures for Management Decisions: The Digital CEO.

This paper presents a Deep Learning (DL) Cluster Structure for Management Decisions that emulates the way the brain learns and makes choices by combining different learning algorithms. The proposed model is based on the Random Neural Network (RNN) Reinforcement Learning for fast local decisions and Deep Learning for long-term memory. The Deep Learning Cluster Structure has been applied in the Cognitive Packet Network (CPN) for routing decisions based on Quality of Service (QoS) metrics (Delay, Loss and Bandwidth) and Cyber Security keys (User, Packet and Node) which includes a layer of DL management clusters (QoS, Cyber and CEO) that take the final routing decision based on the inputs from the DL QoS clusters and RNN Reinforcement Learning algorithm. The model has been validated under different network sizes and scenarios. The simulation results are promising; the presented DL Cluster management structure as a mechanism to transmit, learn and make packet routing decisions is a step closer to emulate the way the brain transmits information, learns the environment and takes decisions.

Deep Learning, Deep Change? Mapping the Development of the Artificial Intelligence General Purpose Technology

General Purpose Technologies (GPTs) that can be applied in many industries are an important driver of economic growth and national and regional competitiveness. In spite of this, the geography of their development and diffusion has not received significant attention in the literature. We address this with an analysis of Deep Learning (DL), a core technique in Artificial Intelligence (AI) increasingly being recognized as the latest GPT. We identify DL papers in a novel dataset from ArXiv, a popular preprints website, and use CrunchBase, a technology business directory to measure industrial capabilities related to it. After showing that DL conforms with the definition of a GPT, having experienced rapid growth and diffusion into new fields where it has generated an impact, we describe changes in its geography. Our analysis shows China's rise in AI rankings and relative decline in several European countries. We also find that initial volatility in the geography of DL has been followed by consolidation, suggesting that the window of opportunity for new entrants might be closing down as new DL research hubs become dominant. Finally, we study the regional drivers of DL clustering. We find that competitive DL clusters tend to be based in regions combining research and industrial activities related to it. This could be because GPT developers and adopters located close to each other can collaborate and share knowledge more easily, thus overcoming coordination failures in GPT deployment. Our analysis also reveals a Chinese comparative advantage in DL after we control for other explanatory factors, perhaps underscoring the importance of access to data and supportive policies for the successful development of this complex, `omni-use' technology.

Fintech Model: The Random Neural Network with Genetic Algorithm

This paper proposes the Random Neural Network with a new learning algorithm based on the genome model. Following genetics, where information is transmitted in the combination of genes rather than the genes themselves, the proposed genetic model transmits information to future generations in the network weights rather than the neurons. The innovative genetic algorithm is implanted in a complex Deep Learning structure that emulates the human brain: Reinforcement Learning takes fast local current decisions, Deep Learning Clusters provide identity and memory, Deep Learning Management Clusters take final strategic decisions and finally Genetic Learning transmits the information learned to future generations. The presented algorithm and Deep Learning structure has been applied and validated in a Fintech Model; a Smart Investment application where an Intelligent Bankers are specialised in Buy and Sell decisions on several Assets, Markets and Risks. The obtained results are promising; the proposed model combines human brain and genetics with Deep Learning structure based on the Random Neural Network model where biology; similar as Artificial Intelligence is learning gradually and continuously while adapting to the environment.