Deep Learning Inference Research Articles

A portable, self-contained neuroprosthetic hand with deep learning-based finger control

Objective. Deep learning-based neural decoders have emerged as the prominent approach to enable dexterous and intuitive control of neuroprosthetic hands. Yet few studies have materialized the use of deep learning in clinical settings due to its high computational requirements. Approach. Recent advancements of edge computing devices bring the potential to alleviate this problem. Here we present the implementation of a neuroprosthetic hand with embedded deep learning-based control. The neural decoder is designed based on the recurrent neural network architecture and deployed on the NVIDIA Jetson Nano—a compacted yet powerful edge computing platform for deep learning inference. This enables the implementation of the neuroprosthetic hand as a portable and self-contained unit with real-time control of individual finger movements. Main results. A pilot study with a transradial amputee is conducted to evaluate the proposed system using peripheral nerve signals acquired from implanted intrafascicular microelectrodes. The preliminary experiment results show the system’s capabilities of providing robust, high-accuracy (95%–99%) and low-latency (50–120 ms) control of individual finger movements in various laboratory and real-world environments. Conclusion. This work is a technological demonstration of modern edge computing platforms to enable the effective use of deep learning-based neural decoders for neuroprosthesis control as an autonomous system. Significance. The proposed system helps pioneer the deployment of deep neural networks in clinical applications underlying a new class of wearable biomedical devices with embedded artificial intelligence.Clinical trial registration: DExterous Hand Control Through Fascicular Targeting (DEFT). Identifier: NCT02994160.

DeepLINK: Deep learning inference using knockoffs with applications to genomics

We propose a deep learning-based knockoffs inference framework, DeepLINK, that guarantees the false discovery rate (FDR) control in high-dimensional settings. DeepLINK is applicable to a broad class of covariate distributions described by the possibly nonlinear latent factor models. It consists of two major parts: an autoencoder network for the knockoff variable construction and a multilayer perceptron network for feature selection with the FDR control. The empirical performance of DeepLINK is investigated through extensive simulation studies, where it is shown to achieve FDR control in feature selection with both high selection power and high prediction accuracy. We also apply DeepLINK to three real data applications to demonstrate its practical utility.

Validating deep learning inference during chest X-ray classification for COVID-19 screening

The new coronavirus unleashed a worldwide pandemic in early 2020, and a fatality rate several times that of the flu. As the number of infections soared, and capabilities for testing lagged behind, chest X-ray (CXR) imaging became more relevant in the early diagnosis and treatment planning for patients with suspected or confirmed COVID-19 infection. In a few weeks, proposed new methods for lung screening using deep learning rapidly appeared, while quality assurance discussions lagged behind. This paper proposes a set of protocols to validate deep learning algorithms, including our ROI Hide-and-Seek protocol, which emphasizes or hides key regions of interest from CXR data. Our protocol allows assessing the classification performance for anomaly detection and its correlation to radiological signatures, an important issue overlooked in several deep learning approaches proposed so far. By running a set of systematic tests over CXR representations using public image datasets, we demonstrate the weaknesses of current techniques and offer perspectives on the advantages and limitations of automated radiography analysis when using heterogeneous data sources.

Fast3DS: A real-time full-convolutional malicious domain name detection system

Detecting malicious domain names generated by domain generation algorithms is critical for defending the network against sophisticated attacks. In the past decade, deep-learning-based detection schemes have proven to be the most effective. However, each of these schemes requires sufficient computation time, which makes it difficult for real-time online detection. In this paper, we propose a real-time malicious domain name detection system which is called fast3DS. The proposed system contains a lightweight full-convolutional detection model, as well as a detection pipeline that contains multiple efficient schemes for high-speed data acquisition, filtering, and inference. The proposed detection model uses a parallel depthwise convolutional architecture to replace the standard convolution layer, and a lightweight global average pooling connection architecture is proposed to replace the fully connected layer, which can effectively reduce the parameters and computation time. To compensate for the decrease in accuracy due to model lightweighting, a lightweight attention mechanism is proposed to improve the accuracy of model detection. The proposed detection pipeline employs some industry-leading network traffic processing architecture and deep learning inference acceleration architecture, which can fully utilize the CPU for processing and computing. The experimental results denote that the proposed detection scheme can achieve accuracy close to the state-of-the-art with significantly fewer parameters, and the system can substantially improve the processing capability compared to the conventional detection system.

A Runtime Reconfigurable Design of Compute-in-Memory–Based Hardware Accelerator for Deep Learning Inference

Compute-in-memory (CIM) is an attractive solution to address the “memory wall” challenges for the extensive computation in deep learning hardware accelerators. For custom ASIC design, a specific chip instance is restricted to a specific network during runtime. However, the development cycle of the hardware is normally far behind the emergence of new algorithms. Although some of the reported CIM-based architectures can adapt to different deep neural network (DNN) models, few details about the dataflow or control were disclosed to enable such an assumption. Instruction set architecture (ISA) could support high flexibility, but its complexity would be an obstacle to efficiency. In this article, a runtime reconfigurable design methodology of CIM-based accelerators is proposed to support a class of convolutional neural networks running on one prefabricated chip instance with ASIC-like efficiency. First, several design aspects are investigated: (1) the reconfigurable weight mapping method; (2) the input side of data transmission, mainly about the weight reloading; and (3) the output side of data processing, mainly about the reconfigurable accumulation. Then, a system-level performance benchmark is performed for the inference of different DNN models, such as VGG-8 on a CIFAR-10 dataset and AlexNet GoogLeNet, ResNet-18, and DenseNet-121 on an ImageNet dataset to measure the trade-offs between runtime reconfigurability, chip area, memory utilization, throughput, and energy efficiency.

Independent verification of brachytherapy treatment plan by using deep learning inference modeling

This study aims to develop a deep learning-based strategy for treatment plan check and verification of high-dose rate (HDR) brachytherapy. A deep neural network was trained to verify the dwell positions and times for a given input brachytherapy isodose distribution. In our modeling, each dwell position is represented by a Gaussian heatmap located in the vicinity of the dwell positions. A deep inception network based architecture was established to learn the mapping between CT, dose distribution and the heatmap volume. The dwell position coordinates were obtained from the predicted heatmap volume by finding the location of the Gaussian peak using non-maximum suppression. An encoder network was employed to predict dwell time by using the same input. 110 HDR brachytherapy cervical patients were used to train the proposed network. Additional 10 patients were employed to evaluate the accuracy of the proposed method through comparing the dwell position coordinates and dwell times with the results from a treatment planning system. The proposed deep learning-based dwell positions and times verification method achieved excellent predictive performance. For the tested patients, the deviation of the deep learning predicted dwell position coordinates was around one pixel from the planned positions (on average, a pixel is ∼0.5 mm), and the relative deviations of the predicted dwell times were within 2%. A deep learning-based plan check and verification method was established for brachytherapy. Our study showed that the model is capable of predicting the dwell positions and times reliably and promises to provide an efficient and accurate tool for independent verification of HDR brachytherapy treatment plan.

An industrial-grade solution for agricultural image classification tasks

With artificial intelligence to promote the rapid development of precision agriculture, the management and detection of agricultural resources through computer vision and deep learning is particularly important. Content-Based image classification has produced extensive research, however relative research is still only at the theoretical level. We therefore put forward a high-performance, low-cost, powerful applicability and very feasible scheme to solve many problems which image classification is facing today in agricultural automation. Firstly, we use a fine-tuning strategy to train models to better performance in classification tasks. Secondly, we utilize neural network pruning techniques to reduce neural network size and computational cost, and subsequently retrain models by knowledge distillation to minimize pruned model performance loss. Thirdly, converting trained models to an available ONNX format, thus simplifying the process from theory to practice. And finally, we deploy and inference models in a high-performance deep learning inference platform with C++, that is, NCNN. We conducted experiments on three representative agricultural image classification datasets, PlantVillage, DeepWeeds and Flowers102. The results demonstrate the effectiveness of our scheme.

New universal sustainability metrics to assess edge intelligence

The single recent focus on deep learning accuracy ignores economic, and environmental cost. Progress towards Green AI is hindered by lack of universal metrics that equally reward accuracy and cost and can help to improve all deep learning algorithms and platforms. We define recognition and training efficiency as new universal metrics to assess deep learning sustainability and compare them to similar, less universal metrics. They are based on energy consumption measurements, on deep learning inference, on recognition gradients, and on number of classes and thus universally balance accuracy, complexity and energy consumption. Well-designed edge accelerators improve recognition and training efficiencies compared to cloud CPUs and GPUs due to reduced communication overhead. Cradle to grave sustainability of edge intelligence models and platforms is assessed with novel deep learning lifecycle efficiency and life cycle recognition efficiency metrics that include the number of times models are used. Artificial and natural intelligence efficiencies are compared leading to insights on deep learning scalability.

Technical perspective: A chiplet prototype system for deep learning inference

Technical perspective: A chiplet prototype system for deep learning inference

Simba

Package-level integration using multi-chip-modules (MCMs) is a promising approach for building large-scale systems. Compared to a large monolithic die, an MCM combines many smaller chiplets into a larger system, substantially reducing fabrication and design costs. Current MCMs typically only contain a handful of coarse-grained large chiplets due to the high area, performance, and energy overheads associated with inter-chiplet communication. This work investigates and quantifies the costs and benefits of using MCMs with finegrained chiplets for deep learning inference, an application domain with large compute and on-chip storage requirements. To evaluate the approach, we architected, implemented, fabricated, and tested Simba, a 36-chiplet prototype MCM system for deep-learning inference. Each chiplet achieves 4 TOPS peak performance, and the 36-chiplet MCM package achieves up to 128 TOPS and up to 6.1 TOPS/W. The MCM is configurable to support a flexible mapping of DNN layers to the distributed compute and storage units. To mitigate inter-chiplet communication overheads, we introduce three tiling optimizations that improve data locality. These optimizations achieve up to 16% speedup compared to the baseline layer mapping. Our evaluation shows that Simba can process 1988 images/s running ResNet-50 with a batch size of one, delivering an inference latency of 0.50 ms.

Memory-efficient deep learning inference with incremental weight loading and data layout reorganization on edge systems

Pattern recognition applications such as face recognition and agricultural product detection have drawn a rapid interest on Cyber–Physical–Social-Systems (CPSS). These CPSS applications rely on the deep neural networks (DNN) to conduct the image classification. However, traditional DNN inference models in the cloud could suffer from network delay fluctuations and privacy leakage problems. In this regard, current real-time CPSS applications are preferred to be deployed on edge-end embedded devices. Constrained by the computing power and memory limitations of edge devices, improving the memory management efficacy is the key to improving the quality of service for model inference. First, this study explored the incremental loading strategy of model weights for the model inference. Second, the memory space at runtime is optimized through data layout reorganization from the spatial dimension. In particular, the proposed schemes are orthogonal to existing models. Experimental results demonstrate that the proposed approach reduced the memory consumption by 61.05% without additional inference time overhead.

Interference-aware execution framework with Co-scheML on GPU clusters

Recently, improving the overall resource utilization through efficient scheduling of applications on graphic processing unit (GPU) clusters has been a concern. Traditional cluster-orchestration platforms providing GPUs exclusively for applications constrain high resource utilization. Co-execution of GPU applications is suggested to utilize limited resources. However, the co-execution of GPU applications without considering their diverse characteristics can lead to their unpredictable performances owing to interference resulting from contention and unbalanced usage of resources among applications. This paper proposes an interference-aware execution framework with Co-scheML for various GPU applications such as high performance computing (HPC), deep learning (DL) training, and DL inference. Various resource-usage characteristics of GPU applications are analyzed and profiled to identify various degrees of their application interference. As interference prediction is challenging owing to the complexity of GPU systems, an interference model is generated by applying defined GPU metrics to machine learning (ML) models. A Co-scheML scheduler deploys applications to minimize the interference using the predicted interference from the constructed model. Experimental results of our framework demonstrated that the resource utilization improved by 24%, the average job completion time (JCT) improved by 23%, and the makespan shortened by 22% on average, compared to baseline schedulers.

Empowering Adaptive Early-Exit Inference with Latency Awareness

With the capability of trading accuracy for latency on-the-fly, the technique of adaptive early-exit inference has emerged as a promising line of research to accelerate the deep learning inference. However, studies in this line of research commonly use a group of thresholds to control the accuracy-latency trade-off, where a thorough and general methodology on how to determine these thresholds has not been conducted yet, especially with regard to the common requirements of average inference latency. To address this issue and enable latency-aware adaptive early-exit inference, in the present paper, we approximately formulate the threshold determination problem of finding the accuracy-maximum threshold setting that meets a given average latency requirement, and then propose a threshold determination method to tackle our formulated non-convex problem. Theoretically, we prove that, for certain parameter settings, our method finds an approximate stationary point of the formulated problem. Empirically, on top of various models across multiple datasets (CIFAR-10, CIFAR-100, ImageNet and two time-series datasets), we show that our method can well handle the average latency requirements, and consistently finds good threshold settings in negligible time.

Edge deep learning for neural implants: a case study of seizure detection and prediction

Objective. Implanted devices providing real-time neural activity classification and control are increasingly used to treat neurological disorders, such as epilepsy and Parkinson’s disease. Classification performance is critical to identifying brain states appropriate for the therapeutic action (e.g. neural stimulation). However, advanced algorithms that have shown promise in offline studies, in particular deep learning (DL) methods, have not been deployed on resource-restrained neural implants. Here, we designed and optimized three DL models or edge deployment and evaluated their inference performance in a case study of seizure detection. Approach. A deep neural network (DNN), a convolutional neural network (CNN), and a long short-term memory (LSTM) network were designed and trained with TensorFlow to classify ictal, preictal, and interictal phases from the CHB-MIT scalp EEG database. A sliding window based weighted majority voting algorithm was developed to detect seizure events based on each DL model’s classification results. After iterative model compression and coefficient quantization, the algorithms were deployed on a general-purpose, off-the-shelf microcontroller for real-time testing. Inference sensitivity, false positive rate (FPR), execution time, memory size, and power consumption were quantified. Main results. For seizure event detection, the sensitivity and FPR for the DNN, CNN, and LSTM models were 87.36%/0.169 h−1, 96.70%/0.102 h−1, and 97.61%/0.071 h−1, respectively. Predicting seizures for early warnings was also feasible. The LSTM model achieved the best overall performance at the expense of the highest power. The DNN model achieved the shortest execution time. The CNN model showed advantages in balanced performance and power with minimum memory requirement. The implemented model compression and quantization achieved a significant saving of power and memory with an accuracy degradation of less than 0.5%. Significance. Inference with embedded DL models achieved performance comparable to many prior implementations that had no time or computational resource limitations. Generic microcontrollers can provide the required memory and computational resources, while model designs can be migrated to application-specific integrated circuits for further optimization and power saving. The results suggest that edge DL inference is a feasible option for future neural implants to improve classification performance and therapeutic outcomes.

GPU coprocessors as a service for deep learning inference in high energy physics

In the next decade, the demands for computing in large scientific experiments are expected to grow tremendously. During the same time period, CPU performance increases will be limited. At the CERN Large Hadron Collider (LHC), these two issues will confront one another as the collider is upgraded for high luminosity running. Alternative processors such as graphics processing units (GPUs) can resolve this confrontation provided that algorithms can be sufficiently accelerated. In many cases, algorithmic speedups are found to be largest through the adoption of deep learning algorithms. We present a comprehensive exploration of the use of GPU-based hardware acceleration for deep learning inference within the data reconstruction workflow of high energy physics. We present several realistic examples and discuss a strategy for the seamless integration of coprocessors so that the LHC can maintain, if not exceed, its current performance throughout its running.

Efficient Pipelined Execution of CNNs Based on In-Memory Computing and Graph Homomorphism Verification

In-memory computing is an emerging computing paradigm enabling deep-learning inference at significantly higher energy-efficiency and reduced latency. The essential idea is mapping the synaptic weights of each layer to one or more in-memory computing (IMC) cores. During inference, these cores perform the associated matrix-vector multiplications in place with O(1) time complexity, obviating the need to move the synaptic weights to additional processing units. Moreover, this architecture enables the execution of these networks in a highly pipelined fashion. However, a key challenge is designing an efficient communication fabric for the IMC cores. In this work, we present one such communication fabric based on a graph topology that is well-suited for the widely successful convolutional neural networks (CNNs). We show that this communication fabric facilitates the pipelined execution of all state-of-the-art CNNs by proving the existence of a homomorphism between the graph representations of these networks and that corresponding to the proposed communication fabric. We then present a quantitative comparison with established communication topologies and show that our proposed topology achieves the lowest bandwidth requirements per communication channel. Finally, we present one hardware implementation and show a concrete example of mapping ResNet-32 onto an IMC core array interconnected via the proposed communication fabric.

Extensive studies of the neutron star equation of state from the deep learning inference with the observational data augmentation

We discuss deep learning inference for the neutron star equation of state (EoS) using the real observational data of the mass and the radius. We make a quantitative comparison between the conventional polynomial regression and the neural network approach for the EoS parametrization. For our deep learning method to incorporate uncertainties in observation, we augment the training data with noise fluctuations corresponding to observational uncertainties. Deduced EoSs can accommodate a weak first-order phase transition, and we make a histogram for likely first-order regions. We also find that our observational data augmentation has a byproduct to tame the overfitting behavior. To check the performance improved by the data augmentation, we set up a toy model as the simplest inference problem to recover a double-peaked function and monitor the validation loss. We conclude that the data augmentation could be a useful technique to evade the overfitting without tuning the neural network architecture such as inserting the dropout.

Hardware-Enabled Efficient Data Processing With Tensor-Train Decomposition

In recent years, tensor computation has become a promising tool for solving big data analysis, machine learning, medical image and EDA problems. To ease the memory and computation intensity of tensor processing, decomposition techniques, especially Tensor-train Decomposition(TTD), are widely adopted to compress the extremely high-dimensional tensor data. Despite TTD’s potential to break the curse of dimensionality, researchers have not yet leveraged its full computational potential, mainly because of two reasons:(1) Executing TTD itself is time-and energy-consuming due to the singular value decomposition(SVD) operation inside each of TTD’s iteration; (2) Additional software/hardware optimizations are often required to process the obtained TT-format data in certain applications such as deep learning inference. In this paper, we address these challenges with two approaches. Firstly, we propose an algorithm-hardware co-design with customized architecture namely TTD Engine to accelerate TTD. We use MRI image compression as a demo application to illustrate the efficacy of the proposed accelerator. Secondly, we present a case study demonstrating the benefit of TT-format data processing and the efficacy of using TTD Engine. In the case study, we use TT approach to realize convolution operation, which is difficult and nontrivial for TT-format data. Experimental results show that, TTD Engine achieves, on average, 14.9×∼36.9× speedup over CPU implementations and 4.1×∼9.9× speedup compared to the GPU baseline. The energy efficiency is also improved by at least 14.4× and 5.4× over CPU and GPU, respectively. Moreover, our hardware-enabled TT-format data processing further leads to more efficient implementations of complicated operations and applications.

Spatial population genomics of a recent mosquito invasion.

Population genomic approaches can characterize dispersal across a single generation through to many generations in the past, bridging the gap between individual movement and intergenerational gene flow. These approaches are particularly useful when investigating dispersal in recently altered systems, where they provide a way of inferring long-distance dispersal between newly established populations and their interactions with existing populations. Human-mediated biological invasions represent such altered systems which can be investigated with appropriate study designs and analyses. Here we apply temporally restricted sampling and a range of population genomic approaches to investigate dispersal in a 2004 invasion of Aedes albopictus (the Asian tiger mosquito) in the Torres Strait Islands (TSI) of Australia. We sampled mosquitoes from 13 TSI villages simultaneously and genotyped 373 mosquitoes at genome-wide single nucleotide polymorphisms (SNPs): 331 from the TSI, 36 from Papua New Guinea (PNG) and four incursive mosquitoes detected in uninvaded regions. Within villages, spatial genetic structure varied substantially but overall displayed isolation by distance and a neighbourhood size of 232-577. Close kin dyads revealed recent movement between islands 31-203km apart, and deep learning inferences showed incursive Ae. albopictus had travelled to uninvaded regions from both adjacent and nonadjacent islands. Private alleles and a co-ancestry matrix indicated direct gene flow from PNG into nearby islands. Outlier analyses also detected four linked alleles introgressed from PNG, with the alleles surrounding 12 resistance-associated cytochrome P450 genes. By treating dispersal as both an intergenerational process and a set of discrete events, we describe a highly interconnected invasive system.

Automated Design Space Exploration for Optimized Deployment of DNN on Arm Cortex-A CPUs

The spread of deep learning on embedded devices has prompted the development of numerous methods to optimise the deployment of deep neural networks (DNN). Works have mainly focused on: i) efficient DNN architectures, ii) network optimisation techniques such as pruning and quantisation, iii) optimised algorithms to speed up the execution of the most computational intensive layers and, iv) dedicated hardware to accelerate the data flow and computation. However, there is a lack of research on cross-level optimisation as the space of approaches becomes too large to test and obtain a globally optimised solution. Thus, leading to suboptimal deployment in terms of latency, accuracy, and memory. In this work, we first detail and analyse the methods to improve the deployment of DNNs across the different levels of software optimisation. Building on this knowledge, we present an automated exploration framework to ease the deployment of DNNs. The framework relies on a Reinforcement Learning search that, combined with a deep learning inference framework, automatically explores the design space and learns an optimised solution that speeds up the performance and reduces the memory on embedded CPU platforms. Thus, we present a set of results for state-of-the-art DNNs on a range of Arm Cortex-A CPU platforms achieving up to 4x improvement in performance and over 2x reduction in memory with negligible loss in accuracy with respect to the BLAS floating-point implementation.