Learning Inference Research Articles

FPGA-based Deep Learning Inference Accelerators: Where Are We Standing?

Recently, artificial intelligence applications have become part of almost all emerging technologies around us. Neural networks, in particular, have shown significant advantages and have been widely adopted over other approaches in machine learning. In this context, high processing power is deemed a fundamental challenge and a persistent requirement. Recent solutions facing such a challenge deploy hardware platforms to provide high computing performance for neural networks and deep learning algorithms. This direction is also rapidly taking over the market. Here, FPGAs occupy the middle ground regarding flexibility, reconfigurability, and efficiency compared to general-purpose CPUs, GPUs, on one side, and manufactured ASICs on the other. FPGA-based accelerators exploit the features of FPGAs to increase the computing performance for specific algorithms and algorithm features. Filling a gap, we provide holistic benchmarking criteria and optimization techniques that work across several classes of deep learning implementations. This article summarizes the current state of deep learning hardware acceleration: More than 120 FPGA-based neural network accelerator designs are presented and evaluated based on a matrix of performance and acceleration criteria, and corresponding optimization techniques are presented and discussed. In addition, the evaluation criteria and optimization techniques are demonstrated by benchmarking ResNet-2 and LSTM-based accelerators.

Invertible Neural Network for Inference Pipeline Anomaly Detection

This study combines research in machine learning and system engineering practices to conceptualize a paradigm-enhancing trustworthiness of a machine learning inference pipeline. We explore the topic of reversibility in deep neural networks and introduce its anomaly detection capabilities to build a framework of integrity verification checkpoints across the inference pipeline of a deployed model. We leverage previous findings and principles regarding several types of autoencoders, deep generative maximumlikelihood training and invertibility of neural networks to propose an improved network architecture for anomaly detection. We hypothesize and experimentally confirm that an Invertible Neural Network (INN) trained as a convolutional autoencoder is a superior alternative naturally suited to solve that task. This remarkable INN’s ability to reconstruct data from its compressed representation and to solve inverse problems is then generalized and applied in the field of Trustworthy AI to achieve integrity verification of an inference pipeline through the concept of an INN-based Trusted Neural Network (TNN) nodes placed around the mission critical parts of the system, as well as the end-to-end outcome verification. This work aspires to enhance robustness and reliability of applications employing artificial intelligence, which are playing increasingly noticeable role in highly consequential decision-making processes across many industries and problem domains. INNs are invertible by construction and tractably trained simultaneously in both directions. This feature has untapped potential to improve the explainability of machine learning pipelines in support of their trustworthiness and is a topic of our current studies.

Toward Carbon-Neutral Edge Computing: Greening Edge AI by Harnessing Spot and Future Carbon Markets

Provisioning dynamic machine learning (ML) inference as a service for artificial intelligence (AI) applications of edge devices faces many challenges, including the trade-off among accuracy loss, carbon emission, and unknown future costs. Besides, many governments are launching carbon emission rights (CER) for operators to reduce carbon emissions further to reverse climate change. Facing these challenges, to achieve carbon-aware ML task offloading under limited carbon emission rights thus to achieve green edge AI, we establish a joint ML task offloading and CER purchasing problem, intending to minimize the accuracy loss under the long-term time-averaged cost budget of purchasing the required CER. However, considering the uncertainty of the resource prices, the CER purchasing prices, the carbon intensity of sites, and ML tasks’ arrivals, it is hard to decide the optimal policy online over a long-running period time. To overcome this difficulty, we leverage the two-timescale Lyapunov optimization technique, of which the T-slot drift-plus-penalty methodology inspires us to propose an online algorithm that purchases CER in multiple timescales (on-preserved in carbon future market and on-demanded in the carbon spot market) and makes decisions about where to offload ML tasks. Considering the NP-hardness of the T-slot problems, we further propose the resource-restricted randomized dependent rounding algorithm to help to gain the near-optimal solution with no help of any future information. Our theoretical analysis and extensive simulation results driven by the real carbon intensity trace show the superior performance of the proposed algorithms.

Retrospective quantification of clinical abdominal DCE-MRI using pharmacokinetics-informed deep learning: a proof-of-concept study.

Dynamic contrast-enhanced (DCE) MRI has important clinical value for early detection, accurate staging, and therapeutic monitoring of cancers. However, conventional multi-phasic abdominal DCE-MRI has limited temporal resolution and provides qualitative or semi-quantitative assessments of tissue vascularity. In this study, the feasibility of retrospectively quantifying multi-phasic abdominal DCE-MRI by using pharmacokinetics-informed deep learning to improve temporal resolution was investigated. Forty-five subjects consisting of healthy controls, pancreatic ductal adenocarcinoma (PDAC), and chronic pancreatitis (CP) were imaged with a 2-s temporal-resolution quantitative DCE sequence, from which 30-s temporal-resolution multi-phasic DCE-MRI was synthesized based on clinical protocol. A pharmacokinetics-informed neural network was trained to improve the temporal resolution of the multi-phasic DCE before the quantification of pharmacokinetic parameters. Through ten-fold cross-validation, the agreement between pharmacokinetic parameters estimated from synthesized multi-phasic DCE after deep learning inference was assessed against reference parameters from the corresponding quantitative DCE-MRI images. The ability of the deep learning estimated parameters to differentiate abnormal from normal tissues was assessed as well. The pharmacokinetic parameters estimated after deep learning have a high level of agreement with the reference values. In the cross-validation, all three pharmacokinetic parameters (transfer constant , fractional extravascular extracellular volume , and rate constant ) achieved intraclass correlation coefficient and R2 between 0.84-0.94, and low coefficients of variation (10.1%, 12.3%, and 5.6%, respectively) relative to the reference values. Significant differences were found between healthy pancreas, PDAC tumor and non-tumor, and CP pancreas. Retrospective quantification (RoQ) of clinical multi-phasic DCE-MRI is possible by deep learning. This technique has the potential to derive quantitative pharmacokinetic parameters from clinical multi-phasic DCE data for a more objective and precise assessment of cancer.

A Coalition Formation Approach for Privacy and Energy-Aware Split Deep Learning Inference in Edge Camera Network

Recently, an increasing amount of application tasks have depended on deep learning (DL) inference models on Internet of Things (IoT) based camera networks. However, it is challenging to perform the inference of such resource-hungry DL models on the computationally limited IoT system. Compared to cloud computing, edge computing deploys resources near the end-users to reduce the transmission delay and retains the raw data on the trusted servers to mitigate privacy concerns. Since resources at the edge are limited, management like user association decisions, edge resources allocation, and device configuration with DL model parameter selection becomes essential. This work proposes a coalition formation game-based algorithm to solve the association problem between IoT-based cameras and the edge nodes. Our goal is to maximize the social welfare that consists of multi-view detection enhancement, the privacy retained preference, and the power savings from the cameras. Besides, we adopt the concept of split-ML to provide more flexibility for networking and computing resources allocation at the edge. The final coalition structure is proved to converge and maintain stability. The simulation results show that each design knob, including association decisions made by coalition formation, DNN layer-level partition, and multi-view detection, is essential under different scenario settings.

SAMBA: Sparsity Aware In-Memory Computing Based Machine Learning Accelerator

Machine Learning (ML) inference is typically dominated by highly data-intensive Matrix Vector Multiplication (MVM) computations that may be constrained by the memory bottleneck due to massive data movement between processor and memory units. Although analog in-memory computing (IMC) ML accelerators have been proposed to execute MVM with high efficiency, the latency and energy of such analog computing systems can be dominated by the large latency and energy costs from analog-to-digital converters (ADC). Leveraging sparsity in ML workloads, reconfigurable ADCs can improve the MVM energy and latency by reducing the required ADC bit precision. However, such improvement in latency can be hindered by non-uniform sparsity of the weight matrices mapped into hardware. Moreover, data movement between MVM processing cores may become another factor that delays the overall system-level performance. To address these issues, we propose SAMBA, Sparsity Aware IMC Based Machine Learning Accelerator. First, we propose load balancing at the mapping of weight matrices into physical crossbars. The goal of load balancing is to accommodate reconfigurable ADCs and address the possible delay of MVM processing caused by non-uniform sparsity in the weight matrices. Second, we propose optimizations in arranging and scheduling the tiled MVM hardware to minimize the overhead of data movement across multiple processing cores. Our evaluations show that the proposed load balancing technique can achieve performance improvement by eliminating the non-uniformity in the sparsity of mapped matrices. Moreover, we demonstrate that data movement optimizations can further improve both performance and energy efficiency regardless of sparsity condition. With the combination of load balancing and data movement optimization in conjunction with reconfigurable ADCs, our proposed approach provides up to 2.38x speed-up and 1.54x energy efficiency over state-of-art analog IMC based ML accelerators for ImageNet datasets on Resnet-50 architecture.

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators

Analog in-memory computing—a promising approach for energy-efficient acceleration of deep learning workloads—computes matrix-vector multiplications but only approximately, due to nonidealities that often are non-deterministic or nonlinear. This can adversely impact the achievable inference accuracy. Here, we develop an hardware-aware retraining approach to systematically examine the accuracy of analog in-memory computing across multiple network topologies, and investigate sensitivity and robustness to a broad set of nonidealities. By introducing a realistic crossbar model, we improve significantly on earlier retraining approaches. We show that many larger-scale deep neural networks—including convnets, recurrent networks, and transformers—can in fact be successfully retrained to show iso-accuracy with the floating point implementation. Our results further suggest that nonidealities that add noise to the inputs or outputs, not the weights, have the largest impact on accuracy, and that recurrent networks are particularly robust to all nonidealities.

Deep Learning from Phylogenies for Diversification Analyses.

Birth-death (BD) models are widely used in combination with species phylogenies to study past diversification dynamics. Current inference approaches typically rely on likelihood-based methods. These methods are not generalizable, as a new likelihood formula must be established each time a new model is proposed; for some models, such a formula is not even tractable. Deep learning can bring solutions in such situations, as deep neural networks can be trained to learn the relation between simulations and parameter values as a regression problem. In this paper, we adapt a recently developed deep learning method from pathogen phylodynamics to the case of diversification inference, and we extend its applicability to the case of the inference of state-dependent diversification models from phylogenies associated with trait data. We demonstrate the accuracy and time efficiency of the approach for the time-constant homogeneous BD model and the Binary-State Speciation and Extinction model. Finally, we illustrate the use of the proposed inference machinery by reanalyzing a phylogeny of primates and their associated ecological role as seed dispersers. Deep learning inference provides at least the same accuracy as likelihood-based inference while being faster by several orders of magnitude, offering a promising new inference approach for the deployment of future models in the field.

Mapping HDX-MS Data to Protein Conformations through Training Ensemble-Based Models.

An original approach that adopts machine learning inference to predict protein structural information using hydrogen-deuterium exchange mass spectrometry (HDX-MS) is described. The method exploits an in-house optimization program that increases the resolution of HDX-MS data from peptides to amino acids. A system is trained using Gradient Tree Boosting as a type of machine learning ensemble technique to assign a protein secondary structure. Using limited training data we generate a discriminative model that uses optimized HDX-MS data to predict protein secondary structure with an accuracy of 75%. This research could form the basis for new methods exploiting artificial intelligence to model protein conformations by HDX-MS.

Efficient Hardware Acceleration Techniques for Deep Learning on Edge Devices: A Comprehensive Performance Analysis

Implementing deep learning models on edge devices presents significant challenges due to the limited computing power, memory limitations, and processing power of these devices As deep learning models become more complex, cf ensuring proper execution on an edge platform is critical for real-time implementation Hardware -Addresses these challenges by exploring ways to accelerate. The problem lies in the resource-hungry nature of today’s deep learning models, which are typically designed for cloud environments with high computing capacity, making them unsuitable for edge environments with restricted resources. The main objective of this review is to analyze and compare various hardware acceleration strategies, such as graphics processing units (GPUs), tensor processing units (TPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs); . . . . (pruning, quantization, and knowledge storage), memory management, and data flow optimization to improve the performance and energy efficiency of deep learning models on edge devices The results of this comprehensive performance analysis suggests that hardware accelerators can significantly improve throughput and reduce latency while maintaining acceptable levels of power consumption and accuracy. In addition to techniques such as quantization and pruning that are seen to reduce computational load and memory footprint, enabling more efficient deep learning inference on edge platforms, the study highlights trade-offs between speed, consumption power efficiency and model accuracy between for each hardware accelerator are emphasized. The findings suggest that by choosing the right hardware and applying the right optimization techniques, edge devices may be able to optimize deep learning models, meeting the requirements of real-time AI applications in resource-constrained environments handle the role there.

Deep learning inference time guarantee for aggregated images at the edge of the network

Deep learning inference time guarantee for aggregated images at the edge of the network

Hierarchical Model Parallelism for Optimizing Inference on Many-core Processor via Decoupled 3D-CNN Structure

The tremendous success of convolutional neural network (CNN) has made it ubiquitous in many fields of human endeavor. Many applications such as biomedical analysis and scientific data analysis involve analyzing volumetric data. This spawns huge demand for 3D-CNN. Although accelerators such as GPU may provide higher throughput on deep learning applications, they may not be available in all scenarios. CPU, especially many-core CPU with non-uniform memory access (NUMA) architecture, remains an attractive choice for deep learning inference in many scenarios. In this article, we propose a distributed inference solution for 3D-CNN that targets on the emerging ARM many-core CPU platform. A hierarchical partition approach is claimed to accelerate 3D-CNN inference by exploiting characteristics of memory and cache on ARM many-core CPU. Based on the hierarchical model partition approach, other optimization techniques such as NUMA-aware thread scheduling and optimization of 3D-img2row convolution are designed to exploit the potential of ARM many-core CPU for 3D-CNN. We evaluate our proposed inference solution with several classic 3D-CNNs: C3D, 3D-resnet34, 3D-resnet50, 3D-vgg11, and P3D. Our experimental results show that our solution can boost the performance of the 3D-CNN inference, and achieve much better scalability, with a negligible fluctuation in accuracy. When employing our 3D-CNN inference solution on ACL libraries, it can outperform naive ACL implementations by 11× to 50× on ARM many-core processor. When employing our 3D-CNN inference solution on NCNN libraries, it can outperform the naive NCNN implementations by 5.2× to 14.2× on ARM many-core processor.

Taming Continuous Posteriors for Latent Variational Dialogue Policies

Utilizing amortized variational inference for latent-action reinforcement learning (RL) has been shown to be an effective approach in Task-oriented Dialogue (ToD) systems for optimizing dialogue success.Until now, categorical posteriors have been argued to be one of the main drivers of performance. In this work we revisit Gaussian variational posteriors for latent-action RL and show that they can yield even better performance than categoricals. We achieve this by introducing an improved variational inference objective for learning continuous representations without auxiliary learning objectives, which streamlines the training procedure. Moreover, we propose ways to regularize the latent dialogue policy, which helps to retain good response coherence. Using continuous latent representations our model achieves state of the art dialogue success rate on the MultiWOZ benchmark, and also compares well to categorical latent methods in response coherence.

Catalysing assistive solutions by deploying light-weight deep learning model on edge devices

ABSTRACT Nowadays, real-time object detection, which is a crucial task, is being performed through image processing and deep learning techniques. As there are several high-performance computing edge devices available, selecting the best-fit device for a particular problem is a tough task and keeping in mind the cost, performance, and weight of the device in mind. One faces several challenges while performing this task in real-time such as a lack of resources in terms of power and mobility. We have provided an insight into the computation power of devices in terms of Frames per Second (FPS) by deploying object detection models on them. This paper will provide insight into selecting the appropriate combination of device and object detection models for real-time applications. Raspberry Pi 3 (RPi3), Raspberry Pi 4 (RPi4), Intel Neural Compute Stick 2 (NCS2), and Nvidia Jetson NANO are popular devices with high computation power used for real-time applications. The memory constraints of devices along with the deployment of different You Only Look Once (YOLO) and Single-Shot Detector (SSD) are the two object detection models that have been explained in this paper. A deep learning inference optimiser, TensorRT, has been used in NANO to achieve high throughput in the performance of object detection. The precision, recall, and F1 score achieved on deploying each tested model have been presented. After observing the devices during experimentation, RPi4+NCS2 showed the best execution with the blend of factors i.e. speed, portability, and user-friendliness.

Optimum: Runtime optimization for multiple mixed model deployment deep learning inference

GPUs used in data centers to perform deep learning inference tasks are underutilized. Previous systems tended to deploy a single model on a GPU to ensure that inference tasks met throughput and latency requirements. The rapid increase in one GPU’s resources, as well as the emergence of scenarios such as small models and small batches, have exacerbated the issue of low GPU utilization. In this case, a mixed model deployment-based solution can significantly improve GPU utilization while also providing greater flexibility to the inference system’s upper layer. The selection of model combinations and optimization strategies in mixed model deployment, however, remain unresolved issues. This paper proposes Optimum, the first model-combination planning and runtime optimization framework for mixed model deployment. Facing enormous search spaces, Optimum uses performance prediction for model combination selection with low search overhead. The predictor is based on a multilayer perceptron. Its input features are the profiling results of the model engine, and the output is the performance degradation. The runtime optimization strategies allow Optimum to perform performance optimization and fine-grained tradeoff. The Optimum prototype is based on CUDA multi-stream and TensorRT. The test results show that we have a flat 10.3% performance improvement over mainstream single-model deployments. We have a performance improvement of up to 7.09% over the state-of-the-art with an order of magnitude reduction in search overhead.

Upward Pass Semantics in Arithmetic Circuit Inference

An Arithmetic Circuit (AC) is a deep learning probabilistic model that is compiled by eliminating every variable in a given Bayesian Network (BN). We introduce a special case of AC, called a p-AC, in which every node is a 1, marginal, or conditional of the joint distribution defined by the given BN. This is accomplished by using wait-sets to restrict the elimination ordering used. We show both theoretical and practical advantages of p-ACs over ACs, including that there is no increase in network size. Lastly, we observe and analyze an interesting graphical relationship between semantics in deep learning inference and causal analysis.

Research on personalized Japanese language teaching model based on big data technology

Abstract In order to better improve students’ Japanese language performance and help them progress, this paper proposes an analysis of personalized Japanese language teaching in a big data technology environment. We build a model of learner characteristics and use collaborative filtering techniques to push learning information from learners with the same or similar interest and preference characteristics. Adjust teaching strategies based on visual user information in the information panel. Build a personalized teaching model based on big data, extract and visualize the data of each student’s learning behavior, find out the gap between the teaching objectives and the preset ones, and record the students’ learning methods. Based on data to dynamically update the teaching content, assign the most appropriate learning tasks, determine the distance of teaching objectives, and improve students’ learning ability and learning effectiveness. Through inductive learning inference on the training dataset, the probabilities of all subsets are calculated, the relevant data subset intervals are merged, and the business sensitivity index is introduced to measure the whole source, which makes the improved decision tree gradually reduce the uncertainty of the division and decrease the weight degree of the attributes, which makes the personalized Japanese teaching better. The analysis results showed that the improved decision tree model reached 71% accuracy in 80 cases and stayed above 73% afterwards, and the ratio of students’ mastered knowledge to the number of knowledges learned was 0.4. Personalized Japanese language teaching under this method led to a significant improvement in students’ performance.

Research and implementation of visual question and answer system based on deep learning

Abstract With the development and improvement of deep learning technology, its application and practice in modal data (image, speech, and text) has been achieved tremendously. In this paper, based on the neural network modality class model in deep learning, we analyze its adaptation to the visual question and answer system, propose a visual question and answer model based on the gated attention mechanism, and construct a question, and answer prediction mechanism adapted to the recurrent neural network transfer model. To address the problem of low accuracy of the model on complex problems, the inference network module is built using visual inference so that the model can extract complex problem features to improve the inference capability of the model. By predicting answers through semantic information about text and visual elements in images, correlations across modalities, and inference, advances in natural language processing and computational vision have led to improved answer accuracy in deep learning-based visual quiz models. Multiple sets of experiments show that models with deep learning inference capabilities answer complex questions with significantly higher accuracy than other existing methods.

On Sequential Bayesian Inference for Continual Learning.

Sequential Bayesian inference can be used for continual learning to prevent catastrophic forgetting of past tasks and provide an informative prior when learning new tasks. We revisit sequential Bayesian inference and assess whether using the previous task's posterior as a prior for a new task can prevent catastrophic forgetting in Bayesian neural networks. Our first contribution is to perform sequential Bayesian inference using Hamiltonian Monte Carlo. We propagate the posterior as a prior for new tasks by approximating the posterior via fitting a density estimator on Hamiltonian Monte Carlo samples. We find that this approach fails to prevent catastrophic forgetting, demonstrating the difficulty in performing sequential Bayesian inference in neural networks. From there, we study simple analytical examples of sequential Bayesian inference and CL and highlight the issue of model misspecification, which can lead to sub-optimal continual learning performance despite exact inference. Furthermore, we discuss how task data imbalances can cause forgetting. From these limitations, we argue that we need probabilistic models of the continual learning generative process rather than relying on sequential Bayesian inference over Bayesian neural network weights. Our final contribution is to propose a simple baseline called Prototypical Bayesian Continual Learning, which is competitive with the best performing Bayesian continual learning methods on class incremental continual learning computer vision benchmarks.

HARBIC: Human activity recognition using bi-stream convolutional neural network with dual joint time–frequency representation

Human activity recognition (HAR) based on wearable devices has progressively advanced in context-aware computing like healthcare, smart homes, and industry 4.0. Since machine learning and Internet-of-Things have been in the spotlight, many deep learning-based approaches have been proposed, achieving very good performance using inertial data. In this study, we explore different joint time–frequency representations of the sensor data, in conjunction with a Convolutional Neural Network (ConvNet) to implement activity classification. We also consider the concurrent use of two different data representations in a cooperative bi-stream ConvNet configuration. Scaling of joint time–frequency representation is explored to reduce the ConvNet model size, and hence the computational complexity of the proposed approach. The classification performance and energy efficiency of the proposed method are evaluated on a real-world public HAR dataset. The performance of pairwise combinations and also, the single-stream of the two prominent representations in the joint time–frequency domain are compared to that of the state-of-the-art. The proposed method is implemented on Google’s Edge TPU platform to investigate the trade-off between the energy footprint and classification performance, which is critical in IoT applications where machine learning inference needs to be performed on resource-constrained edge devices. The highest overall classification accuracy and F1 score achieved by our method was 98.39% and 96.86%, respectively, which represents a significant improvement over the state-of-the-art.