Neural Network Inference Research Articles

Homomorphic inference of deep neural networks for zero-knowledge verification of nuclear warheads

Disarmament treaties have been the driving force towards reducing the large nuclear stockpile assembled during the Cold War. Further efforts are built around verification protocols capable of authenticating nuclear warheads while preventing the disclosure of confidential information. This type of problem falls under the scope of zero-knowledge protocols, which aim at multiple parties agreeing on a statement without conveying any information beyond the statement itself. A protocol capable of achieving all the authentication and security requirements is still not completely formulated. Here we propose a protocol that leverages the isotopic capabilities of NRF measurements and the classification abilities of neural networks. Two key elements guarantee the security of the protocol, the implementation of the template-based approach in the network’s architecture and the use of homomorphic inference. Our results demonstrate the potential of developing zero-knowledge protocols for the verification of nuclear warheads using Siamese networks on encrypted spectral data.

MJOA-MU: End-to-edge collaborative computation for DNN inference based on model uploading

As an emerging computing paradigm, edge computing can assist user equipments (UEs) in executing computation-intensive deep neural network (DNN) inference tasks, thereby satisfying the stringent QoS requirement and relieving the burden of UEs. Due to the customizability of DNN models and limited capacity of the edge server, it is more realistic to upload DNN models on demand during end-to-edge co-inference, instead of deploying all DNN models at the edge server in advance. Existing works adopt the serial model uploading manner that uploads subsequent DNN layers only after antecedent DNN layers finish execution, inevitably prolonging the DNN execution latency. To this end, we innovatively design a parallel-efficient model uploading mechanism that allows subsequent DNN layers to be uploaded simultaneously when executing antecedent DNN layers, so as to efficiently mitigate the performance drop caused by model uploading. On this basis, we propose a Multi-UE Joint Optimization Algorithm based on Model Uploading (MJOA-MU) to optimize DNN partitioning and resource allocation for heterogeneous UEs. Specifically, MJOA-MU includes a Pruned Binary Tree based DNN Partitioning (PBT-DP) sub-algorithm to efficiently make the near-optimal partitioning decision for chain and non-chain models based on the long-term influence between DNN layers, and an Asynchronous Resource Allocation (ARA) sub-algorithm to allocate computation and communication resources for UEs by quantifying the inner- and inter-association, so as to match with individual demand and resource budget. Extensive simulation results demonstrate that MJOA-MU outperforms the state-of-the-art in terms of the DNN execution latency, and specifically achieves up to 64.5% reduction.

Optical neural network via loose neuron array and functional learning

This research proposes a deep-learning paradigm, termed functional learning (FL), to physically train a loose neuron array, a group of non-handcrafted, non-differentiable, and loosely connected physical neurons whose connections and gradients are beyond explicit expression. The paradigm targets training non-differentiable hardware, and therefore solves many interdisciplinary challenges at once: the precise modeling and control of high-dimensional systems, the on-site calibration of multimodal hardware imperfectness, and the end-to-end training of non-differentiable and modeless physical neurons through implicit gradient propagation. It offers a methodology to build hardware without handcrafted design, strict fabrication, and precise assembling, thus forging paths for hardware design, chip manufacturing, physical neuron training, and system control. In addition, the functional learning paradigm is numerically and physically verified with an original light field neural network (LFNN). It realizes a programmable incoherent optical neural network, a well-known challenge that delivers light-speed, high-bandwidth, and power-efficient neural network inference via processing parallel visible light signals in the free space. As a promising supplement to existing power- and bandwidth-constrained digital neural networks, light field neural network has various potential applications: brain-inspired optical computation, high-bandwidth power-efficient neural network inference, and light-speed programmable lens/displays/detectors that operate in visible light.

PIMCA: A Programmable In-Memory Computing Accelerator for Energy-Efficient DNN Inference

This article presents a programmable in-memory computing accelerator (PIMCA) for low-precision (1–2 b) deep neural network (DNN) inference. The custom 10T1C bitcell in the in-memory computing (IMC) macro has four additional transistors and one capacitor to perform capacitive-coupling-based multiply and accumulation (MAC) in analog-mixed-signal (AMS) domain. A macro containing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$256\ttimes 128$</tex-math> </inline-formula> bitcells can simultaneously activate all the rows, and as a result, it can perform a matrix-vector multiplication (VMM) in one cycle. PIMCA integrates 108 of such IMC static random-access memory (SRAM) macros with the custom six-stage pipeline and the custom instruction set architecture (ISA) for instruction-level programmability. The results of IMC macros are fed to a single-instruction-multiple-data (SIMD) processor for other computations such as partial sum accumulation, max-pooling, activation functions, etc. To effectively use the IMC and SIMD datapath, we customize the ISA especially by adding hardware loop support, which reduces the program size by up to 73%. The accelerator is prototyped in a 28-nm technology, and integrates a total of 3.4-Mb IMC SRAM and 1.5-Mb off-the-shelf activation SRAM, demonstrating one of the largest IMC accelerators to date. It achieves the system-level energy efficiency of 437 TOPS/W and the peak throughput of 49 TOPS at the 42-MHz clock frequency and 1-V supply for the VGG9 and the ResNet-18 on the CIFAR-10 dataset.

Throughput Maximization of Delay-Aware DNN Inference in Edge Computing by Exploring DNN Model Partitioning and Inference Parallelism

Mobile Edge Computing (MEC) has emerged as a promising paradigm catering to overwhelming explosions of mobile applications, by offloading the compute-intensive tasks to an MEC network for processing. The surging of deep learning brings new vigor and vitality to shape the prospect of intelligent Internet of Things (IoT), and edge intelligence arises to provision real-time deep neural network (DNN) inference services for users. In this paper, we study a novel delay-aware DNN inference throughput maximization problem by accelerating each DNN inference through jointly exploring DNN partitioning and multi-thread parallelism. Specifically, we consider the problem under both offline and online request arrival settings: a set of DNN inference requests is given in advance, and a sequence of DNN inference requests arrives one by one without the knowledge of future arrivals, respectively. We first show that the defined problems are NP-hard. We then devise a novel constant approximation algorithm for the problem under the offline setting. We also propose an online algorithm with a provable competitive ratio for the problem under the online setting. We finally evaluate the performance of the proposed algorithms through experimental simulations. Experimental results demonstrate that the proposed algorithms are promising.

A Mapping Method Tolerating SAF and Variation for Memristor Crossbar Array Based Neural Network Inference on Edge Devices

There is an increasing demand for running neural network inference on edge devices. Memristor crossbar array (MCA) based accelerators can be used to accelerate neural networks on edge devices. However, reliability issues in memristors, such as stuck-at faults (SAF) and variations, lead to weight deviation of neural networks and therefore have a severe influence on inference accuracy. In this work, we focus on the reliability issues in memristors for edge devices. We formulate the reliability problem as a 0–1 programming problem, based on the analysis of sum weight variation (SWV) . In order to solve the problem, we simplify the problem with an approximation - different columns have the same weights, based on our observation of the weight distribution. Then we propose an effective mapping method to solve the simplified problem. We evaluate our proposed method with two neural network applications on two datasets. The experimental results on the classification application show that our proposed method can recover 95% accuracy considering SAF defects and can increase by up to 60% accuracy with variation σ =0.4. The results of the neural rendering application show that our proposed method can prevent render quality reduction.

Deep Learning Accelerated Design of Mechanically Efficient Architected Materials

Lattice structures are known to have high performance-to-weight ratios because of their highly efficient material distribution in a given volume. However, their inherently large void fraction leads to low mechanical properties compared to the base material, high anisotropy, and brittleness. Most works to date have focused on modifying the spatial arrangement of beam elements to overcome these limitations, but only simple beam geometries are adopted due to the infinitely large design space associated with probing and varying beam shapes. Herein, we present an approach to enhance the elastic modulus, strength, and toughness of lattice structures with minimal tradeoffs by optimizing the shape of beam elements for a suite of lattice structures. A generative deep learning-based approach is employed, which leverages the fast inference of neural networks to accelerate the optimization process. Our optimized lattice structures possess superior stiffness (+59%), strength (+49%), toughness (+106%), and isotropy (+645%) compared to benchmark lattices consisting of cylindrical beams. We fabricate our lattice designs using additive manufacturing to validate the optimization approach; experimental and simulation results show good agreement. Remarkable improvement in mechanical properties is shown to be the effect of distributed stress fields and deformation modes subject to beam shape and lattice type.

Gradient distribution-aware INT8 training for neural networks

Recently, low bit-width quantization (e.g., INT8) has been commonly used in deep neural network inference acceleration, but fewer researchers have focused on low-precision training quantization techniques. Considering that the backward propagation in deep neural network training is more computationally intensive and has a heavier energy overhead than the inference process, the quantization of backward propagation is of great interest for the training of very large-scale neural networks as well as for low-power devices with online training requirements. However, the shape specificity and continuous variability of the gradient distribution make gradient quantization difficult, and many studies propose various complex quantization methods for the gradient to reduce training accuracy loss. In this paper, we propose two innovative techniques mainly for INT8 quantization training, including the Data-aware Dynamic Segmentation Quantization scheme to quantize various special gradient distributions and the Update Direction Periodic Search strategy to achieve lower quantization errors. Then, we build a distribution-aware INT8 quantization training framework based on these two methods and conduct experiments on various models and tasks. Experimental results show that our proposed INT8 quantization training method achieves a negligible loss in final training accuracy compared to the full-precision floating-point counterpart on different models, including ResNet, MobileNetV2, VGG, AlexNet, and LSTM. By replacing floating-point computing with 8-bit integer computing for network training, this INT8 quantization training framework provides the possibility of deploying online training directly on low-power devices in the future.

Probability propagation for faster and efficient point cloud segmentation using a neural network

Neural networks (NN) have shown promising performance in point cloud segmentation (PCS). However, the measured points are too numerous to be used as model input at once. It results in a long inference time and high computational cost due to iterative sampling and inference. This study proposes Probability Propagation (PP) as a stochastic upsampling method. PP propagates the predicted probability of a sampled part of a point cloud into the other unpredicted points by considering proximity. By replacing the iterative inference of NN with PP, large point clouds can be dealt with quickly and efficiently. We investigated the effectiveness of PP using the ShapeNet benchmark on various settings: sampling methods (random, farthest point, and Poisson disk sampling) with sampling ratios (5%, 10%, 20%, 39%, and 78%) for NN and the stochastic mapping conditions (uniform, linear, cosine, Gaussian, and exponential distributions) for PP. Using NN with PP achieved higher performance and faster inference speed than when using NN alone. For the farthest point sampling method of 5% sampling ratio, NN+PP improved the instance mIoU by 2.457%p with 102 times faster speed compared to that when using NN alone. The result indicates that PP can significantly contribute to the improvement of performance and efficiency in PCS when used in edge AI systems.

B-LNN: Inference-time linear model for secure neural network inference

Machine Learning as a Service (MLaaS) provides clients with well-trained neural networks for predicting private data. Conventional prediction processes of MLaaS require clients to send sensitive inputs to the server, or proprietary models must be stored on the client-side device. The former reveals client privacy, while the latter harms the interests of model providers. Existing works on privacy-preserving MLaaS introduce cryptographic primitives to allow two parties to perform neural network inference without revealing either party's data. However, nonlinear activation functions bring high computational overhead and response delays to the inference process of these schemes.In this paper, we analyze the mechanism by which activation functions enhance model expressivity, and design an activation function S-cos that is friendly to secure neural network inference. Our proposed S-cos can be re-parameterized into a linear layer during the inference phase. Further, we propose an inference-time linear model called Beyond Linear Neural Network (B-LNN) equipped with S-cos, which exhibits promising performance on several benchmark datasets.

Efficient grouping approach for fault tolerant weight mapping in memristive crossbar array

The ability of resistive memory (ReRAM) to naturally conduct vector–matrix multiplication (VMM), which is the primary operation carried out during the training and inference of neural networks, has caught the interest of researchers. The memristor crossbar is one of the desirable architectures to perform VMM because it offers various benefits over other memory technologies, including in-memory computing, low power, and high density. Direct downloading and chip-on-the-loop approaches are typically used to train ReRAM-based neural networks. In these methods, all weight computations are carried out by a host machine, and the computed weights are downloaded in the crossbar. It has been seen that the network does not deliver the same precision as promised by the host system once the weights have been downloaded. This is because crossbars contain a significant number of faulty memristors and suffer from cell resistance variations because of immature manufacturing technologies. As a result, a cell may not be able to take the exact weight values that the host system generates, and may lead to incorrect inferences. Existing techniques for fault-tolerant mapping either involve network retraining or employ a graph-matching strategy that comes with hardware, power, and latency overheads. In this paper, we propose a mapping method to tolerate the effect of defective memristors. In order to lessen the impact of faulty memristors, the mapping is done in a way that allows network weights to cover up faulty memristors. Further, this work prioritizes the different faults based on the frequency of occurrence. The mapping efficiency is found to increase significantly with low power, area and latency overheads in the proposed approach. Experimental analyses show considerable improvement as compared to state-of-the-art works.

LHDNN: Maintaining High Precision and Low Latency Inference of Deep Neural Networks on Encrypted Data

The advancement of deep neural networks (DNNs) has prompted many cloud service providers to offer deep learning as a service (DLaaS) to users across various application domains. However, in current DLaaS prediction systems, users’ data are at risk of leakage. Homomorphic encryption allows operations to be performed on ciphertext without decryption, which can be applied to DLaaS to ensure users’ data privacy. However, mainstream homomorphic encryption schemes only support homomorphic addition and multiplication, and do not support the ReLU activation function commonly used in the activation layers of DNNs. Previous work used approximate polynomials to replace the ReLU activation function, but the DNNs they implemented either had low inference accuracy or high inference latency. In order to achieve low inference latency of DNNs on encrypted data while ensuring inference accuracy, we propose a low-degree Hermite deep neural network framework (called LHDNN), which uses a set of low-degree trainable Hermite polynomials (called LotHps) as activation layers of DNNs. Additionally, LHDNN integrates a novel weight initialization and regularization module into the LotHps activation layer, which makes the training process of DNNs more stable and gives a stronger generalization ability. Additionally, to further improve the model accuracy, we propose a variable-weighted difference training (VDT) strategy that uses ReLU-based models to guide the training of LotHps-based models. Extensive experiments on multiple benchmark datasets validate the superiority of LHDNN in terms of inference speed and accuracy on encrypted data.

Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml

Recurrent neural networks have been shown to be effective architectures for many tasks in high energy physics, and thus have been widely adopted. Their use in low-latency environments has, however, been limited as a result of the difficulties of implementing recurrent architectures on field-programmable gate arrays (FPGAs). In this paper we present an implementation of two types of recurrent neural network layers—long short-term memory and gated recurrent unit—within the hls4ml framework. We demonstrate that our implementation is capable of producing effective designs for both small and large models, and can be customized to meet specific design requirements for inference latencies and FPGA resources. We show the performance and synthesized designs for multiple neural networks, many of which are trained specifically for jet identification tasks at the CERN Large Hadron Collider.

CNNParted: An open source framework for efficient Convolutional Neural Network inference partitioning in embedded systems

Applications such as autonomous driving or assistive robotics heavily rely on the usage of Deep Neural Networks. In particular, Convolutional Neural Networks (CNNs) provide precise and reliable results in image processing tasks like camera-based object detection or semantic segmentation. However, to achieve even better results, CNNs are becoming more and more complex. Deploying these networks in distributed embedded systems thereby imposes new challenges, due to additional constraints regarding performance and energy consumption in the near-sensor compute platforms, i.e. the sensor nodes. Processing all data in the central node, however, is disadvantageous since raw data of camera consumes large bandwidth and running CNN inference of multiple tasks requires certain performance. Moreover, sending raw data over the interconnect is not advisable for privacy reasons. Hence, offloading CNN workload to the sensor nodes in the system can lead to reduced traffic on the link and a higher level of data security.However, due to the limited hardware-resources on the sensor nodes, partitioning CNNs has to be done carefully to meet overall latency requirements and energy constraints. Therefore, we present CNNParted, an open-source framework for efficient, hardware-aware CNN inference partitioning targeting embedded AI applications. It automatically searches for potential partitioning points in the CNN to find a beneficial workload distribution between sensor nodes and a central edge node. Thereby, CNNParted not only analyzes the CNN architecture but also takes hardware components, such as dedicated hardware accelerators and memories, into consideration to evaluate inference partitioning regarding latency and energy consumption.Exemplary, we apply CNNParted to three commonly used feed forward CNNs in embedded systems. Thereby, the framework first searches for several potential partitioning points and then evaluates the latter regarding inference latency and energy consumption. Based on the results, beneficial partitioning points can be identified depending on the system constraints. Using the framework, we are able to find and evaluate 10 potential partitioning points for FCN ResNet-50, 13 partitioning points for GoogLeNet, and 8 partitioning points for SqueezeNet V1.1 within 520 s, 330 s, and 140 s, respectively, on an AMD EPYC 7702P running 8 concurrent threads. For GoogLeNet, we determine two partitioning points that provide a good trade-off between required bandwidth, latency and energy consumption. We also provide insights into further interesting findings that can be derived from the evaluation results.

Intrinsic resistive switching in ultrathin SiOx memristors for neuromorphic inference accelerators

Two-dimensional (2D) materials are promising memristive materials owing to their outstanding physical properties, electrical tunability and fast switching capability. However, 2D layered materials generally require sophisticated deposition techniques at the expense of limited large-scale homogeneity and high growth temperatures. Furthermore, the variability and reliability of the 2D materials based memristor should be further improved. Here, we report an intrinsic memristor based on ultrathin 2D-like nonlayered amorphous SiOx which is derived from a native oxidation process at ambient temperature. Such 2D-like memristors demonstrate low switching variability (3.7 %), nanosecond switching speed (15 ns), good endurance (>106 cycles) and high retention (>103 s at 85 °C). We verified the resistive switching mechanism via in-depth X-ray photoemission spectroscopy (XPS) analysis at different resistance states and confirmed the oxygen vacancies movement under the electric field between the SnOx reservoir layer and the SiOx layer. Moreover, Simulation results for an MNIST image classifier based on the ultrathin SiOx memristor show a high recognition accuracy of ∼ 98 %, manifesting its potential for the practical implementation of nonlayered 2D-like materials based neural network inference accelerator.

Automated Exploration and Implementation of Distributed CNN Inference at the Edge

For model inference of convolutional neural networks (CNNs), we nowadays witness a shift from the Cloud to the Edge. Unfortunately, deploying and inferring large, compute- and memory-intensive CNNs on Internet of Things devices at the Edge is challenging as they typically have limited resources. One approach to address this challenge is to leverage all available resources across multiple edge devices to execute a large CNN by properly partitioning it and running each CNN partition on a separate edge device. However, there currently does not exist a design and programming framework that takes a trained CNN model as input and subsequently allows for efficiently exploring and automatically implementing a range of different CNN partitions on multiple edge devices to facilitate distributed CNN inference. Therefore, in this article, we propose a novel framework that automates the splitting of a CNN model into a set of submodels as well as the code generation needed for the distributed and collaborative execution of these submodels on multiple, possibly heterogeneous, edge devices, while supporting the exploitation of parallelism among and within the edge devices. In addition, since the number of different CNN mapping possibilities on multiple edge devices is vast, our framework also features a multistage and hierarchical design space exploration methodology to efficiently search for (near-)optimal distributed CNN inference implementations. Our experimental results demonstrate that our work allows for rapidly finding and realizing distributed CNN inference implementations with reduced energy consumption and memory usage per edge device, and under certain conditions, with improved system throughput as well.

A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nm

The energy efficiency of deep neural network (DNN) inference can be improved with custom accelerators. DNN inference accelerators often employ specialized hardware techniques to improve energy efficiency, but many of these techniques result in catastrophic accuracy loss on transformer-based DNNs, which have become ubiquitous for natural language processing (NLP) tasks. This article presents a DNN accelerator designed for efficient execution of transformers. The proposed accelerator implements per-vector scaled quantization (VSQ), which employs an independent scale factor for each 64-element vector to enable the use of 4-bit arithmetic with little accuracy loss and low energy overhead. Using a multilevel dataflow to maximize reuse, the 5-nm prototype achieves 95.6 tera-operations per second per Watt (TOPS/W) at 0.46 V on a 4-bit benchmarking layer with VSQ. At a nominal voltage of 0.67 V, the accelerator achieves 1734 inferences/s/W (38.7 TOPS/W) with only 0.7% accuracy loss on BERT-Base and 4714 inferences/s/W (38.6 TOPS/W) with 0.15% accuracy loss on ResNet-50 by using quantization-aware fine-tuning to recover accuracy, demonstrating a practical accelerator design for energy-efficient DNN inference.

Study of Intelligent Fire Identification System Based on Back Propagation Neural Network

In order to detect and identify fire accidents accurately and efficiently, an intelligent fire identification system based on neural network algorithm is designed, which can overcome the shortcomings of single information, complex wiring, poor adaptability, etc. The characteristic extraction of sensors is adopted in the information layer to solve the problems in multi-sensor fusion. The fire data are transmitted to the main controller through LoRa wireless module and fused by back propagation neural network, which is self-learning and adaptive. The output of neural network and fuzzy inference with other factors are used for decision criteria to improve the identification accuracy. The common combustibles and various interference sources are selected for fire tests. The result shows that the detection accuracy is up to 100% and the false alarm rate is lower than 0.1%, meanwhile, the system has the advantages of fast response and high detection efficiency.

Model-driven Cluster Resource Management for AI Workloads in Edge Clouds

Since emerging edge applications such as Internet of Things (IoT) analytics and augmented reality have tight latency constraints, hardware AI accelerators have been recently proposed to speed up deep neural network (DNN) inference run by these applications. Resource-constrained edge servers and accelerators tend to be multiplexed across multiple IoT applications, introducing the potential for performance interference between latency-sensitive workloads. In this article, we design analytic models to capture the performance of DNN inference workloads on shared edge accelerators, such as GPU and edgeTPU, under different multiplexing and concurrency behaviors. After validating our models using extensive experiments, we use them to design various cluster resource management algorithms to intelligently manage multiple applications on edge accelerators while respecting their latency constraints. We implement a prototype of our system in Kubernetes and show that our system can host 2.3× more DNN applications in heterogeneous multi-tenant edge clusters with no latency violations when compared to traditional knapsack hosting algorithms.

Prediction Analysis of Surface Roughness of Aluminum Al6061 in End Milling CNC Machine Using Soft Computing Techniques

Computer numerically controlled (CNC) milling has been one of the most commonly used manufacturing processes for the performance of multiple operations, from tiny integrated circuits to heavy-duty mining machine gearboxes. It is a well-known machining process that offers close tolerances and repeated operations. However, the choice of machining parameters to achieve a desired part’s surface roughness (SR) remains a challenge. In the present study, artificial neural network (ANN) and adaptive network-based fuzzy inference system (ANFIS) approaches have been used to predict and monitor the surface roughness of aluminum Al6061 machined blocks. Furthermore, both models have been hybridized with genetic algorithm (GA) and particle swarm optimization (PSO) to investigate the potential enhancement in the prediction performance of the hybrid approach. The results show that factors such as the population size, the acceleration values, the choice of membership functions, and the number of neurons and layers significantly influence the prediction performance of the proposed models. Through a parametric analysis, this study demonstrates how the configuration of the models could affect the prediction performance. While exhibiting the impact of models’ hyperparameter combination on the prediction ability, this study provides insight into the development of suitable prediction models and the potential of soft computing techniques to predict the surface roughness of aluminum Al6061 blocks on CNC machines.