High-performance Inference Research Articles

SMaLL: Software for Rapidly Instantiating Machine Learning Libraries

Interest in deploying deep neural network (DNN) inference on edge devices has resulted in an explosion of the number and types of hardware platforms that machine learning (ML) libraries must support. High-level programming interfaces, such as TensorFlow, can be readily ported across different devices; however, maintaining performance when porting the low-level implementation is more nuanced. High-performance inference implementations require an effective mapping of the high-level interface to the target hardware platform. Commonly, this mapping may use optimizing compilers to generate code at compile time or high-performance vendor libraries that have been specialized to the target platform. Both approaches rely on expert knowledge across levels to produce an efficient mapping. This makes supporting new architectures difficult and time-consuming. In this work, we present a DNN library framework, SMaLL, that is easily extensible to new architectures. The framework uses a unified loop structure and shared, cache-friendly data format across all intermediate layers, eliminating the time and memory overheads incurred by data transformation between layers. Each layer is implemented by specifying its dimensions and a kernel , the key computing operation of that layer. The unified loop structure and kernel abstraction allows the reuse of code across layers and computing platforms. New architectures only require a few hundred lines in the kernel to be redesigned. To show the benefits of our approach, we have developed software that supports a range of layer types and computing platforms; this software is easily extensible for rapidly instantiating high-performance DNN libraries. An evaluation of the portability of our framework is shown by instantiating end-to-end networks from the MLPerf:tiny benchmark suite on five ARM platforms and one x86 platform (an AMD Zen 2). We also show that the end-to-end performance is comparable to or better than ML frameworks such as TensorFlow, TVM, and LibTorch.

A 218 GOPS neural network accelerator based on a novel cost-efficient surrogate gradient scheme for pattern classification

The accuracy and hardware efficiency of a neural system depends critically on the choice of an activation function. Rectified linear unit (ReLU) is a contemporary activation function that yields high accuracy and allows the construction of efficient neural chips, but it results in a lot of dead neurons, especially at the output layer. This problem is more pronounced in case of multichannel, multiclass classification. This is due to the fact that ReLU cancels out negative values altogether, as a result of which the corresponding values cannot be successfully backpropagated. This phenomenon is referred to as the dying ReLU problem. In this article, we present a novel ‘surrogate gradient’ learning scheme in order to solve gradient vanishing and the dying ReLU problems. To the best of our knowledge, this is the first learning scheme that enables the use of ReLU for all network layers while solving the Dying ReLU problem. We also present a high-performance inference engine that uses ReLU-based actuators for all the network layers in order to achieve high hardware efficiency. The design is excellent for online learning as well, since the derivative of the activation is equal to a constant and can be implemented using low-complexity components. The proposed technique significantly outperforms various contemporary schemes for the CIFAR-10 dataset, and can successfully yield about 98.39% accuracy on MNIST dataset while using less than 159k synapses. Moreover, the proposed hardware implementation is able to perform about 218 giga operation per second (GOPS) while consuming only about 3.95 slice registers and 25.89 slice look-up tables per synapse on a low-end Virtex 6 FPGA. The system is able to operate at a clock frequency of 93.2 MHz.

TransFusionNet: Semantic and Spatial Features Fusion Framework for Liver Tumor and Vessel Segmentation Under JetsonTX2.

Liver cancer is one of the most common malignant diseases worldwide. Segmentation and reconstruction of liver tumors and vessels in CT images can provide convenience for physicians in preoperative planning and surgical intervention. In this paper, we introduced a TransFusionNet framework, which consists of a semantic feature extraction module, a local spatial feature extraction module, an edge feature extraction module, and a multi-scale feature fusion module to achieve fine-grained segmentation of liver tumors and vessels. In addition, we applied the transfer learning approach to pre-train using public datasets and then fine-tune the model to further improve the fitting effect. Furthermore, we proposed an intelligent quantization scheme to compress the model weights and achieved high performance inference on JetsonTX2. The TransFusionNet framework achieved mean IoU of 0.854 in vessel segmentation task, and achieved mean IoU of 0.927 in liver tumor segmentation task. When profiling the Computational Performance of the quantized inference, our quantized model achieved 4TFLOPs on Node with NVIDIA RTX3090 and 132GFLOPs on JetsonTX2. This unprecedented segmentation effect solves the accuracy and performance bottleneck of automated segmentation to a certain extent.

A compositional approach to creating architecture frameworks with an application to distributed AI systems

Artificial intelligence (AI) in its various forms finds more and more its way into complex distributed systems. For instance, it is used locally, as part of a sensor system, on the edge for low-latency high-performance inference, or in the cloud, e.g. for data mining. Modern complex systems, such as connected vehicles, are often part of an Internet of Things (IoT). This poses additional architectural challenges. To manage complexity, architectures are described with architecture frameworks, which are composed of a number of architectural views connected through correspondence rules. Despite some attempts, the definition of a mathematical foundation for architecture frameworks that are suitable for the development of distributed AI systems still requires investigation and study.In this paper, we propose to extend the state of the art on architecture framework by providing a mathematical model for system architectures, which is scalable and supports co-evolution of different aspects for example of an AI system. Based on Design Science Research, this study starts by identifying the challenges with architectural frameworks in a use case of distributed AI systems. Then, we derive from the identified challenges four rules, and we formulate them by exploiting concepts from category theory. We show how compositional thinking can provide rules for the creation and management of architectural frameworks for complex systems, for example distributed systems with AI. The aim of the paper is not to provide viewpoints or architecture models specific to AI systems, but instead to provide guidelines based on a mathematical formulation on how a consistent framework can be built up with existing, or newly created, viewpoints. To put in practice and test the approach, the identified and formulated rules are applied to derive an architectural framework for the EU Horizon 2020 project “Very efficient deep learning in the IoT” (VEDLIoT) in the form of a case study.

High performance inference of gait recognition models on embedded systems

Edge computing is gaining importance in the realm of Deep Learning, particularly after powerful devices such as recent heterogeneous embedded systems have demonstrated remarkable skills for accelerating their challenging computational requirements. In this work, we evaluate different hardware and software optimizations applied to state-of-the-art gait recognition approaches deployed on two Jetson devices with very different hardware capabilities. Specifically, we have selected three models with different characteristics in order to provide an in-depth deployment evaluation. This way, a 2D convolution-based model allows us to evaluate devices performance when a huge number of parameters must be managed. A model based on 3D convolutions is deployed to study devices capability to perform these kinds of operations. Finally, a novel model with a small number of parameters but with a huge number of activations is also evaluated. Obtained results show that different hardware and software optimizations are able to improve up to 5.4× energy consumption and 4.2× execution time w.r.t. baseline deployment, depending on the model and target device.11Code will be available at https://github.com/PaulaRuizB/Embedded-Gait.

GramBeddings: A New Neural Network for URL Based Identification of Phishing Web Pages Through N-gram Embeddings

There has been ever-growing use of Internet and progress within many communication channels such as social media and this escalates the need for rapid and low source demanding phishing detection mechanisms. In this very study, we propose a new deep neural model for phishing URL identification so-called GramBeddings introducing some distinguishing novelties by (1) proposing the use of n-gram embeddings, computed on the fly, requiring no pre-training stage, (2) removing the necessity of word and sub-word level information, (3) providing a smart and efficient n-gram selection pipeline, and benefiting from attention mechanism. Other than that, we share a publicly available, large-scale and novel dataset22https://web.cs.hacettepe.edu.tr/~selman/grambeddings-dataset/ including 800K real-world phishing and legitimate URLs. Our scheme suggests an adjustable and automated n-gram selection and filtering mechanism along with a new neural network architecture concatenating four-channel information flow through cascading CNN, LSTM, and attention layers. With that, discriminative multi-level character patterns can be discovered without any hand-crafted operation and are enabled to contribute to prediction. As a result, the proposed system provides the following features in the problem domain: (i) real-time, end-to-end and high performance inference, (ii) language-agnostic prediction, and (iii) removal of the necessity of any third-party service or hand-crafted feature. These experiments show that our approach outperforms the other models in the literature with an accuracy of 98.27%. Moreover, the comparative study conducted with several datasets clearly verifies the superiority of our model in all tests. We also examine the robustness of our model against a real-world adversarial attack and discuss the methods of overcoming such an attack. Our codebase33The code and our supplementary material will be made available at https://www.github.com/fcdalgic/GramBeddings is shared with the community to be used for benchmarking purposes in the future.

TensorRT-Based Framework and Optimization Methodology for Deep Learning Inference on Jetson Boards

As deep learning inference applications are increasing in embedded devices, an embedded device tends to equip neural processing units (NPUs) in addition to a multi-core CPU and a GPU. NVIDIA Jetson AGX Xavier is an example. For fast and efficient development of deep learning applications, TensorRT is provided as the SDK for high-performance inference, including an optimizer and runtime that delivers low latency and high throughput for deep learning inference applications. Like most deep learning frameworks, TensorRT assumes that the inference is executed on a single processing element, GPU or NPU, not both. In this article, we present a TensorRT-based framework supporting various optimization parameters to accelerate a deep learning application targeted on an NVIDIA Jetson embedded platform with heterogeneous processors, including multi-threading, pipelining, buffer assignment, and network duplication. Since the design space of allocating layers to diverse processing elements and optimizing other parameters is huge, we devise a parameter optimization methodology that consists of a heuristic for balancing pipeline stages among heterogeneous processors and fine-tuning the process for optimizing parameters. With nine real-life benchmarks, we could achieve 101%~680% performance improvement and up to 55% energy reduction over the baseline inference using a GPU only.

User-friendly image-activated microfluidic cell sorting technique using an optimized, fast deep learning algorithm.

Image-activated cell sorting is an essential biomedical research technique for understanding the unique characteristics of single cells. Deep learning algorithms can be used to extract hidden cell features from high-content image information to enable the discrimination of cell-to-cell differences in image-activated cell sorters. However, such systems are challenging to implement from a technical perspective due to the advanced imaging and sorting requirements and the long processing times of deep learning algorithms. Here, we introduce a user-friendly image-activated microfluidic sorting technique based on a fast deep learning model under the TensorRT framework to enable sorting decisions within 3 ms. The proposed sorter employs a significantly simplified operational procedure based on the use of a syringe connected to a piezoelectric actuator. The sorter has a 2.5 ms latency. The utility of the sorter was demonstrated through real-time sorting of fluorescent polystyrene beads and cells. The sorter achieved 98.0%, 95.1%, and 94.2% sorting purities for 15 μm and 10 μm beads, HL-60 and Jurkat cells, and HL-60 and K562 cells, respectively, with a throughput of up to 82.8 events per second (eps).

High Performance Neural Network Inference, Streaming, and Visualization of Medical Images Using FAST

Deep convolutional neural networks have quickly become the standard for medical image analysis. Although there are many frameworks focusing on training neural networks, there are few that focus on high performance inference and visualization of medical images. Neural network inference requires an inference engine (IE), and there are currently several IEs available including Intel’s OpenVINO, NVIDIA’s TensorRT, and Google’s TensorFlow which supports multiple backends, including NVIDIA’s cuDNN, AMD’s ROCm and Intel’s MKL-DNN. These IEs only work on specific processors and have completely different application programming interfaces (APIs). In this paper, we presents methods for extending FAST, an open-source high performance framework for medical imaging, to use any IE with a common programming interface. Thereby making it easier for users to deploy and test their neural networks on different processors. This article provides an overview of current IEs and how they can be combined with existing software such as FAST. The methods are demonstrated and evaluated on three performance demanding medical use cases: real-time ultrasound image segmentation, computed tomography (CT) volume segmentation, and patch-wise classification of whole slide microscopy images. Runtime performance was measured on the three use cases with several different IEs and processors. This revealed that the choice of IE and processor can affect performance of medical neural network image analysis considerably. In the most extreme case of processing 171 ultrasound frames, the difference between the fastest and slowest configuration were half a second vs. 24 seconds. For volume processing, using the CPU or the GPU, showed a difference of 2 vs. 53 seconds, and for processing an whole slide microscopy image, the difference was 81 seconds vs. almost 16 minutes. Source code, binary releases and test data can be found online on GitHub at https://github.com/smistad/FAST/ .

UNIRED II: The high performance inference processor for the parallel inference machine PIE64

UNIRED II is the high performance inference processor for the parallel inference machine PIE64. It is designed for the committed choice language Fleng, and for use as an element processor of parallel machines. Its main features are: 1) a tag architecture, 2) three independent memory buses (instruction fetching, data reading, and data writing), 3) a dedicated instruction set for efficient execution of Fleng, 4) multi-context processing for reducing pipeline interlocking and overhead of interprocessor synchronization. With the multi-context processing mechanism, the internal pipeline is shared by several independent instruction streams (contexts), and which contexts are to be executed is determined cycle by cycle. So, UNIRED II acts as a shared-pipeline MIMD processor. In this paper, several architectural features including the multi-context processing and the instruction set are described. Performance measurement results by simulation are also presented. On a 10MHz UNIRED II, 920K Reduction Per Second has been achieved, and it is shown that the multi-context processing mechanism is very effective for improved performance.

Towards a parallel inference machine: the APIM project

This paper gives an overview of the APIM (Aristotle Parallel Inference Machine) research project, which aims to the design of a new multiprocessor architecture for the parallel execution of logic programs. The APIM system is designed to support both OR-parallelism and independent AND-parallelism in an all solutions approach. It has been developed by adapting the RPM, a sequential high performance inference machine, developed in our Lab.