Fast Deep Neural Network Research Articles

Stochastic Cumulative DNN Inference With RL-Aided Adaptive IoT Device-Edge Collaboration

The advances in artificial intelligence (AI) and edge computing enable edge intelligence to support pervasive intelligent Internet of Things (IoT) applications in the future wireless networks. We focus on deep neural network (DNN) based classification tasks, and investigate how to improve the confidence level and delay performance of DNN inference via device-edge collaboration. We first develop a stochastic cumulative DNN inference scheme that aggregates multiple random DNN inference results and generates a cumulative DNN inference result with improved confidence level. Then, based on a computation-efficient DNN model deployment strategy with shared computation between a locally deployed fast DNN model and a full DNN model partitioned between the device and edge, a closed-loop adaptive device-edge collaboration scheme is developed to support cumulative DNN inference for multiple devices. We adaptively determine how to offload DNN inference computation to the edge and how to allocate transmission and edge computing resources among multiple devices, for quality-of-service (QoS) satisfaction in terms of both confidence level and inference delay with resource and energy efficiency. A reinforcement learning (RL) approach is used for adaptive offloading decision, which relies on a resource allocation solution for reward calculation. Simulation results demonstrate the effectiveness of the adaptive device-edge collaboration scheme for cumulative DNN inference, in terms of confidence level improvement, delay violation minimization, network resource efficiency, and device energy efficiency.

PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters

Recently, pipeline parallelism for large-scale Deep Neural Network (DNN) training has been developed, which partitions the DNN model across multiple devices (e.g., GPUs) and improves the training efficiency by processing data divided into minibatches as a pipeline. However, existing model partitioning algorithms are mostly designed for homogeneous clusters with the same GPU devices and network connections (e.g., bandwidths), while heterogeneous GPU clusters are widely used in mainstream computing infrastructures. In heterogeneous environment, devices are equipped with different GPUs and network connections, and the efficiency of previous approaches is unsatisfactory due to the unbalanced load of the pipeline stages. In this paper, we propose PipePar, a model partitioning and task placement algorithm for pipeline parallel DNN training in heterogeneous GPU clusters. PipePar is based on dynamic programming with search space pruning that takes into consideration both the heterogeneity of GPUs and network bandwidth. PipePar can profile the DNN model for each type of GPU and conduct model partitioning and task placement based on given GPUs and network connections, which can optimize pipeline load balancing in heterogeneous environments and thus improve training efficiency. We design and implement a pipeline-based distributed deep learning training system in a heterogeneous GPU cluster and show through extensive experiments that PipePar outperforms the baseline approaches in the speed of large-scale DNN training.

Characterizing the Performance of Accelerated Jetson Edge Devices for Training Deep Learning Models

Deep Neural Network (DNN) models are becoming ubiquitous in a variety of contemporary domains such as Autonomous Vehicles, Smart cities and Healthcare. They help drones to navigate, identify suspicious activities from safety cameras, and perform diagnostics over medical imaging. Fast DNN inferencing close to the data source is enabled by a growing class of accelerated edge devices such as NVIDIA Jetson and Google Coral which host low-power Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) along with ARM CPUs in a compact form-factor to offer a superior performance-to-energy ratio. E.g., the NVIDIA Jetson AGX Xavier kit has a 512-core Volta GPU, an 8-core ARM CPU and 32GB LPDDR4x memory, that operates within 65W of power, costs US999 and is smaller than a paperback novel. Recently, there has been a push towards training DNN models on the edge [2]. This is driven by the massive growth in data collected from edge devices in Cyber-Physical Systems (CPS) and Internet of Things (IoT), the need to refresh the models periodically, the bandwidth constraints in moving all this data to Cloud data centers for training, and a heightened emphasis on privacy by retaining data on the edge. This has led to techniques like federated and geo-distributed learning that train DNN models locally on data on an edge device and aggregate them centrally. In this abstract, we summarise and highlight key results from our full paper [5].

Spatio-temporal Mapping of Cotton Blooms Appearance Using Deep Learning

In this paper, we present an approach to create spatio-temporal maps using deep learning to visualize cotton bloom appearance over time. Specifically, we manually annotate cotton flower image data and train three state-of-the-art fast deep neural network models to count cotton blooms and their frequency over time prior to harvesting. We use the detection results of the best performing model combined with traditional pixel-based image analysis methods to create a map of where past and future blooms grow on a mid-stage cotton plant. The training results of our best model show a visual understanding of how many cotton flowers grow with high F1 Score of more than 0.95, a true positive rate of 98%, false negative and false positive rates both under 10%, and millisecond-scale inference time for real-time processing.

Model compression and simplification pipelines for fast deep neural network inference in FPGAs in HEP

Resource utilization plays a crucial role for successful implementation of fast real-time inference for deep neural networks (DNNs) and convolutional neural networks (CNNs) on latest generation of hardware accelerators (FPGAs, SoCs, ACAPs, GPUs). To fulfil the needs of the triggers that are in development for the upgraded LHC detectors, we have developed a multi-stage compression approach based on conventional compression strategies (pruning and quantization) to reduce the memory footprint of the model and knowledge transfer techniques, crucial to streamline the DNNs simplifying the synthesis phase in the FPGA firmware and improving explainability. We present the developed methodologies and the results of the implementation in a working engineering pipeline used as pre-processing stage to high level synthesis tools (HLS4ML, Xilinx Vivado HLS, etc.). We show how it is possible to build ultra-light deep neural networks in practice, by applying the method to a realistic HEP use-case: a toy simulation of one of the triggers planned for the HL-LHC.

Semantic Point Cloud Segmentation Using Fast Deep Neural Network and DCRF.

Accurate segmentation of entity categories is the critical step for 3D scene understanding. This paper presents a fast deep neural network model with Dense Conditional Random Field (DCRF) as a post-processing method, which can perform accurate semantic segmentation for 3D point cloud scene. On this basis, a compact but flexible framework is introduced for performing segmentation to the semantics of point clouds concurrently, contribute to more precise segmentation. Moreover, based on semantics labels, a novel DCRF model is elaborated to refine the result of segmentation. Besides, without any sacrifice to accuracy, we apply optimization to the original data of the point cloud, allowing the network to handle fewer data. In the experiment, our proposed method is conducted comprehensively through four evaluation indicators, proving the superiority of our method.

Comparison of Input Image Dimensions for Ship Detection from KOMPSAT-5 SAR Image Using Deep Neural Network

Park, W.; Baek, W.-K.; Won, J.-S., and Jung, H.-S., 2020. Comparison of input image size for ship detection from KOMPSAT-5 SAR image using deep neural networks (DNNs). In: Jung, H.-S.; Lee, S.; Ryu, J.-H., and Cui, T. (eds.), Advances in Geospatial Research of Coastal Environments. Journal of Coastal Research, Special Issue No. 102, pp. 208-217. Coconut Creek (Florida), ISSN 0749-0208.This study shows the results of ship detection in KOMPSAT-5 X-band SAR images using Deep Learning (DL) based the fast Deep Neural Network (DNN) method with the iterative kernel-based false alram detection algorithm. In addition, this study verifies the detection accuracy according to the size of the input data for are typical errors in SAR images such as backscattering noise, side-lobe, and ghost effect. The test results of the Fast DNN method confirmed that the error caused by the back scattering noise and side-lobe decreased as the input data size increased, however, the error for the ghost effect did not improve significantly. Also, the False Alarm Rate (FAR) was greatly improved using the iterative kernel-based false alram detection algorithm. However, caution is needed when detecting small ships because very small boats disappear. The results of this study showed that the size of the input data was very effective at 51 × 51 or higher when the fast DNN method and the iterative kernel-based false alram detection algorithm were applied to the KOMPSAT-5 images to detect ships, allowing the FAR and Intersaction over Union (IoU) showing relatively high accuracy at 0.270 and 0.611.

The Interpretable Fast Multi-Scale Deep Decoder for the Standard HEVC Bitstreams

It is a research hotspot to restore decoded videos with existing bitstreams by applying deep neural network to improve compression efficiency at decoder-end. Existing research has verified that the utilization of redundancy at decoder-end, which is underused by the encoder, can bring an increase of compression efficiency. However, most existing research neglects the abundant multi-scale information among video frames as a typical type of such redundancy. It remains an interesting yet challenging topic how to build an effective, interpretable and fast deep neural network for the purpose of using the multi-scale similarity at decoder-end and further enhancing compression efficiency. To this end, this paper considers the use of underused inter multi-scale information and proposes the Fast Multi-Scale Deep Decoder (Fast MSDD) for the state-of-the-art video coding standard HEVC. The advantages of Fast MSDD are three-fold. First, it achieves a higher coding efficiency without modifying any encoding algorithm. Second, Fast MSDD is interpretable based on the framework of using the underused redundancy. Third, it guarantees the model's inference speed while fully using the multi-scale similarity among video frames. Extensive experimental results verify Fast MSDD's effectiveness, interpretability, and computational efficiency. Fast MSDD obtains averagely 14.3%, 10.8%, 8.5% and 7.6% BD gains for AI, LP, LB and RA respectively. Compared with our previous work MSDD, Fast MSDD achieves increases of 59.3%, 49.1%, 61.0% and 29.3%. Meanwhile, 16.9%, 11.2%, 9.2% and 8.3% BD gains are observed on videos with scale changes, which validate the interpretability of the proposed method. Furthermore, Fast MSDD can save at most 56.3% time compared to MSDD.

Fast deep neural networks for image processing using posits and ARM scalable vector extension

With the advent of image processing and computer vision for automotive under real-time constraints, the need for fast and architecture-optimized arithmetic operations is crucial. Alternative and efficient representations for real numbers are starting to be explored, and among them, the recently introduced posit$$^{\mathrm{TM}}$$ number system is highly promising. Furthermore, with the implementation of the architecture-specific mathematical library thoroughly targeting single-instruction multiple-data (SIMD) engines, the acceleration provided to deep neural networks framework is increasing. In this paper, we present the implementation of some core image processing operations exploiting the posit arithmetic and the ARM scalable vector extension SIMD engine. Moreover, we present applications of real-time image processing to the autonomous driving scenario, presenting benchmarks on the tinyDNN deep neural network (DNN) framework.

Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs

Since its creation, the ImageNet-1k benchmark set has played a significant role as a benchmark for ascertaining the accuracy of different deep neural net (DNN) models on the image classification problem. Moreover, in recent years it has also served as the principal benchmark for assessing different approaches to DNN training. Finishing a 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. This training requires $10^{18}$1018 single precision operations in total. On the other hand, the world's current fastest supercomputer can finish $3 \times 10^{17}$3×1017 single precision operations per second (according to the Nov 2018 Top 500 results). If we can make full use of the computing capability of the fastest supercomputer, we should be able to finish the training in several seconds. Over the last two years, researchers have focused on closing this significant performance gap through scaling DNN training to larger numbers of processors. Most successful approaches to scaling ImageNet training have used the synchronous mini-batch stochastic gradient descent (SGD). However, to scale synchronous SGD one must also increase the batch size used in each iteration. Thus, for many researchers, the focus on scaling DNN training has translated into a focus on developing training algorithms that enable increasing the batch size in data-parallel synchronous SGD without losing accuracy over a fixed number of epochs. In this paper, we investigate supercomputers’ capability of speeding up DNN training. Our approach is to use a large batch size, powered by the Layer-wise Adaptive Rate Scaling (LARS) algorithm, for efficient usage of massive computing resources. Our approach is generic, as we empirically evaluate the effectiveness on five neural networks: AlexNet, AlexNet-BN, GNMT, ResNet-50, and ResNet-50-v2 trained with large datasets while preserving the state-of-the-art test accuracy. Compared to the baseline of a previous study from Goyal et al. [1] , our approach shows higher test accuracy on batch sizes that are larger than 16K. When we use the same baseline, our results are better than Goyal et al. for all the batch sizes (Fig. 20 ). Using 2,048 Intel Xeon Platinum 8160 processors, we reduce the 100-epoch AlexNet training time from hours to 11 minutes. With 2,048 Intel Xeon Phi 7250 Processors, we reduce the 90-epoch ResNet-50 training time from hours to 20 minutes. Our implementation is open source and has been released in the Intel distribution of Caffe, Facebook's PyTorch, and Google's TensorFlow. The difference between this paper and the conference-version of our work [2] includes: (1) we implement our approach on Google's cloud Tensor Processing Unit (TPU) platform, which verifies our previous success on CPUs and GPUs. (2) we scale the batch size of ResNet-50-v2 to 32K and achieve 76.3 percent accuracy, which is better than the 75.3 percent accuracy achieved in our conference paper. (3) we apply our approach to Google's Neural Machine Translation (GNMT) application, which helps us to achieves 4× speedup on the cloud TPUs.

심층 신경망의 영상 인식 분류 성능 균일성 향상을 위한 명시적 상호보완 앙상블 구조

본 논문에서는 영상 인식 분류 시스템의 주요한 성능평가 고려사항 중 하나인 분류 대상 별 성능 편차문제를 완화시키기 위한 새로운 심층 신경망 기반 조화 앙상블(Harmony) 기법을 제안한다. 조화 앙상블 기법은 표적 모델(Target model), 보완 모델(Complementary model) 그리고 지휘자 모델(Conductor model)로 명명된 3가지의 하위 모듈들로 구성된다. 제안하는 기법에서의 분류 대상은 지휘자 모델에서의 결과에 따라 표적 모델 또는 보완 모델을 통해 선택적으로 처리된다. 표적 모델은 종래의 분류모델과 동일한 방식으로 학습된다. 반면에 보완 모델은 표적 모델의 취약한 분류 대상에 대하여 상호보완적인 분류 성능을 가지도록 학습된다. 지휘자 모델은 분류 대상이 표적 모델의 취약 분류대상에 속하는지를 사전에 분류하여 분류대상을 표적 모델과 보완 모델 중에 어떤 모델을 사용하여 분류 할 것인지를 결정한다. 논문의 끝에서는 실험적 결과를 통하여 제안하는 조화 앙상블 기법을 통해 평균 분류성능의 대한 손실 없이 분류 대상 별 성능 편차문제를 효과적으로 완화시킬 수 있음을 확인한다.

Fast Deep Neural Networks With Knowledge Guided Training and Predicted Regions of Interests for Real-Time Video Object Detection

It has been recognized that deeper and wider neural networks are continuously advancing the state-of-the-art performance of various computer vision and machine learning tasks. However, they often require large sets of labeled data for effective training and suffer from extremely high computational complexity, preventing them from being deployed in real-time systems, for example vehicle object detection from vehicle cameras for assisted driving. In this paper, we aim to develop a fast deep neural network for real-time video object detection by exploring the ideas of knowledge-guided training and predicted regions of interest. Specifically, we will develop a new framework for training deep neural networks on datasets with limited labeled samples using cross-network knowledge projection which is able to improve the network performance while reducing the overall computational complexity significantly. A large pre-trained teacher network is used to observe samples from the training data. A projection matrix is learned to project this teacher-level knowledge and its visual representations from an intermediate layer of the teacher network to an intermediate layer of a thinner and faster student network to guide and regulate the training process. To further speed up the network, we propose to train a low-complexity object detection using traditional machine learning methods, such as support vector machine. Using this low-complexity object detector, we identify the regions of interest that contain the target objects with high confidence. We obtain a mathematical formula to estimate the regions of interest to save the computation for each convolution layer. Our experimental results on vehicle detection from videos demonstrated that the proposed method is able to speed up the network by up to 16 times while maintaining the object detection performance.

Soft Memory Box: A Virtual Shared Memory Framework for Fast Deep Neural Network Training in Distributed High Performance Computing

Deep learning is one of the major promising machine learning methodologies. Deep learning is widely used in various application domains, e.g., image recognition, voice recognition, and natural language processing. In order to improve learning accuracy, deep neural networks have evolved by: 1) increasing the number of layers and 2) increasing the number of parameters in massive models. This implies that distributed deep learning platforms need to evolve to: 1) deal with huge/complex deep neural networks and 2) process with high-performance computing resources for massive training data. This paper proposes a new virtual shared memory framework, called Soft Memory Box (SMB), which enables sharing the memory of remote node among distributed processes in the nodes so as to improve communication performance via parameter sharing. According to data-intensive performance evaluation results, the communication time of deep learning using the proposed SMB is 2.1 times faster than that using the massage passing interface (MPI). In addition, the communication time of the SMB-based asynchronous parameter update becomes 2–7 times faster than that using the MPI depending on deep learning models and the number of deep learning workers.

Fast and Accurate Real Time Pedestrian Detection Using Convolutional Neural Network

Recently, pedestrian detection has become an important problem of interest. Our work primarily depends on robust and fast deep neural network architectures. This paper used very efficient and recent methods for pedestrian detection. Recently, pedestrian detection has become an important problem of interest. This paper suggests robust convolutional neural network models to solve this problem. We primarily evaluate accuracy and speed. Our work primarily depends on robust and fast deep neural network architectures; substantial changes to those models achieve results that are competitive with prior state-of-the-art methods. As a result, we outperformed all the prior state-of-the-art pedestrian detection methods. We also overtook other models that use extra information during testing and training. All experiments used three pedestrian detection challenge benchmarks: Caltech-USA, INRIA, and ETH.