Computation Of Deep Neural Networks Research Articles

A population-based DNN-augmented optimization method for designing truss structures

Truss design problems have been extensively investigated for various purposes. How to efficiently optimize truss structures and obtain diverse optimized designs is still a challenging task. In this paper, a population-based DNN (deep neural network) -augmented method is proposed to address this issue. In the proposed method, truss design problems are firstly transformed into a DNN-based design loss minimization problem. The DNN model is gradually generated in an adaptive updating manner, and is used to indicate the truss design problem. In addition, the transformed minimization problem can be efficiently solved by DNN computation and a population of trusses can be optimized accordingly. Through the investigation of four truss design problems (two standard optimization problems and two optimization problems considering the design diversity), the effectiveness of the proposed method is validated. In addition, three aspects affecting the performance of the proposed method are also investigated, and the robustness and applicability of the proposed method are demonstrated.

Mixture-of-Rookies: Saving DNN computations by predicting ReLU outputs

Deep Neural Networks (DNNs) are widely used in many application domains. However, they require a vast amount of computations and memory accesses to deliver outstanding accuracy. In this paper, we propose a scheme to predict whether the output of each ReLu activated neuron will be a zero or a positive number in order to skip the computation of those neurons that will likely output a zero. Our predictor, named Mixture-of-Rookies, combines two inexpensive components. The first one exploits the high linear correlation between binarized (1-bit) and full-precision (8-bit) dot products, whereas the second component clusters together neurons that tend to output zero at the same time. We propose a novel clustering scheme based on analysis of angles, as the sign of the dot product of two vectors depends on the cosine of the angle between them. We implement our hybrid zero output predictor on top of a state-of-the-art DNN accelerator. Experimental results show that our scheme introduces a small area overhead of 5.3% while achieving a speedup of 1.2x and reducing energy consumption by 16.5% on average for a set of diverse DNNs.

DNN acceleration in vehicle edge computing with mobility-awareness: A synergistic vehicle–edge and edge–edge framework

In recent years, vehicular networks have seen a proliferation of applications and services such as image tagging, lane detection, and speech recognition. Many of these applications rely on Deep Neural Networks (DNNs) and demand low-latency computation. To meet these requirements, Vehicular Edge Computing (VEC) has been introduced to augment the abundant computation capacity of vehicular networks to complement limited computation resources on vehicles. Nevertheless, offloading DNN tasks to MEC (Multi-access Edge Computing) servers effectively and efficiently remains a challenging topic due to the dynamic nature of vehicular mobility and varying loads on the servers. In this paper, we propose a novel and efficient distributed DNN Partitioning And Offloading (DPAO), leveraging the mobility of vehicles and the synergy between vehicle–edge and edge–edge computing. We exploit the variations in both computation time and output data size across different layers of DNN to make optimized decisions for accelerating DNN computations while reducing the transmission time of intermediate data. In the meantime, we dynamically partition and offload tasks between MEC servers based on their load differences. We have conducted extensive simulations and testbed experiments to demonstrate the effectiveness of DPAO. The evaluation results show that, compared to offloaded all tasks to MEC server, DPAO reduces the latency of DNN tasks by 2.4x. DPAO with queue reservation can further reduce the task average completion time by 10%.

A DNN partitioning framework with controlled lossy mechanisms for edge-cloud collaborative intelligence

Collaborative intelligence (CI) is an approach used to accelerate deep neural network computations on IoT edge devices by offloading certain DNN computations to high-performance cloud servers. This offloading process involves transferring a large volume of feature data between the IoT edge device and the cloud servers, resulting in significant transfer time overhead that contributes to the overall end-to-end latency. Despite its importance, CI research has largely overlooked the impact of an unstable network environment on transfer time, leading to suboptimal latency in real-world scenarios. In light of this, we propose CO-PILOT, a novel partition framework that aims to minimize end-to-end latency while preserving the original model accuracy. CO-PILOT achieves this by finely controlling various lossy mechanisms, such as compression level and transport protocol, and strategically selecting error-robust layers as partition points to enhance resilience against network conditions and lossy mechanisms. The results demonstrate that CO-PILOT can reduce end-to-end latency by up to 85% with less than 1% accuracy degradation, all without requiring model retraining.

Sequential Offloading for Distributed DNN Computation in Multiuser MEC Systems

This paper studies a sequential task offloading problem for a multiuser mobile edge computing (MEC) system. While most of the existing works consider static one-shot offloading optimization with fixed wireless channel conditions and fixed computational tasks, we consider a dynamic optimization approach, which embraces wireless channel fluctuations and random deep neural network (DNN) task arrivals over an infinite horizon. Specifically, we introduce a local CPU workload queue (WD-QSI) and an MEC server workload queue (MEC-QSI) to model the dynamic workload of DNN tasks at each WD and the MEC server, respectively. The transmit power and the partitioning of the local DNN task at each WD are dynamically determined based on the instantaneous channel conditions (to capture the transmission opportunities) and the instantaneous WD-QSI and MEC-QSI (to capture the dynamic urgency of the tasks) to minimize the average latency of the DNN tasks. The joint optimization can be formulated as an ergodic Markov decision process (MDP), in which the optimality condition is characterized by a centralized Bellman equation. However, the brute force solution of the MDP is not viable due to the curse of dimensionality as well as the requirement for knowledge of the global state information. To overcome these issues, we first decompose the MDP into multiple lower dimensional sub-MDPs, each of which can be associated with a WD or the MEC server. Next, we further develop a parametric online Q-learning algorithm, so that each sub-MDP is solved locally at its associated WD or the MEC server. The proposed solution is completely decentralized in the sense that the transmit power for sequential offloading and the DNN task partitioning can be determined based on the local channel state information (CSI) and the local WD-QSI at the WD only. Additionally, no prior knowledge of the distribution of the DNN task arrivals or the channel statistics will be needed for the MEC server. The proposed solution can achieve the superb performance over various state-of-the-art baselines.

Accurate Weight Update in an Electrochemical Random‐Access Memory Based Cross‐Point Array Using Channel‐High Half‐Bias Scheme for Deep Learning Accelerator

AbstractRecently cross‐point arrays of synaptic memory devices have been intensively studied to accelerate deep neural network computations. Among various synaptic devices, electrochemical random‐access memory (ECRAM) is emerging as a promising non‐volatile memory candidate owing to its superior synaptic characteristics. However, an optimized update scheme for a three‐terminal ECRAM‐based cross‐point array is yet to be developed. In this study, a metal‐oxide‐based ECRAM (MO‐ECRAM) shows superior synaptic characteristics and the weight update of devices in the MO‐ECRAM cross‐point array is analyzed using the half‐bias (HB) scheme. Additionally, A channel‐high half‐bias (CHB) scheme is proposed to overcome the degraded selectivity of the weight update caused by the three‐terminal configuration of the ECRAM device. In the CHB scheme, the conductance change in the selected device can be increased considerably by applying a calculated additional voltage to the channel. Using the CHB scheme, parallel and selective updates are successfully performed in a 2 × 2 MO‐ECRAM cross‐point array. Finally, an experimental demonstration of the training algorithm shows the impact of selective updates when using the CHB scheme. This new update scheme is expected to improve training accuracy in ECRAM cross‐point array‐based deep learning accelerators.

Multi-exit DNN inference acceleration for intelligent terminal with heterogeneous processors

Recently, there has been a burgeoning popularity in the deployment of deep learning vision applications upon terminal devices. However, as the number of layers in deep neural networks (DNNs) and structural complexity increase, although the performance of DNN in handling computer vision tasks has become powerful, model inference tasks on computation resource constrained intelligent terminal devices are frequently incapable of meeting latency requirement. A commonly adopted solution to inference acceleration presents multi-exit DNNs to reduce latency via the provision of early exits. However, existing methods do not fully utilize the potential of heterogeneous processors (GPU/CPU) on intelligent terminal devices to cooperatively accelerate multi-exit DNN inference in parallel. Furthermore, the impact of complex image and video input on multi-exit DNNs, as well as the effects of different power consumption modes on processors within intelligent terminal devices, remain inadequately explored. To address these issues, we comprehensively considered the computing performance of heterogeneous processors in different power consumption modes, the structure and characteristic of multi-exit DNNs in inference acceleration, and proposed the Collaborative Inference Acceleration mechanism for intelligent terminal with Heterogeneous Processors (CIAHP). CIAHP includes a deep neural network computation time prediction model and a multi-exit DNN task allocation algorithm with heterogeneous processors. Our experiments demonstrate that CIAHP performs multi-exit DNN inference 2.31× faster than CPU alone, and is 1.23× faster than GPU alone when processing complex image samples.

UAV-Enabled Mobile-Edge Computing for AI Applications: Joint Model Decision, Resource Allocation, and Trajectory Optimization

Due to the flexible mobility and agility, unmanned aerial vehicles (UAVs) are expected to be deployed as aerial base stations (BSs) in future air-ground integrated wireless networks, providing temporary and controllable coverage and additional computation capabilities for ground Internet of Things (IoT) devices with or without infrastructure support. Meanwhile, with the breakthrough of artificial intelligence (AI), more and more AI applications relying on AI methods such as deep neural networks (DNNs) are expected to be applied in various fields such as smart homes, smart factories and smart cities to improve our lifestyles and efficiency dramatically. However, AI applications are generally computation-intensive, latency-sensitive, and energy-consuming, making resource-constrained IoT devices unable to benefit from AI anytime and anywhere. In this paper, we study mobile edge computing (MEC) for AI applications in air-ground integrated wireless networks. Our goal is to minimize the service latency while ensuring the learning accuracy requirements and energy consumption. To achieve that, we take DNN as the typical AI application and formulate an optimization problem that optimizes the DNN model decision, computation and communication resource allocation, and UAV trajectory control, subject to the energy consumption, latency, computation and communication resource constraints. Considering the formulated problem is non-convex, we decompose it into multiple convex subproblems and then alternately solve them till they converge to the desired solution. Simulation results show that the proposed algorithm significantly improves the system performance for AI applications.

Low-Latency Federated Learning With DNN Partition in Distributed Industrial IoT Networks

Federated Learning (FL) empowers Industrial Internet of Things (IIoT) with distributed intelligence of industrial automation thanks to its capability of distributed machine learning without any raw data exchange. However, it is rather challenging for lightweight IIoT devices to perform computation-intensive local model training over large-scale deep neural networks (DNNs). Driven by this issue, we develop a communication-computation efficient FL framework for resource-limited IIoT networks that integrates DNN partition technique into the standard FL mechanism, wherein IIoT devices perform local model training over the bottom layers of the objective DNN, and offload the top layers to the edge gateway side. Considering imbalanced data distribution, we derive the device-specific participation rate to involve the devices with better data distribution in more communication rounds. Upon deriving the device-specific participation rate, we propose to minimize the training delay under the constraints of device-specific participation rate, energy consumption and memory usage. To this end, we formulate a joint optimization problem of device scheduling and resource allocation (i.e. DNN partition point, channel assignment, transmit power, and computation frequency), and solve the long-term min-max mixed integer non-linear programming based on the Lyapunov technique. In particular, the proposed dynamic device scheduling and resource allocation (DDSRA) algorithm can achieve a trade-off to balance the training delay minimization and FL performance. We also provide the FL convergence bound for the DDSRA algorithm with both convex and non-convex settings. Experimental results demonstrate the derived device-specific participation rate in terms of feasibility, and show that the DDSRA algorithm outperforms baselines in terms of test accuracy and convergence time.

A Speculative Computation Approach for Energy-Efficient Deep Neural Network

Deep neural networks (DNNs) have been widely used for data processing and analysis nowadays. Many computational techniques have been proposed to improve the energy efficiency of executing DNNs, which is critical for emerging smart edge applications. This article presents a speculative computation approach to improving the energy efficiency of DNN computations. The proposed approach employs the techniques of input channel partitioning and threshold-based negative masking to predict and eliminate unnecessary computations. Moreover, a systematic procedure of threshold optimization is proposed to achieve the best tradeoff between the energy and accuracy performance. Finally, an energy-efficient DNN processor architecture was designed and implemented to support the proposed speculative computation approach. The experimental results show that the proposed DNN processor with speculative computation can enhance the energy efficiency of the processor by 22.8%, with only 0.96% accuracy degradation and 1% implementation overhead.

Reconfigurability, Why It Matters in AI Tasks Processing: A Survey of Reconfigurable AI Chips

Nowadays, artificial intelligence (AI) technologies, especially deep neural networks (DNNs), play an vital role in solving many problems in both academia and industry. In order to simultaneously meet the demand of performance, energy efficiency and flexibility in DNN processing, various reconfigurable AI chips have been proposed in the past several years. They are based on FPGA or CGRA platforms and have domain-specific reconfigurability to customize the computing units and data paths for different DNN tasks without re-produce the chips. This paper surveys typical reconfigurable AI chips from three reconfiguration hierarchies: processing element level, processing element array level, and chip level. Each reconfiguration hierarchy covers a set of important optimization techniques for DNN computation which are frequently adopted in real life. This paper lists the reconfigurable AI chip works in chronological order, discusses the hardware development process for each optimization techniques, and analyzes the necessity of reconfigurability in AI tasks processing. The trends of each reconfiguration hierarchy and insights about the cooperation of techniques from different hierarchies are also proposed.

Transforming Large-Size to Lightweight Deep Neural Networks for IoT Applications

Deep Neural Networks (DNNs) have gained unprecedented popularity due to their high-order performance and automated feature extraction capability. This has encouraged researchers to incorporate DNN in different Internet of Things (IoT) applications in recent years. However, the colossal requirement of computation, energy, and storage of DNNs make their deployment prohibitive on resource-constrained IoT devices. Therefore, several compression techniques have been proposed in recent years to reduce the energy, storage, and computation requirements of the DNN. These techniques have utilized a different perspective for compressing a DNN with minimal accuracy compromise. This encourages us to comprehensively overview DNN compression techniques for the IoT. This article presents a comprehensive overview of existing literature on compressing the DNN that reduces energy consumption, storage, and computation requirements for IoT applications. We divide the existing approaches into five broad categories—network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous—based upon the mechanism incorporated for compressing the DNN. The article discusses the challenges associated with each category of DNN compression techniques and presents some prominent applications using IoT in conjunction with a compressed DNN. Finally, we provide a quick summary of existing work under each category with the future direction in DNN compression.

Accelerating Deep Learning Inference via Model Parallelism and Partial Computation Offloading

With the rapid development of Internet-of-Things (IoT) and the explosive advance of deep learning, there is an urgent need to enable deep learning inference on IoT devices in Mobile Edge Computing (MEC). To address the computation limitation of IoT devices in processing complex Deep Neural Networks (DNNs), computation offloading is proposed as a promising approach. Recently, partial computation offloading is developed to dynamically adjust task assignment strategy in different channel conditions for better performance. In this paper, we take advantage of intrinsic DNN computation characteristics and propose a novel Fused-Layer-based (FL-based) DNN model parallelism method to accelerate inference. The key idea is that a DNN layer can be converted to several smaller layers in order to increase partial computation offloading flexibility, and thus further create the better computation offloading solution. However, there is a trade-off between computation offloading flexibility as well as model parallelism overhead. Then, we investigate the optimal DNN model parallelism and the corresponding scheduling and offloading strategies in partial computation offloading. In particular, we propose a Particle Swarm Optimization with Minimizing Waiting (PSOMW) method, which explores and updates the FL strategy, path scheduling strategy, and path offloading strategy to reduce time complexity and avoid invalid solutions. Finally, we validate the effectiveness of the proposed method in commonly used DNNs. The results show that the proposed method can reduce the DNN inference time by an average of 12.75 times compared to the legacy No FL (NFL) algorithm, and is very close to the optimal solution achieved by the Brute Force (BF) algorithm with the difference of less than 0.04%.

Hierarchical Model Compression via Shape-Edge Representation of Feature Maps—an Enlightenment From the Primate Visual System

The cumbersome computation of deep neural networks (DNNs) limits the practical deployment of DNNs on resource-constrained mobile multimedia devices. To deploy DNNs on devices with limited power and computing resources, model compression techniques are leveraged to accelerate the networks, where network pruning can improve the inference efficiency for deployed DNNs by removing redundant weights and structures. As one of the important components of DNNs, the feature maps (FMs) can be leveraged to evaluate the importance of network structures for DNN pruning. However, previous methods neglect to fully explore the characteristics of FMs to prune the DNNs. In this paper, we investigate the high capacity and resource efficient analogy-ventral dual-pathway primates visual system to propose a hierarchical pruning framework (dubbed as HPSE). In an efficient primate visual system (PVS), the analog pathway analyzes low-frequency information to facilitate the high-frequency information inference in ventral stream. In HPSE, we extract the low-frequency shape information and high-frequency edge information from FMs to present a novel pruning pipeline that resembles the analysis mechanism of PVS. In particular, we first imitate the analogy pathway to group different FMs in each layer by calculating the shape-feature overlap. Secondly, we leverage the edge information modulated by the grouping results of the first step to prune the network. The effectiveness of HPSE is verified by pruning various DNNs on different benchmarks, and the results have achieved satisfactory performances. For example, for ResNet-56 on CIFAR-10, HPSE reduces 52.9% of FLOPs with a slight accuracy improvement; for ResNet-50 on ImageNet, we achieve 54.3%-FLOPs drop with only 0.49% Top-1 accuracy loss.

Accelerating Tensor Swapping in GPUs With Self-Tuning Compression

Data swapping between CPUs and GPUs is widely used to address the GPU memory shortage issue when training deep neural networks (DNNs) requiring a larger amount of memory than that a GPU may have. Data swapping may become a bottleneck when its latency is longer than the latency of DNN computations. Tensor compression in GPUs can reduce the data swapping time. However, existing works on compressing tensors in the virtual memory of GPUs have three major issues: lack of portability because its implementation requires additional (de)compression units in memory controllers, sub-optimal compression performance for varying tensor compression ratios and sizes, and poor adaptation to dense tensors because they only focus on sparse tensors. We propose a self-tuning tensor compression framework, named <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CSwap+</small> , for improving the virtual memory management of GPUs. It uses GPUs for (de)compression directly and thus has high portability and is minimally dependent on GPU architecture features. Furthermore, it only applies compression on tensors that are deemed to be cost-effective considering their compression ratio, size, and the characteristics of compression algorithms at runtime. Finally, to adapt to DNN models with dense tensors, it also supports cost-effective lossy compression for dense tensors with nearly no model training accuracy degradation. We conduct the experiments through six representative memory-intensive DNN models. Compared to vDNN, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CSwap+</small> reduces tensor swapping latency by up to 50.9% and 46.1% with NVIDIA V100 GPU, for DNN models with sparse and dense tensors, respectively.

Stuck-at-Fault Immunity Enhancement of Memristor-Based Edge AI Systems

Deep Neural Networks (DNNs) are widely used in edge AI. But the complex perception and decision-making demands the overlarge computation and makes the DNN architecture very sophisticated. Memristors have multilevel resistance property that enables faster in-memory DNN computation to remove the bottleneck caused by the von Neumann architecture and CMOS technology. However, the Stuck-at-Fault (SAF) defect of memristor generated from immature fabrication and heavy device utilization makes the memristor-based edge AI commercially unavailable. To mitigate this problem, an Adaptive Mapping Method (AMM) is proposed in this paper. Based on the analysis for the VGG8 model with CIFAR10 dataset, the experiment results show that the AMM is efficient in restoring the inference accuracy up to 90% (the original accuracy without SAF) under SAFs from 0.1% to 50%, where Stuck-at-One (SA1): Stuck-at-Zero (SA0) = 5:1, 1:5, and 1:1. Additionally, the AMM has a significant immunity against the nonlinearity and conductance drift. The AMM improves the accuracy more than 50% in presence of high nonlinearity LTP = 4 and LTD = −4 and the standard conductance drift (10 years at 85 degree centigrade) nearly has no influence on the inference accuracy of the DNN in edge AI with the AMM.

A Reconfigurable Neural Architecture for Edge–Cloud Collaborative Real-Time Object Detection

Although recent advances in deep neural networks (DNNs) have enabled remarkable performance on various computer vision tasks, it is challenging for edge devices to perform real-time inference of complex DNN models due to their stringent resource constraints. To enhance the inference throughput, recent studies have proposed collaborative intelligence (CI), which splits DNN computation into edge and cloud platforms, mostly for simple tasks, such as image classification. However, for general DNN-based object detectors with a branching architecture, CI is highly restricted because of a significant feature transmission overhead. To resolve this issue, in this study, we propose a reconfigurable DNN architecture for real-time object detection that can configure the optimal split point according to the edge–cloud CI environment. The proposed architecture allows the DNN model to be splittable by a feature reconstruction network and asymmetric scaling. Based on the splittable architecture, we integrate independent splittable models for each split point into a single-weight reconfigurable model that enables multipath inference by switchable quantization and distribution matching. Finally, we introduce an adaptive application procedure of the reconfigurable model for efficient CI, which includes asymmetric scale configuration and split point selection. The performance evaluation using YOLOv5 as the baseline showed that the proposed architecture achieved 30 frames/s ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.6\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.6\times $ </tex-math></inline-formula> higher than edge-only and cloud-only inference, respectively), on the NVIDIA Jetson TX2 platform in a WiFi environment.

SmartVM: A Smart Contract Virtual Machine for Fast On-Chain DNN Computations

Blockchain-based artificial intelligence (BC-AI) has been applied for protecting deep neural network (DNN) data from being tampered with, which is expected to further boost trusted distributed AI applications in many fields. However, due to smart contract execution environment architectural defects, it is challenging for previous BC-AI systems to support computing-intensive tasks on-chain performing such as DNN convolution operations. They have to offload computations and a large amount of data from blockchain to off-chain platforms to execute smart contracts as native code. This failure to take advantage of data locality has become one of the major critical performance bottlenecks in BC-AI system. To this end, in this article, we propose SmartVM with optimization methods to support on-chain DNN inference for BC-AI system. The key idea is to design and optimize the computing mechanism and storage structure of smart contract execution environment according to the characteristics of DNN such as high computational parallelism and large data volume. We decompose SmartVM into three components: 1) a compact DNN-oriented instruction set to describe computations in a short number of instructions to reduce interpretation time. 2) a memory management mechanism to make SmartVM memory dynamic free/allocated according to the size of DNN feature maps. 3) a block-based weight prefetching and parallel computing method to organize each layer's computing and weights prefetching in a pipelined manner. We perform the typical image classification in a private Ethereum blockchain testbed to evaluate SmartVM performance. Experimental results highlight that SmartVM can support DNN inference on-chain with roughly the same efficiency against the native code execution. Compared with the traditional off-chain computing, SmartVM can speed up the overall execution by <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">70×</b> , <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">16×</b> , <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">11×</b> , and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">12×</b> over LeNet5, AlexNet, ResNet18, and MobileNet, respectively. The memory footprint can be reduced by <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">84%</b> , <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">90.8%</b> , <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">94.3%</b> , and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">93.7%</b> over the above four models, while offering the same level model accuracy. This article sheds light on the design space of the smart contract virtual machine for DNN computation and is promising to further boost BC-AI applications.

Server load and network-aware adaptive deep learning inference offloading for edge platforms

This work presents DIAMOND, a deep neural network computation offloading scheme consisting of a lightweight client-to-server latency profiling component combined with a server inference time estimation module to accurately assess the expected latency of a deep learning model inference. Latency predictions for both the network and server are comprehensively used to make dynamic (partial) model offloading decisions at the client in run-time. Compared to previous work, DIAMOND targets to minimize network latency estimation overhead and considers the concurrent processing nature of state-of-the-art deep learning inference server designs. Our extensive evaluations with an NVIDIA Jetson Nano client connected to an NVIDIA Triton server shows that DIAMOND completes inference operations with noticeably reduced computational/energy overhead and latency compared to previously proposed model offloading approaches. Furthermore, our results show that DIAMOND well-adapts to practical server load and network dynamics.

An adaptive DNN inference acceleration framework with end–edge–cloud collaborative computing

Deep Neural Networks (DNNs) based on intelligent applications have been intensively deployed on mobile devices. Unfortunately, resource-constrained mobile devices cannot meet stringent latency requirements due to a large amount of computation required by these intelligent applications. Both exiting cloud-assisted DNN inference approaches and edge-assisted DNN inference approaches can reduce end-to-end inference latency through offloading DNN computations to the cloud server or edge servers, but they suffer from unpredictable communication latency caused by long wide-area massive data transmission or performance degeneration caused by the limited computation resources. In this paper, we propose an adaptive DNN inference acceleration framework, which accelerates DNN inference by fully utilizing the end–edge–cloud collaborative computing. First, a latency prediction model is built to estimate the layer-wise execution latency of a DNN on different heterogeneous computing platforms, which use neural networks to learn non-linear features related to inference latency. Second, a computation partitioning algorithm is designed to identify two optimal partitioning points, which adaptively divide DNN computations into end devices, edge servers, and the cloud server for minimizing DNN inference latency. Finally, we conduct extensive experiments on three widely-adopted DNNs, and the experimental results show that our latency prediction models can improve the prediction accuracy by about 72.31% on average compared with four baseline approaches, and our computation partitioning approach can reduce the end-to-end latency by about 20.81% on average against six baseline approaches under three wireless networks.