Resource-constrained Edge Devices Research Articles

Self-aware distributed deep learning framework for heterogeneous IoT edge devices

Implementing artificial intelligence (AI) in the Internet of Things (IoT) involves a move from the cloud to the heterogeneous and low-power edge, following an urgent demand for deploying complex training tasks in a distributed and reliable manner. This work proposes a self-aware distributed deep learning (DDL) framework for IoT applications, which is applicable to heterogeneous edge devices aiming to improve adaptivity and amortize the training cost. The self-aware design including the dynamic self-organizing approach and the self-healing method enhances the system reliability and resilience. Three typical edge devices are adopted with cross-platform Docker deployment: Personal Computers (PC) for general computing devices, Raspberry Pi 4Bs (Rpi) for resource-constrained edge devices, and Jetson Nanos (Jts) for AI-enabled edge devices. Benchmarked with ResNet-32 on CIFAR-10, the training efficiency of tested distributed clusters is increased by 8.44× compared to the standalone Rpi. The cluster with 11 heterogeneous edge devices achieves a training efficiency of 200.4 images/s and an accuracy of 92.45%. Results prove that the self-organizing approach functions well with dynamic changes like devices being removed or added. The self-healing method is evaluated with various stabilities, cluster scales, and breakdown cases, testifying that the reliability can be largely enhanced for extensively distributed deployments. The proposed DDL framework shows excellent performance for training implementation with heterogeneous edge devices in IoT applications with high-degree scalability and reliability.

BlindTrust: Oblivious Remote Attestation for Secure Service Function Chains

With the rapidly evolving next-generation systems-of-systems, we face new security, resilience, and operational assurance challenges. In the face of the increasing attack landscape, it is necessary to cater to efficient mechanisms to verify software and device integrity to detect run-time modifications. Towards this direction, remote attestation is a promising defense mechanism that allows a third party, the verifier, to ensure a remote device's (the prover's) integrity. However, many of the existing families of attestation solutions have strong assumptions on the verifying entity's trustworthiness, thus not allowing for privacy preserving integrity correctness. Furthermore, they suffer from scalability and efficiency issues. This paper presents a lightweight dynamic configuration integrity verification that enables inter and intra-device attestation without disclosing any configuration information and can be applied on both resource-constrained edge devices and cloud services. Our goal is to enhance run-time software integrity and trustworthiness with a scalable solution eliminating the need for federated infrastructure trust.

Living on the Edge

Voluminous time-series data streams produced in continuous sensing environments impose challenges pertaining to ingestion, storage, and analytics. In this study, we present a holistic approach based on data sketching to address these issues. We propose a hyper-sketching algorithm that combines discretization and frequency-based sketching to produce compact representations of the multi-feature, time-series data streams. We generate an ensemble of data sketches to make effective use of capabilities at the resource-constrained edge devices, the links over which data are transmitted, and the server pool where this data must be stored. The data sketches can be queried to construct datasets that are amenable to processing using popular analytical engines. We include several performance benchmarks using real-world data from different domains to profile the suitability of our design decisions. The proposed methodology can achieve up to ∼ 13 × and ∼ 2, 207 × reduction in data transfer and energy consumption at edge devices. We observe up to a ∼ 50% improvement in analytical job completion times in addition to the significant improvements in disk and network I/O.

Channel pruning guided by spatial and channel attention for DNNs in intelligent edge computing

Deep Neural Networks (DNNs) have achieved remarkable success in many computer vision tasks recently, but the huge number of parameters and the high computation overhead hinder their deployments on resource-constrained edge devices. It is worth noting that channel pruning is an effective approach for compressing DNN models. A critical challenge is to determine which channels are to be removed, so that the model accuracy will not be negatively affected. In this paper, we first propose Spatial and Channel Attention (SCA), a new attention module combining both spatial and channel attention that respectively focuses on “where” and “what” are the most informative parts. Guided by the scale values generated by SCA for measuring channel importance, we further propose a new channel pruning approach called Channel Pruning guided by Spatial and Channel Attention (CPSCA). Experimental results indicate that SCA achieves the best inference accuracy, while incurring negligibly extra resource consumption, compared to other state-of-the-art attention modules. Our evaluation on two benchmark datasets shows that, with the guidance of SCA, our CPSCA approach achieves higher inference accuracy than other state-of-the-art pruning methods under the same pruning ratios.

Area and power efficient post-quantum cryptosystem for IoT resource-constrained devices

Internet of Things (IoT) connects a myriad of small devices over a huge network, encompassing many different and varied applications and environments. As the IoT network continues to grow, providing end-to-end security over IoT is becoming a paramount issue. To mitigate existing and future security risks within IoT, two important factors should be considered. First, some resource-constrained edge devices have an insufficient area to contain the security part. Second, the advent of quantum computers threatens the security of current public-key cryptography algorithms. In response to these challenges, lattice-based cryptography (LBC) has emerged as a promising technique for IoT security in the quantum era. The feasibility of LBC integration onto resource-constrained devices has been demonstrated in previous research. Multiplication is the main operation in Ring-BinLWE, a type of LBC. In this paper, a new multiplication method is proposed, which is called In-place modular Reduction and anti-circular Rotation Column-based Multiplication (In-place Rot-Col-Mul), and new Ring-BinLWE architecture is designed. In-place Rot-Col-Mul performs a column-based multiplication in which one rotation is executed per cycle. The design was implemented on TSMC-65nm technology and FPGA platforms. ASIC implementation results show a respective improvement in power and area over the state-of-the-art design by 48.42% and 57.8%, respectively.

Stacked Autoencoder-Based Probabilistic Feature Extraction for On-Device Network Intrusion Detection

Due to the outbreak of recent network attacks, it is necessary to develop a robust network intrusion detection system (NIDS) that can quickly and effectively identify the network attack. Although the state-of-the-art detection algorithms have shown quite promising detection performance, they suffer from computationally intensive operations and large memory footprint, making themselves infeasible to applications at the resourceconstrained edge devices. We propose a lightweight yet effective NIDS scheme that incorporates a stacked autoencoder with a network pruning technique. By removing a set of ineffective neurons across layers in the autoencoder network with a certain probability based on their importance, a considerably large portion of relatively nominal training parameters are reduced. Then, the pruned and pretrained encoder network is used as-is and is connected with a separate classifier network for attack type inference, avoiding a full retraining from scratch. Experimental results indicate that our stacked autoencoder-based classification network with probabilistic feature extraction has outperformed the state-of-the-art NIDSs in terms of attack detection rate. Further, we have shown that our lightweight NIDS scheme has significantly reduced the computational complexity throughout the architecture, making it feasible to the edge, while maintaining a similar attack type detection quality compared with its original fully connected neural network.

A Heterogeneous Hardware Accelerator for Image Classification in Embedded Systems

Convolutional neural networks (CNN) have been extensively employed for image classification due to their high accuracy. However, inference is a computationally-intensive process that often requires hardware acceleration to operate in real time. For mobile devices, the power consumption of graphics processors (GPUs) is frequently prohibitive, and field-programmable gate arrays (FPGA) become a solution to perform inference at high speed. Although previous works have implemented CNN inference on FPGAs, their high utilization of on-chip memory and arithmetic resources complicate their application on resource-constrained edge devices. In this paper, we present a scalable, low power, low resource-utilization accelerator architecture for inference on the MobileNet V2 CNN. The architecture uses a heterogeneous system with an embedded processor as the main controller, external memory to store network data, and dedicated hardware implemented on reconfigurable logic with a scalable number of processing elements (PE). Implemented on a XCZU7EV FPGA running at 200 MHz and using four PEs, the accelerator infers with 87% top-5 accuracy and processes an image of pixels in 220 ms. It consumes 7.35 W of power and uses less than 30% of the logic and arithmetic resources used by other MobileNet FPGA accelerators.

Privacy-Aware Fuzzy Range Query Processing Over Distributed Edge Devices

Range query processing is a common edge computing and service in the Internet of things, which can extract user-interest information from distributed edge devices. How to design lightweight privacy-preserving range query processing methods remains a challenging task. Existing secure range query approaches suffer from both high communication cost and long response time, which makes them unsuitable for edge computing over resource-constrained edge devices. In this article, we propose two privacy-aware fuzzy query processing schemes based on fuzzy theory. Linguistic range variables, fuzzy overlap information, and its recovery mechanism are introduced. In addition, two distributed privacy-aware fuzzy range query processing algorithms are devised. Our approaches not only serve for privacy protection, but also aim to provide other optimal performances in terms of reliability, energy efficiency, and real-time response. Theoretical analysis and experimental evaluations based on real-world datasets validated our motivation.

Radio Frequency Fingerprinting on the Edge

Deep learning methods have been very successful at radio frequency fingerprinting tasks, predicting the identity of transmitting devices with high accuracy. We study radio frequency fingerprinting deployments at resource-constrained edge devices. We use structured pruning to jointly train and sparsify neural networks tailored to edge hardware implementations. We compress convolutional layers by a <inline-formula><tex-math notation="LaTeX">$27.2\times$</tex-math></inline-formula> factor while incurring a negligible prediction accuracy decrease (less than 1 percent). We demonstrate the efficacy of our approach over multiple edge hardware platforms, including a Samsung Gallaxy S10 phone and a Xilinx-ZCU104 FPGA. Our method yields significant inference speedups, <inline-formula><tex-math notation="LaTeX">$11.5\times$</tex-math></inline-formula> on the FPGA and <inline-formula><tex-math notation="LaTeX">$3\times$</tex-math></inline-formula> on the smartphone, as well as high efficiency: the FPGA processing time is <inline-formula><tex-math notation="LaTeX">$17\times$</tex-math></inline-formula> smaller than in a V100 GPU. To the best of our knowledge, we are the first to explore the possibility of compressing networks for radio frequency fingerprinting; as such, our experiments can be seen as a means of characterizing the informational capacity associated with this specific learning task.

Optimising Deep Learning at the Edge for Accurate Hourly Air Quality Prediction.

Accurate air quality monitoring requires processing of multi-dimensional, multi-location sensor data, which has previously been considered in centralised machine learning models. These are often unsuitable for resource-constrained edge devices. In this article, we address this challenge by: (1) designing a novel hybrid deep learning model for hourly PM2.5 pollutant prediction; (2) optimising the obtained model for edge devices; and (3) examining model performance running on the edge devices in terms of both accuracy and latency. The hybrid deep learning model in this work comprises a 1D Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) to predict hourly PM2.5 concentration. The results show that our proposed model outperforms other deep learning models, evaluated by calculating RMSE and MAE errors. The proposed model was optimised for edge devices, the Raspberry Pi 3 Model B+ (RPi3B+) and Raspberry Pi 4 Model B (RPi4B). This optimised model reduced file size to a quarter of the original, with further size reduction achieved by implementing different post-training quantisation. In total, 8272 hourly samples were continuously fed to the edge device, with the RPi4B executing the model twice as fast as the RPi3B+ in all quantisation modes. Full-integer quantisation produced the lowest execution time, with latencies of 2.19 s and 4.73 s for RPi4B and RPi3B+, respectively.

Resource-Constrained Neural Architecture Search on Edge Devices

The performance requirement of deep learning inevitably brings up with the expense of high computational complexity and memory requirements, to make it problematic for the deployment on resource-constrained devices. Edge computing, which distributedly organizes the computing node close to the data source and end-device, provides a feasible way to tackle the high-efficiency demand and substantial computational load. Whereas given edge device is resource-constrained and energy-sensitive, designing effective neural network architecture for specific edge device is urgent in the sense that deploys the deep learning application by the edge computing solution. Undoubtedly manually design the high-performing neural architectures is burdensome, let alone taking account of the resource-constraint for the specific platform. Fortunately, the success of Neural Architecture Search techniques come up with hope recently. This paper dedicates to directly employ multi-objective NAS on the resource-constrained edge devices. We first propose the framework of multi-objective NAS on edge device, which comprehensively considers the performance and real-world efficiency. Our improved MobileNet-V2 search space also strikes the scalability and practicality, so that a series of Pareto-optimal architectures are received. Benefits from the directness and specialization during search procedure, our experiment on JETSON NANO shows the comparable result with the state-of-the-art models on ImageNet.

Reconfigurable Binary Neural Network Accelerator with Adaptive Parallelism Scheme

Binary neural networks (BNNs) have attracted significant interest for the implementation of deep neural networks (DNNs) on resource-constrained edge devices, and various BNN accelerator architectures have been proposed to achieve higher efficiency. BNN accelerators can be divided into two categories: streaming and layer accelerators. Although streaming accelerators designed for a specific BNN network topology provide high throughput, they are infeasible for various sensor applications in edge AI because of their complexity and inflexibility. In contrast, layer accelerators with reasonable resources can support various network topologies, but they operate with the same parallelism for all the layers of the BNN, which degrades throughput performance at certain layers. To overcome this problem, we propose a BNN accelerator with adaptive parallelism that offers high throughput performance in all layers. The proposed accelerator analyzes target layer parameters and operates with optimal parallelism using reasonable resources. In addition, this architecture is able to fully compute all types of BNN layers thanks to its reconfigurability, and it can achieve a higher area–speed efficiency than existing accelerators. In performance evaluation using state-of-the-art BNN topologies, the designed BNN accelerator achieved an area–speed efficiency 9.69 times higher than previous FPGA implementations and 24% higher than existing VLSI implementations for BNNs.

Arithmetic Coding-Based 5-Bit Weight Encoding and Hardware Decoder for CNN Inference in Edge Devices

Convolutional neural networks (CNNs) have gained a huge attention for real-world artificial intelligence (AI) applications such as image classification and object detection. On the other hand, for better accuracy, the size of the CNNs’ parameters (weights) has been increasing, which in turn makes it difficult to enable on-device CNN inferences in resource-constrained edge devices. Though weight pruning and 5-bit quantization methods have shown promising results, it is still challenging to deploy large CNN models in edge devices. In this paper, we propose an encoding and hardware-based decoding technique which can be applied to 5-bit quantized weight data for on-device CNN inferences in resource-constrained edge devices. Given 5-bit quantized weight data, we employ arithmetic coding with range scaling for lossless weight compression, which is performed offline. When executing on-device inferences with underlying CNN accelerators, our hardware decoder enables a fast in-situ weight decompression with small latency overhead. According to our evaluation results with five widely used CNN models, our arithmetic coding-based encoding method applied to 5-bit quantized weights shows a better compression ratio by 9.6× while also reducing the memory data transfer energy consumption by 89.2%, on average, as compared to the case of uncompressed 32-bit floating-point weights. When applying our technique to pruned weights, we obtain better compression ratios by 57.5×–112.2× while reducing energy consumption by 98.3%–99.1% as compared to the case of 32-bit floating-point weights. In addition, by pipelining the weight decoding and transfer with the CNN execution, the latency overhead of our weight decoding with 16 decoding unit (DU) hardware is only 0.16%–5.48% and 0.16%–0.91% for non-pruned and pruned weights, respectively. Moreover, our proposed technique with 4-DU decoder hardware reduces system-level energy consumption by 1.1%–9.3%.

Hardware-Aware Design for Edge Intelligence

With the rapid growth of the number of devices connected to the Internet, there is a trend to move intelligent processing of the generated data with deep neural networks (DNNs) from cloud servers to the network edge. Performing inference and training of DNNs in edge hardware is motivated by latency constraints, security and privacy concerns, and restricted network bandwidth. However, implementation of DNNs is challenging in resource-constrained edge devices. This article surveys recent advances in the efficient processing of DNNs, highlighting present research trends and future challenges. Specifically, we start by reviewing optimization methods for hardware-aware deployment of DNNs. We then present some case studies of promising new directions towards low-complexity on-chip training. Finally, we discuss future challenges and their potential solutions for efficient deployment of DNNs at the edge.

FogFL: Fog-Assisted Federated Learning for Resource-Constrained IoT Devices

In this article, we propose a fog-enabled federated learning framework—FogFL—to facilitate distributed learning for delay-sensitive applications in resource-constrained IoT environments. While federated learning (FL) is a popular distributed learning approach, it suffers from communication overheads and high computational requirements. Moreover, global aggregation in FL relies on a centralized server, prone to malicious attacks, resulting in inefficient training models. We address these issues by introducing geospatially placed fog nodes into the FL framework as local aggregators. These fog nodes are responsible for defined demographics, which help share location-based information for applications with similar environments. Furthermore, we formulate a greedy heuristic approach for selecting an optimal fog node for assuming a global aggregator’s role at each round of communication between the edge and cloud, thereby reducing the dependence on the execution at the centralized server. Fog nodes in the FogFL framework reduce communication latency and energy consumption of resource-constrained edge devices without affecting the global model’s convergence rate, thereby increasing the system’s reliability. Extensive deployment and experimental results corroborate that, in addition to a decrease in global aggregation rounds, FogFL reduces energy consumption and communication latency by 92% and 85%, respectively, as compared to state of the art.

CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices

Recent advances in artificial intelligence have driven increasing intelligent applications at the network edge, such as smart home, smart factory, and smart city. To deploy computationally intensive Deep Neural Networks (DNNs) on resource-constrained edge devices, traditional approaches have relied on either offloading workload to the remote cloud or optimizing computation at the end device locally. However, the cloud-assisted approaches suffer from the unreliable and delay-significant wide-area network, and the local computing approaches are limited by the constrained computing capability. Towards high-performance edge intelligence, the cooperative execution mechanism offers a new paradigm, which has attracted growing research interest recently. In this paper, we propose CoEdge, a distributed DNN computing system that orchestrates cooperative DNN inference over heterogeneous edge devices. CoEdge utilizes available computation and communication resources at the edge and dynamically partitions the DNN inference workload adaptive to devices' computing capabilities and network conditions. Experimental evaluations based on a realistic prototype show that CoEdge outperforms status-quo approaches in saving energy with close inference latency, achieving up to 25.5% ~ 66.9% energy reduction for four widely-adopted CNN models.

Real-time and accurate object detection on edge device with TensorFlow Lite

Objection detection is of vital importance to many fields, such as autonomous driving, outdoor robotics, and computer vision. Existing approaches on object detection can hardly run on the resource-constrained edge devices. In order to mitigate this dilemma, we propose to apply TensorFlow Lite to convert Float32 neural network model to unit8 neural network with subtle or even no accuracy loss. Two advantages are here for conversion. First, it reduces the model size to a quarter so that it fits for devices with limited storage. Second, it achieves much faster inference time. I conduct an experiment on MSCOCO dataset. Experimental results show that our proposed method achieves mAP 72.1 and FPS 23 on edge device.

Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices

The increasing computational cost of deep neural network models limits the applicability of intelligent applications on resource-constrained edge devices. While a number of neural network pruning methods have been proposed to compress the models, prevailing approaches focus only on parametric operators (e.g., convolution), which may miss optimization opportunities. In this paper, we present a novel fusion-catalyzed pruning approach, called FuPruner, which simultaneously optimizes the parametric and non-parametric operators for accelerating neural networks. We introduce an aggressive fusion method to equivalently transform a model, which extends the optimization space of pruning and enables non-parametric operators to be pruned in a similar manner as parametric operators, and a dynamic filter pruning method is applied to decrease the computational cost of models while retaining the accuracy requirement. Moreover, FuPruner provides configurable optimization options for controlling fusion and pruning, allowing much more flexible performance-accuracy trade-offs to be made. Evaluation with state-of-the-art residual neural networks on five representative intelligent edge platforms, Jetson TX2, Jetson Nano, Edge TPU, NCS, and NCS2, demonstrates the effectiveness of our approach, which can accelerate the inference of models on CIFAR-10 and ImageNet datasets.

Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks.

The deployment of machine learning algorithms on resource-constrained edge devices is an important challenge from both theoretical and applied points of view. In this brief, we focus on resource-efficient randomly connected neural networks known as random vector functional link (RVFL) networks since their simple design and extremely fast training time make them very attractive for solving many applied classification tasks. We propose to represent input features via the density-based encoding known in the area of stochastic computing and use the operations of binding and bundling from the area of hyperdimensional computing for obtaining the activations of the hidden neurons. Using a collection of 121 real-world data sets from the UCI machine learning repository, we empirically show that the proposed approach demonstrates higher average accuracy than the conventional RVFL. We also demonstrate that it is possible to represent the readout matrix using only integers in a limited range with minimal loss in the accuracy. In this case, the proposed approach operates only on small n -bits integers, which results in a computationally efficient architecture. Finally, through hardware field-programmable gate array (FPGA) implementations, we show that such an approach consumes approximately 11 times less energy than that of the conventional RVFL.

On-Edge Multi-Task Transfer Learning: Model and Practice With Data-Driven Task Allocation

On edge devices, data scarcity occurs as a common problem where transfer learning serves as a widely-suggested remedy. Nevertheless, transfer learning imposes a heavy computation burden to resource-constrained edge devices. Existing task allocation works usually assume all submitted tasks are equally important, leading to inefficient resource allocation at a task level when directly applied in Multi-task Transfer Learning (MTL). To address these issues, we first reveal that it is crucial to measure the impact of tasks on overall decision performance improvement and quantify \emph{task importance}. We then show that task allocation with task importance for MTL (TATIM) is a variant of the NP-complete Knapsack problem, where the complicated computation to solve this problem needs to be conducted repeatedly under varying contexts. To solve TATIM with high computational efficiency, we propose a Data-driven Cooperative Task Allocation (DCTA) approach. Finally, we evaluate the performance of DCTA by not only a trace-driven simulation, but also a new comprehensive real-world AIOps case study that bridges model and practice via a new architecture and main components design within the AIOps system. Extensive experiments show that our DCTA reduces 3.24 times of processing time, and saves 48.4\% energy consumption compared with the state-of-the-art when solving TATIM.