Edge Devices Research Articles

Multi-Agent Deep Reinforcement Learning-Based Inference Task Scheduling and Offloading for Maximum Inference Accuracy under Time and Energy Constraints

The journey towards realizing real-time AI-driven IoT applications is facing a significant hurdle caused by the limited resources of IoT devices. Particularly for battery-powered edge devices, the decision between performing task inference locally or by offloading to edge servers, all while ensuring timely results and conserving energy, is a critical issue. This problem is further complicated when an edge device houses multiple local inference models. The challenge of effectively allocating inference models to tasks between local models and edge server models under strict time and energy constraints while maximizing overall accuracy is recognized as a strongly NP-hard problem and has not been addressed in the literature. Therefore, in this work we propose MASITO, a novel multi-agent deep reinforcement learning framework designed to address this intricate problem. By dividing the problem into two sub-problems namely task scheduling and edge server selection we propose a cooperative multi-agent system for addressing each sub-problem. MASITO’s design allows for faster training and more robust schedules using cooperative behavior where agents compensate for each other’s sub-optimal actions. Moreover, MASITO dynamically adapts to different network configurations which allows for high-mobility edge computing applications. Experiments on the ImageNet-mini dataset demonstrate the framework’s efficacy, outperforming genetic algorithms (GAs), simulated annealing (SA), and particle swarm optimization (PSO) in scheduling times by providing lower times ranging from 60% up to 90% while maintaining comparable average accuracy in worst-case scenarios and superior accuracy in best-case scenarios.

An ECC based Secure Authentication Protocol for M2M Communication in Industrial IOT Edge Device

Abstract: Machine to machine communication in industrial IoT is becoming a common phenomenon now-a-days where all the machines like IoT edge devices communicate independently. It’s always been a challenging task to create a protocol for authentication that is both secure and efficient with regard to resources for use in industrial IoT environments' M2M communication. Here we present a safe and effective authentication method that makes use of elliptic curve cryptography (ECC). We demonstrated these vulnerabilities by testing the current protocol with various assaults, including SSL strip attack and MITM attack. To further evaluate the protocol's security against various IoT threats, we employed both official and informal approaches, such as the Scyther tool and BAN logic. Secured from a variety of IoT Threats, such as Man-in-the-middle (MITM) attacks, replay attacks, and impersonation attacks, our suggested protocol, absence of device anonymity, weak mutual authentication etc. but also suitable for resource constraint IoT edge devices like RFID devices, sensors, routers and gateways.

Server-Assisted Security Model for Edge Computing: A Packet Tracking Approach

Abstract: In current network systems, many applications run on edge computing devices, but these devices often lack safety and security measures to protect against hackers. To address this issue, we have developed a prototype model that enhances security. Our model connects the edge computing device to a main server via a socket connection. If the edge device detects any threats, it can alert the server immediately

YOLO Models for Security and Surveillance Applications

Abstract: Since 2015, the YOLO (You Only Look Once) series has evolved to YOLO-v8, prioritizing real-time processing and high accuracy for security and surveillance applications. Architectural enhancements in each iteration, culminating in YOLOv9, cater to rapid detection, precision, and adaptability to resource-constrained edge devices. This study examines YOLO’s evolution, emphasizing its relevance to security and surveillance contexts. Notable improvements in architecture, coupled with practical deployments for defect detection, underscore YOLO’s alignment with stringent security and surveillance requirements.

Lightweight Oriented Detector for Insulators in Drone Aerial Images

Due to long-term exposure to the wild, insulators are prone to various defects that affect the safe operation of the power system. In recent years, the combination of drones and deep learning has provided a more intelligent solution for insulator automatic defect inspection. Positioning insulators is an important prerequisite step for defect detection, and the accuracy of insulator positioning greatly affects defect detection. However, traditional horizontal detectors lose directional information and it is difficult to accurately locate tilted insulators. Although oriented detectors can predict detection boxes with rotation angles to solve this problem, these models are complex and difficult to apply to edge devices with limited computing power. This greatly limits the practical application of deep learning methods in insulator detection. To address these issues, we proposed a lightweight insulator oriented detector. First, we designed a lightweight insulator feature pyramid network (LIFPN). It can fuse features more efficiently while reducing the number of parameters. Second, we designed a more lightweight insulator oriented detection head (LIHead). It has less computational complexity and can predict rotated detection boxes. Third, we deployed the detector on edge devices and further improved its inference speed through TensorRT. Finally, a series of experiments demonstrated that our method could reduce the computational complexity of the detector by approximately 49 G and the number of parameters by approximately 30 M while ensuring almost no decrease in the detection accuracy. It can be easily deployed to edge devices and achieve a detection speed of 41.89 frames per second (FPS).

Motor PHM on Edge Computing with Anomaly Detection and Fault Severity Estimation through Compressed Data Using PCA and Autoencoder

The motor is essential for manufacturing industries, but wear can cause unexpected failure. Predictive and health management (PHM) for motors is critical in manufacturing sites. In particular, data-driven PHM using deep learning methods has gained popularity because it reduces the need for domain expertise. However, the massive amount of data poses challenges to traditional cloud-based PHM, making edge computing a promising solution. This study proposes a novel approach to motor PHM in edge devices. Our approach integrates principal component analysis (PCA) and an autoencoder (AE) encoder achieving effective data compression while preserving fault detection and severity estimation integrity. The compressed data is visualized using t-SNE, and its ability to retain information is assessed through clustering performance metrics. The proposed method is tested on a custom-made experimental platform dataset, demonstrating robustness across various fault scenarios and providing valuable insights for practical applications in manufacturing.

Hierarchical sharding blockchain storage solution for edge computing

The fusion of edge computing architecture and blockchain technology faces challenges such as heterogeneous edge device performance limitations and the inefficiency and high storage redundancy of blockchain systems, resulting in suboptimal fusion results. To address these issues, this paper proposes a layered and sharded blockchain storage model tailored for edge computing. The model adopts a master–slave structure for layered blockchain deployment, which means deploying the main chain in the cloud center to integrate the state information of multiple shard subchains, achieving consistency in the state of the master and slave chains, with each shard subchain responsible for storing data. Meanwhile, considering the performance variations and distribution characteristics of edge devices, this paper designs subchains in the edge chain layer by clustering edge devices with similar performance and proximity into the same type of shard using the concept of community discovery, ensuring balanced storage performance among nodes within the shard. On this basis, this paper also proposes a weight-based consistent hashing algorithm that is capable of distributing data to heterogeneous edge devices according to their weights. Simulation experiments demonstrate that the proposed solution offers advantages in terms of high throughput and low storage redundancy.

Researching the CNN Collaborative Inference Mechanism for Heterogeneous Edge Devices.

Convolutional Neural Networks (CNNs) have been widely applied in various edge computing devices based on intelligent sensors. However, due to the high computational demands of CNN tasks, the limited computing resources of edge intelligent terminal devices, and significant architectural differences among these devices, it is challenging for edge devices to independently execute inference tasks locally. Collaborative inference among edge terminal devices can effectively utilize idle computing and storage resources and optimize latency characteristics, thus significantly addressing the challenges posed by the computational intensity of CNNs. This paper targets efficient collaborative execution of CNN inference tasks among heterogeneous and resource-constrained edge terminal devices. We propose a pre-partitioning deployment method for CNNs based on critical operator layers, and optimize the system bottleneck latency during pipeline parallelism using data compression, queuing, and "micro-shifting" techniques. Experimental results demonstrate that our method achieves significant acceleration in CNN inference within heterogeneous environments, improving performance by 71.6% compared to existing popular frameworks.

Deep Neural Network-Based Flood Monitoring System Fusing RGB and LWIR Cameras for Embedded IoT Edge Devices

Floods are among the most common disasters, causing loss of life and enormous damage to private property and public infrastructure. Monitoring systems that detect and predict floods help respond quickly in the pre-disaster phase to prevent and mitigate flood risk and damages. Thus, this paper presents a deep neural network (DNN)-based real-time flood monitoring system for embedded Internet of Things (IoT) edge devices. The proposed system fuses long-wave infrared (LWIR) and RGB cameras to overcome a critical drawback of conventional RGB camera-based systems: severe performance deterioration at night. This system recognizes areas occupied by water using a DNN-based semantic segmentation network, whose input is a combination of RGB and LWIR images. Flood warning levels are predicted based on the water occupancy ratio calculated by the water segmentation result. The warning information is delivered to authorized personnel via a mobile message service. For real-time edge computing, the heavy semantic segmentation network is simplified by removing unimportant channels while maintaining performance by utilizing the network slimming technique. Experiments were conducted based on the dataset acquired from the sensor module with RGB and LWIR cameras installed in a flood-prone area. The results revealed that the proposed system successfully conducts water segmentation and correctly sends flood warning messages in both daytime and nighttime. Furthermore, all of the algorithms in this system were embedded on an embedded IoT edge device with a Qualcomm QCS610 System on Chip (SoC) and operated in real time.

PackMASNet: An information integration approach for quality inspection in industry 5.0

Industry 5.0 (I5.0) is increasingly governed by Operation Technology and Information Technology. It hosts Deep Learning (DL) solutions as a premier tool for the informatization of Quality Inspection (QI) of Mass Produced Products (MPP) and Mass Customized Products (MCP) at the on-premise edge devices. Besides, these ever-evolving MCPs in I5.0 require newer DL models to informatize customized quality parameters, mandating significant storage and computing resources, that is challenging to achieve at resource-constraint edge platform. This work proposes PackMASNet, a continual learning-based information integration scheme that sustainably incorporates the QI information of multiple MCPs in a single Deep Neural Network (DNN) under the following constraints: (i) resource-limited on-premise edge devices, (ii) no access to prior product data, and (iii) negligible deterioration of quality information of frequently manufactured MPPs. A lightweight Cosine-similarity threshold mechanism is incorporated to leverage the benefits of knowledge transfer (cloning) while avoiding its adverse impact in multi-varied MCPs. The scheme proposes DNN performance, storage, computation, data, and time efficient solutions for industries implementing the edge computing paradigm on I5.0. The efficacy of the proposed scheme is validated for the predictive QI of real-world use case in injection molding. The scheme preserves the frequently used information of MPP by 99.07%, saves computation cost by 132 times, reduces the DNN training time by 20.45%, and improves overall performance by 43.60% over the state-of-the-art schemes.

Proposal of an open-source accelerators library for inference of transformer networks in edge devices based on Linux

Proposal of an open-source accelerators library for inference of transformer networks in edge devices based on Linux

Federated Learning-Oriented Edge Computing Framework for the IIoT.

With the maturity of artificial intelligence (AI) technology, applications of AI in edge computing will greatly promote the development of industrial technology. However, the existing studies on the edge computing framework for the Industrial Internet of Things (IIoT) still face several challenges, such as deep hardware and software coupling, diverse protocols, difficult deployment of AI models, insufficient computing capabilities of edge devices, and sensitivity to delay and energy consumption. To solve the above problems, this paper proposes a software-defined AI-oriented three-layer IIoT edge computing framework and presents the design and implementation of an AI-oriented edge computing system, aiming to support device access, enable the acceptance and deployment of AI models from the cloud, and allow the whole process from data acquisition to model training to be completed at the edge. In addition, this paper proposes a time series-based method for device selection and computation offloading in the federated learning process, which selectively offloads the tasks of inefficient nodes to the edge computing center to reduce the training delay and energy consumption. Finally, experiments carried out to verify the feasibility and effectiveness of the proposed method are reported. The model training time with the proposed method is generally 30% to 50% less than that with the random device selection method, and the training energy consumption under the proposed method is generally 35% to 55% less.

Semantic-embedded similarity prototype for scene recognition

Due to the high inter-class similarity caused by the complex composition and the co-existing objects across scenes, numerous studies have explored object semantic knowledge within scenes to improve scene recognition. However, a resulting challenge emerges as object information extraction techniques require heavy computational costs, thereby burdening the network considerably. This limitation often renders object-assisted approaches incompatible with edge devices in practical deployment. In contrast, this paper proposes a semantic knowledge-based similarity prototype, which can help the scene recognition network achieve superior accuracy without increasing the computational cost in practice. It is simple and can be plug-and-played into existing pipelines. More specifically, a statistical strategy is introduced to depict semantic knowledge in scenes as class-level semantic representations. These representations are used to explore correlations between scene classes, ultimately constructing a similarity prototype. Furthermore, we propose to leverage the similarity prototype to support network training from the perspective of Gradient Label Softening and Batch-level Contrastive Loss, respectively. Comprehensive evaluations on multiple benchmarks show that our similarity prototype enhances the performance of existing networks, all while avoiding any additional computational burden in practical deployments. Code and the statistical similarity prototype will be available at https://github.com/ChuanxinSong/SimilarityPrototype.

A Faster and Lightweight Lane Detection Method in Complex Scenarios

Lane detection is a crucial visual perception task in the field of autonomous driving, serving as one of the core modules in advanced driver assistance systems (ADASs).To address the insufficient real-time performance of current segmentation-based models and the conflict between the demand for high inference speed and the excessive parameters in resource-constrained edge devices (such as onboard hardware, mobile terminals, etc.) in complex real-world scenarios, this paper proposes an efficient and lightweight auxiliary branch network (CBGA-Auxiliary) to tackle these issues. Firstly, to enhance the model’s capability to extract feature information in complex scenarios, a row anchor-based feature extraction method based on global features was adopted. Secondly, employing ResNet as the backbone network and CBGA (Conv-Bn-GELU-SE Attention) as the fundamental module, we formed the auxiliary segmentation network, significantly enhancing the segmentation training speed of the model. Additionally, we replaced the standard convolutions in the branch network with lightweight GhostConv convolutions. This reduced the parameters and computational complexity while maintaining accuracy. Finally, an additional enhanced structural loss function was introduced to compensate for the structural defect loss issue inherent in the row anchor-based method, further improving the detection accuracy. The model underwent extensive experimentation on the Tusimple dataset and the CULane dataset, which encompass various road scenarios. The experimental results indicate that the model achieved the highest F1 scores of 96.1% and 71.0% on the Tusimple and CULane datasets, respectively. At a resolution of 288 × 800, the ResNet18 and ResNet34 models achieved maximum inference speeds of 410FPS and 280FPS, respectively. Compared to existing SOTA models, it demonstrates a significant advantage in terms of inference speed. The model achieved a good balance between accuracy and inference speed, making it suitable for deployment on edge devices and validates the effectiveness of the model.

Robust knowledge distillation based on feature variance against backdoored teacher model

Benefiting from large well-trained deep neural networks (DNNs), model compression has captured special attention for computing resource limited equipment, especially edge devices. Knowledge distillation (KD) is one of the widely used compression techniques for edge deployment, by obtaining a lightweight student model from a well-trained teacher model released on public platforms. However, it has been empirically noticed that the backdoor in the teacher model will be transferred to the student model during the process of KD. Although numerous KD methods have been proposed, most of them focus on the distillation of a high-performing student model without robustness consideration. Besides, some research adopts KD techniques as effective backdoor mitigation tools, but they fail to perform model compression at the same time. Consequently, it is still an open problem to well achieve two objectives of robust KD, i.e., student model’s performance and backdoor mitigation. To address these issues, we propose RobustKD, a robust knowledge distillation that compresses the model while mitigating backdoor based on feature variance. Specifically, RobustKD distinguishes the previous works in three key aspects: (1) effectiveness - by distilling the feature map of the teacher model after detoxification, the main task performance of the student model is comparable to that of the teacher model; (2) robustness - by reducing the characteristic variance between the teacher model and the student model, it mitigates the backdoor of the student model under backdoored teacher model scenario; (3) generic - RobustKD still has good performance in the face of multiple data models (e.g., WRN 28-4, Pyramid-200) and diverse DNNs (e.g., ResNet50, MobileNet). Comprehensive experiments are conducted on four datasets, six models, two distillation methods, and two backdoor attack methods, compared with four baselines, and the results verified that the proposed method achieves the state-of-the-art performance in both aspects of accuracy and robustness. In addition, RobustKD is still effective when adaptive attacks are considered. The code of RobustKD is open-sourced at https://github.com/Xming-Z/RobustKD.

A Forest Fire Smoke Monitoring System Based on a Lightweight Neural Network for Edge Devices

Forest resources are one of the indispensable resources of the earth, which are the basis for the survival and development of human society. With the swift advancements in computer vision and artificial intelligence technology, the utilization of deep learning for smoke detection has achieved remarkable results. However, the existing deep learning models have poor performance in forest scenes and are difficult to deploy because of numerous parameters. Hence, we introduce an optimized forest fire smoke monitoring system for embedded edge devices based on a lightweight deep learning model. The model makes full use of the multi-scale variable attention mechanism of Transformer architecture to strengthen the ability of image feature extraction. Considering the needs of application scenarios, we propose an improved lightweight network model LCNet for feature extraction, which can reduce the parameters and enhance detecting ability. In order to improve running speed, a simple semi-supervised label knowledge distillation scheme is used to enhance the overall detection capability. Finally, we design and implement a forest fire smoke detection system on an embedded device, including the Jetson NX hardware platform, high-definition camera, and detection software system. The lightweight model is transplanted to the embedded edge device to achieve rapid forest fire smoke detection. Also, an asynchronous processing framework is designed to make the system highly available and robust. The improved model reduces three-fourths of the parameters and increases speed by 3.4 times with similar accuracy to the original model. This demonstrates that our system meets the precision demand and detects smoke in time.

An Efficient Lightweight Authentication and Access Control for IoT Edge Devices

An Efficient Lightweight Authentication and Access Control for IoT Edge Devices

Machine learning workflow for edge computed arrhythmia detection in exploration class missions

Deep-space missions require preventative care methods based on predictive models for identifying in-space pathologies. Deploying such models requires flexible edge computing, which Open Neural Network Exchange (ONNX) formats enable by optimizing inference directly on wearable edge devices. This work demonstrates an innovative approach to point-of-care machine learning model pipelines by combining this capacity with an advanced self-optimizing training scheme to classify periods of Normal Sinus Rhythm (NSR), Atrial Fibrillation (AFIB), and Atrial Flutter (AFL). 742 h of electrocardiogram (ECG) recordings were pre-processed into 30-second normalized samples where variable mode decomposition purged muscle artifacts and instrumentation noise. Seventeen heart rate variability and morphological ECG features were extracted by convoluting peak detection with Gaussian distributions and delineating QRS complexes using discrete wavelet transforms. The decision tree classifier’s features, parameters, and hyperparameters were self-optimized through stratified triple nested cross-validation ranked on F1-scoring against cardiologist labeling. The selected model achieved a macro F1-score of 0.899 with 0.993 for NSR, 0.938 for AFIB, and 0.767 for AFL. The most important features included median P-wave amplitudes, PRR20, and mean heart rates. The ONNX-translated pipeline took 9.2 s/sample. This combination of our self-optimizing scheme and deployment use case of ONNX demonstrated overall accurate operational tachycardia detection.

STAFFormer: Spatio-temporal adaptive fusion transformer for efficient 3D human pose estimation

Existing two-stage methods for 3D Human Pose Estimation often use 2D poses as input, which are then lifted to obtain 3D representations. This typically involves frame-by-frame estimation, whether in 2D or 3D, resulting in high computational demands unsuitable for edge devices. Due to the continuity of human movement, the differences between adjacent frames can be minimal, a question arises: is frame-by-frame estimation necessary? Previous works demonstrated the feasibility of using Transformer-based models to estimate poses with sparse frames, focusing on either temporal or spatial dependencies but neglecting holistic spatio-temporal correlations. To address this, we introduce the Spatio-Temporal Adaptive Fusion Transformer (STAFFormer). First, STAFFormer recovers dense temporal frames from sparsely sampled ones obtained from a 2D pose estimator through the Temporal Dense Frame Recovery (TDFR) module. This significantly reduces the computational complexity. Second, STAFFormer employs an adaptive fusion attention mechanism, enhancing accuracy by attentively navigating both spatial and temporal dimensions through the Spatio-Temporal Adaptive Fusion (STAF) module. Furthermore, We introduce a kinematic coherence loss to adapt to subtle joint movements, improving pose estimation fidelity. Finally, we explore the possibility of integrating different pre-training strategies using extensive marker-based datasets. Experimental results on challenging datasets show our network achieves state-of-the-art performance with low computational complexity.

Lightweight and Efficient Tiny-Object Detection Based on Improved YOLOv8n for UAV Aerial Images

The task of multiple-tiny-object detection from diverse perspectives in unmanned aerial vehicles (UAVs) using onboard edge devices is a significant and complex challenge within computer vision. In order to address this challenge, we propose a lightweight and efficient tiny-object-detection algorithm named LE-YOLO, based on the YOLOv8n architecture. To improve the detection performance and optimize the model efficiency, we present the LHGNet backbone, a more extensive feature extraction network, integrating depth-wise separable convolution and channel shuffle modules. This integration facilitates a thorough exploration of the inherent features within the network at deeper layers, promoting the fusion of local detail information and channel characteristics. Furthermore, we introduce the LGS bottleneck and LGSCSP fusion module incorporated into the neck, aiming to decrease the computational complexity while preserving the detector’s accuracy. Additionally, we enhance the detection accuracy by modifying its structure and the size of the feature maps. These improvements significantly enhance the model’s capability to capture tiny objects. The proposed LE-YOLO detector is examined in ablation and comparative experiments on the VisDrone2019 dataset. In contrast to YOLOv8n, the proposed LE-YOLO model achieved a 30.0% reduction in the parameter count, accompanied by a 15.9% increase in the mAP(0.5). These comprehensive experiments indicate that our approach can significantly enhance the detection accuracy and optimize the model efficiency through the organic combination of our suggested enhancements.