Edge Devices Research Articles

XimSwap: Many-to-Many Face Swapping for TinyML

The unprecedented development of deep learning approaches for video processing has caused growing privacy concerns. To ensure data analysis while maintaining privacy, it is essential to address how to protect individuals’ identities. One solution is to anonymize data at the source, avoiding the transmission or storage of information that could lead to identification. This study introduces XimSwap, a novel deep learning technique for real-time video anonymization, which can remove facial identification features directly on edge devices with minimal computational resources. Our approach offers a comprehensive solution that guarantees privacy by design. This novel method for implementing face-swapping ensures that the pose and expression of a target face remain unchanged and can be used on embedded devices with very limited computational resources. By incorporating style transfer layers into convolutional ones and optimizing the network’s operation, we achieved a reduction of over 98% in the required operations and parameters compared with state-of-the-art architectures. Our approach also significantly reduces RAM usage, making it possible to implement the anonymization process on tiny edge devices, including microcontrollers, such as the STM32H743.

Quantized non-volatile nanomagnetic domain wall synapse based autoencoder for efficient unsupervised network anomaly detection

Abstract Anomaly detection in real-time using autoencoders implemented on edge devices is exceedingly challenging due to limited hardware, energy, and computational resources. We show that these limitations can be addressed by designing an autoencoder with low-resolution non-volatile memory-based synapses and employing an effective quantized neural network learning algorithm. We further propose nanoscale ferromagnetic racetracks with engineered notches hosting magnetic domain walls (DW) as exemplary non-volatile memory based autoencoder synapses, where limited state (5-state) synaptic weights are manipulated by spin orbit torque (SOT) current pulses to write different magnetoresistance states. The performance of anomaly detection of the proposed autoencoder model is evaluated on the NSL-KDD dataset. Limited resolution and DW device stochasticity aware training of the autoencoder is performed, which yields comparable anomaly detection performance to the autoencoder having floating-point precision weights. While the limited number of quantized states and the inherent stochastic nature of DW synaptic weights in nanoscale devices are typically known to negatively impact the performance, our hardware-aware training algorithm is shown to leverage these imperfect device characteristics to generate an improvement in anomaly detection accuracy (90.98%) compared to accuracy obtained with floating-point synaptic weights that are extremely memory intensive. Furthermore, our DW-based approach demonstrates a remarkable reduction of at least three orders of magnitude in weight updates during training compared to the floating-point approach, implying significant reduction in operation energy for our method. This work could stimulate the development of extremely energy efficient non-volatile multi-state synapse-based processors that can perform real-time training and inference on the edge with unsupervised data.

Empirical Study of Fully Homomorphic Encryption Using Microsoft SEAL

In the context of the increasing integration of Internet of Things technologies and the growing importance of data lakes, the need for robust cybersecurity measures to protect privacy without compromising data utility becomes key. Aiming to address the privacy–security challenge in such digital ecosystems, this study explores the application of Fully Homomorphic Encryption (FHE) using the Microsoft SEAL library. FHE allows for operations on encrypted data, offering a promising opportunity for maintaining data confidentiality during processing. Our research employs systematic experimental tests on datasets to evaluate the performance of homomorphic encryption in terms of CPU usage and execution time, executed across traditional PC configurations and a NVIDIA Jetson Nano device to assess the scalability and practicality of FHE in edge computing. The results reveal a performance disparity between computing environments, with the PC showing stable performance and the Jetson Nano revealing the limitations of edge devices in handling encryption tasks due to computational and memory constraints.

Less-supervised learning with knowledge distillation for sperm morphology analysis

ABSTRACT Sperm Morphology Analysis (SMA) is pivotal in diagnosing male infertility. However, manual analysis is subjective and time-intensive. Artificial intelligence presents automated alternatives, but hurdles like limited data and image quality constraints hinder its efficacy. These challenges impede Deep Learning (DL) models from grasping crucial sperm features. A solution enabling DL models to learn sample nuances, even with limited data, would be invaluable. This study proposes a Knowledge Distillation (KD) method to distinguish normal from abnormal sperm cells, leveraging the Modified Human Sperm Morphology Analysis dataset. Despite low-resolution, blurry images, our method yields relevant results. We exclusively utilize normal samples to train the model for anomaly detection, crucial in scenarios lacking abnormal data – a common issue in medical tasks. Our aim is to train an Anomaly Detection model using a dataset comprising unclear images and limited samples, without direct exposure to abnormal data. Our method achieves Receiver ROC/AUC scores of 70.4%, 87.6%, and 71.1% for head, vacuole, and acrosome, respectively, our method matches traditional DL model performance with less than 70% of the data. This less-supervised approach shows promise in advancing SMA despite data scarcity. Furthermore, KD enables model adaptability to edge devices in fertility clinics, requiring less processing power.

Online Nonparametric Process Monitoring for IoT Systems Using Edge Computing

Facilitated by modern sensing and integrated circuit technology, Internet of Things (IoT) systems now have the ability to intelligently and automatically collect data, perform computational tasks on edge, and efficiently make decisions locally or remotely. However, existing multivariate statistical process control (MSPC) methods commonly assume that sensors directly transmit the raw measurements to the central server, and thus the edge computing capability of the IoT systems is not effectively harnessed. To fill this literature gap, we propose a monitoring scheme tailored for IoT systems by leveraging their edge computational power. In particular, inspired by the inherent hierarchical structure of the IoT system, we construct the monitoring statistics hierarchically. At each edge device, we first construct a nonparametric stream-level statistic for each data stream and aggregate them to derive the device-level statistics. To eliminate the inconsistency in device-level statistics across different edge devices, the device-level statistics are transformed into time-related statistics, which are then sent to the central server. At the central server, a system-level monitoring statistic is constructed, and the most informative edge devices that should transmit statistics are intelligently determined. Theoretical analyses are carried out to justify the effectiveness of the data transmission strategy. Both simulations and a case study, which is based on an IoT system we built, are conducted to evaluate the performance and validate the superiority of the proposed monitoring scheme.

Distilling particle knowledge for fast reconstruction at high-energy physics experiments

Knowledge distillation is a form of model compression that allows artificial neural networks of different sizes to learn from one another. Its main application is the compactification of large deep neural networks to free up computational resources, in particular on edge devices. In this article, we consider proton-proton collisions at the High-Luminosity Large Hadron Collider (HL-LHC) and demonstrate a successful knowledge transfer from an event-level graph neural network (GNN) to a particle-level small deep neural network (DNN). Our algorithm, DistillNet, is a DNN that is trained to learn about the provenance of particles, as provided by the soft labels that are the GNN outputs, to predict whether or not a particle originates from the primary interaction vertex. The results indicate that for this problem, which is one of the main challenges at the HL-LHC, there is minimal loss during the transfer of knowledge to the small student network, while improving significantly the computational resource needs compared to the teacher. This is demonstrated for the distilled student network on a CPU, as well as for a quantized and pruned student network deployed on an field programmable gate array. Our study proves that knowledge transfer between networks of different complexity can be used for fast artificial intelligence (AI) in high-energy physics that improves the expressiveness of observables over non-AI-based reconstruction algorithms. Such an approach can become essential at the HL-LHC experiments, e.g. to comply with the resource budget of their trigger stages.

Efficient pitch‐estimation network for edge devices

AbstractPitch estimation is the task of finding the most conspicuous frequency in a complex audio signal. Many methods that use deep neural networks have significantly increased the accuracy of pitch estimation; however, their real‐time performance results were achieved on high‐performance devices. Because pitch estimation is widely used in real‐time applications on low‐power devices, we propose an efficient method for estimating pitch on edge devices. The network architecture of the proposed method uses a depth‐scaling strategy and fully leverages convolutional networks. We further introduce a channel attention mechanism to increase accuracy without increasing computational overhead. We compared the proposed model with state‐of‐the‐art (SOTA) and conventional methods using two public datasets. The experimental results show that the proposed method has a better classification accuracy than FCNF0++, which is the best performing SOTA model. Furthermore, it reduces the processing time obtained by FCNF0++ on a personal computer and two edge devices by 48% on average. These experimental results confirm that the proposed method efficiently classifies pitch on edge devices.

A federated learning-based zero trust intrusion detection system for Internet of Things

The exponential growth of Internet of Things (IoT) devices poses distinctive challenges to safeguarding the security and privacy of interconnected systems. As the frequency of cyberattacks continues to escalate, the development of an effective and scalable Intrusion Detection System (IDS) based on Federated Learning (FL) for IoT becomes increasingly complex. Existing methodologies struggle to balance spatial and temporal feature extraction, particularly when confronted with dynamic and evolving cyber threats. Additionally, the lack of diversity in datasets employed for FL-based IDS evaluations further hinders progress. Furthermore, a notable tradeoff between performance and scalability emerges, particularly as the number of edge devices in communication grows. Motivated by the aforementioned challenges, this article presents a horizontal FL model that combines Convolutional Neural Networks (CNN) and Bidirectional Long-Term Short Memory (BiLSTM) for effective intrusion detection. This hybrid approach aims to address the limitations of existing methods and enhance the effectiveness of intrusion detection in the context of FL for IoT. Specifically, CNN plays a pivotal role in spatial feature extraction, allowing the model to identify and comprehend local patterns indicative of potential intrusions, and the BiLSTM component contributes by capturing temporal dependencies and learning sequential patterns within the data. The proposed IDS adheres to a zero-trust model by keeping the data on local edge devices, sharing only the learned weights with the centralized FL server. In turn, the FL server aggregates updates from diverse sources to optimize the accuracy of the global learning model. The experimental results using CICIDS2017 and Edge-IIoTset prove the effectiveness of the proposed approach over centralized and federated deep learning-based IDS.

Designing and Training of Lightweight Neural Networks on Edge Devices Using Early Halting in Knowledge Distillation

Designing and Training of Lightweight Neural Networks on Edge Devices Using Early Halting in Knowledge Distillation

DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices

DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices

A Laplace operator-based active contour model with improved image edge detection performance

Active contour model (ACM) is an important branch in the field of image segmentation, since it adapts to different image types and scenes. However, traditional ACMs rely on local image information, which makes them sensitive to initial contours and time-consuming. In this paper, a Laplace operator based ACM (LOACM) with improved image edge detection performance is proposed. LOACM combines the global pre-fitting function which is constructed by using the image's second-order gradient information with the adaptive boundary indicator function to build a hybrid model. Its compact code size allows it to be executed on a wide range of edge devices. To substantiate the segmentation capability of the LOACM model, the BSD500 dataset was utilized for experimentation. The results indicate that LOACM exhibits excellent performance, with average IOU and DSC scores of 0.861 and 0.924 respectively. Additionally, it consistently handles processes within an average time of less than 0.3s, significantly less than that of comparative models, providing compelling evidence of LOACM's exceptional efficiency and precision.

A decentralized federated learning based interoperable and heterogeneity aware predictive optimization method for energy and comfort in smart homes environment

It is a known fact that a comfortable residential space or workspace is crucial to productivity and well-being. In smart home environments, sensors can be used to sense environment and occupant data and actuators can be controlled accordingly for maximizing thermal comfort. However, keeping in view the growing demand of energy and scarcity of energy generation sources, it is mandatory to optimize the energy usage complementary to optimize thermal comfort. In order for the developed method to be interoperable and heterogeneity aware across different edge device vendors and for ease of hardware connectivity in smart homes, a REST API based framework is developed. Sensors mounted with IoT devices collect data from its respective smart home environment and store the data in its local database. An edge server is used to control the connectivity among these IoT devices and to initiate the decentralized federated learning mechanism. Each IoT device train local deep learning models for predicting energy usage, indoor temperature, indoor humidity for maintaining a comfortable environment state using minimal energy. Models from each IoT devices are aggregated at each IoT device iteratively to make federated learning independent of central server. Global deep learning model is iteratively trained for each prediction requirement. Later, these models are used to predict indoor temperature, humidity and energy consumption for time stamp (t+1), keeping in view environmental and occupant related sensed data at time stamp (t). Optimization of energy consumption and thermal comfort is mathematically modeled and based on the predictions of DL models, environment is controlled by handling actuators using a particle swarm optimization (PSO) mechanism. Relevant constraints are incorporated in PSO for limiting the positional variations in particles for efficient learning. The method continually learns and adapts to changes in the surroundings, providing real-time monitoring and control of thermal comfort to the end users. To validate our approach, data is collected and experiments are conducted in a real-time smart home’s environment. The results are compared with those obtained without using the proposed predictive optimization mechanism. The study demonstrates the effectiveness of proposed predictive optimization mechanism in providing a comfortable and productive smart home or workspace environment utilizing 38% less energy as compared to competitor.

Elastic Federated Learning with Kubernetes Vertical Pod Autoscaler for edge computing

Federated Learning (FL) is an emerging paradigm for training machine learning models across decentralized edge devices, ensuring data privacy and reducing computational tasks on the central cloud. However, the dynamic and resource-constrained nature of edge environments raises significant challenges in deploying FL applications efficiently. This paper introduces an Elastic Federated Learning framework that leverages Kubernetes Vertical Pod Autoscaler to improve the performance of FL applications. The framework enables dynamic adjustment of computational resources, facilitating effective resource utilization for model training and allowing quicker attainment of the targeted accuracy levels for the trained models. Our experiments demonstrate that the proposed framework significantly enhances FL efficiency, maintains model training progress during resource adjustments, and effectively addresses the resource allocation challenges in edge environments. This work advances the capabilities of FL in edge computing, opening doors to more efficient and scalable AI applications while preserving data privacy.

Bat2Web: A Framework for Real-Time Classification of Bat Species Echolocation Signals Using Audio Sensor Data.

Bats play a pivotal role in maintaining ecological balance, and studying their behaviors offers vital insights into environmental health and aids in conservation efforts. Determining the presence of various bat species in an environment is essential for many bat studies. Specialized audio sensors can be used to record bat echolocation calls that can then be used to identify bat species. However, the complexity of bat calls presents a significant challenge, necessitating expert analysis and extensive time for accurate interpretation. Recent advances in neural networks can help identify bat species automatically from their echolocation calls. Such neural networks can be integrated into a complete end-to-end system that leverages recent internet of things (IoT) technologies with long-range, low-powered communication protocols to implement automated acoustical monitoring. This paper presents the design and implementation of such a system that uses a tiny neural network for interpreting sensor data derived from bat echolocation signals. A highly compact convolutional neural network (CNN) model was developed that demonstrated excellent performance in bat species identification, achieving an F1-score of 0.9578 and an accuracy rate of 97.5%. The neural network was deployed, and its performance was evaluated on various alternative edge devices, including the NVIDIA Jetson Nano and Google Coral.

CRAMP: Clustering-based RANs association and MEC placement for delay-sensitive applications

With advancements in networking technology and ubiquitous computing, there has been a significant increase in the number of edge devices and delay-sensitive applications. To facilitate efficient processing, mobile edge computing (MEC) technology provides resources through MEC servers, which are deployed at the radio access networks (RANs) of 5G networks. However, these MEC servers possess a limited amount of resources, making their effective management of these resources a critical challenge. This is due to the uneven distribution of resource utilization, where some resources become overutilized while others remain underutilized. Addressing the issue above while simultaneously satisfying user requirements for delay-sensitive applications poses a significant challenge at the edge. In this paper, we propose a clustering-based efficient RANs association and MEC server placement model to tackle this challenge. Our primary objective is to minimize MEC server deployment costs while ensuring that the delays of these applications are effectively managed. We propose a greedy algorithm called the clustering-based radio access networks association and mobile edge computing placement (CRAMP) algorithm, which determines the optimal location of MEC servers to associate with RANs. Simulation results demonstrate that our proposed algorithm outperforms existing approaches regarding cost efficiency and delay management.

PASS: Patch Automatic Skip Scheme for Efficient On-Device Video Perception.

Real-time video perception tasks are often challenging on resource-constrained edge devices due to the issues of accuracy drop and hardware overhead, where saving computations is the key to performance improvement. Existing methods either rely on domain-specific neural chips or priorly searched models, which require specialized optimization according to different task properties. These limitations motivate us to design a general and task-independent methodology, called Patch Automatic Skip Scheme (PASS), which supports diverse video perception settings by decoupling acceleration and tasks. The gist is to capture inter-frame correlations and skip redundant computations at patch level, where the patch is a non-overlapping square block in visual. PASS equips each convolution layer with a learnable gate to selectively determine which patches could be safely skipped without degrading model accuracy. Specifically, we are the first to construct a self-supervisory procedure for gate optimization, which learns to extract contrastive representations from frame sequences. The pre-trained gates can serve as plug-and-play modules to implement patch-skippable neural backbones, and automatically generate proper skip strategy to accelerate different video-based downstream tasks, e.g., outperforming state-of-the-art MobileHumanPose in 3D pose estimation and FairMOT in multiple object tracking, by up to 9.43 × and 12.19 × speedups, respectively, on NVIDIA Jetson Nano devices.

Wake Word and Speech Recognition Application on Edge Device: A Case of Improving the Electric Wheelchair

Wake Word and Speech Recognition Application on Edge Device: A Case of Improving the Electric Wheelchair

A Blockchain-Based Proxy Re-Encryption Method for Safe Data Sharing in the Internet of Things

Data sharing has emerged as one of the Internet of Things' most beneficial uses in cloud computing. Even though this technology has been visually appealing, one of its challenges is data security because improper usage of data can result in various harms. We suggest a safe data exchange method for cloud environments in this article: proxy re-encryption. Data owners can utilize identity-based encryption to send their encrypted data to the cloud, where it will be accessible to authorized users through proxy re- encryption construction. An edge device serves as a proxy server to manage demanding calculations since Internet of Things devices are resource- constrained. Additionally, by efficiently delivering cached content in the proxy and utilizing the capabilities of information-centric networking, we enhance service quality and optimize network bandwidth. Additionally, the foundation of our system concept is blockchain, a revolutionary technology that permits decentralization in data sharing. Fine-grained access control to data is achieved, and bottlenecks in centralized systems are lessened.

YOLO‐AFPN: Marrying YOLO and AFPN for external damage detection of transmission lines

AbstractTo better detect targets that may cause external damage to transmission lines, the authors present You Only Look Once‐Asymptotic Feature Pyramid Network (YOLO‐AFPN), a lightweight but efficient model. Firstly, the authors adopt a feature comparison strategy based on the knowledge of transmission line scenes, which facilitates increased attention to target features during the training. Secondly, the YOLOv8 detection network is built, and the backbone adds three layers of simple parameter‐free attention module, which extracts features while maintaining lightness, and improves the detection capability in complex scenarios. Then, in the feature fusion stage, an AFPN is constructed, which improves the multi‐scale target detection capability while reducing the number of model parameters by asymptotically fusing features that have small semantic gaps between neighbouring layers. When during the training process, the improved Mosaic data augmentation method is used to enhance the number of distributions of small targets, improve the robustness of the model. Finally, the improved model is validated, and the experimental results show that the improved model can achieve mean average precision of 86.1% at 6.6 MB, which is better than the original network for detection and meets the requirements for deployment on edge devices.

A novel trusted hardware-based scalable security framework for IoT edge devices

The Internet of Things (IoT) devices are pervasively deployed and embedded into our daily lives. Over several years, the massive assimilation of IoT devices has given rise to smart cities, smart factories, smart farms, etc. While IoTs can collect immense amounts of user and environmental information, the lack of adequate security measures is among the major concerns about information security and privacy. With the ever-increasing presence of IoTs, information flow between IoT devices in a single or multi-layered network is becoming increasingly vulnerable to attacks. Although several software-based solutions have been developed over the years that improve the security of resource-constrained IoT devices, there are limited efforts on hardware-oriented IoT security solutions. This paper proposes a Trusted HArdware-based Scalable Secure Architecture (THASSA) for IoT Edge systems. Leveraging standard cryptographic algorithms, THASSA provides a secure mechanism for data transfer and off-loads cryptography operations to trusted hardware. The modularized THASSA architecture allows devices to be added or removed without user intervention. Also, THASSA could be scaled to multiple subsystems using multiple trusted hardware to form IoT clusters with secure data path(s) for mission-critical applications. An experimental study has verified the correctness and effectiveness of THASSA.