Real-time Processing Requirements Research Articles

Energy-aware bio-inspired spiking reinforcement learning system architecture for real-time autonomous edge applications.

Mobile, low-cost, and energy-aware operation of Artificial Intelligence (AI) computations in smart circuits and autonomous robots will play an important role in the next industrial leap in intelligent automation and assistive devices. Neuromorphic hardware with spiking neural network (SNN) architecture utilizes insights from biological phenomena to offer encouraging solutions. Previous studies have proposed reinforcement learning (RL) models for SNN responses in the rat hippocampus to an environment where rewards depend on the context. The scale of these models matches the scope and capacity of small embedded systems in the framework of Internet-of-Bodies (IoB), autonomous sensor nodes, and other edge applications. Addressing energy-efficient artificial learning problems in such systems enables smart micro-systems with edge intelligence. A novel bio-inspired RL system architecture is presented in this work, leading to significant energy consumption benefits without foregoing real-time autonomous processing and accuracy requirements of the context-dependent task. The hardware architecture successfully models features analogous to synaptic tagging, changes in the exploration schemes, synapse saturation, and spatially localized task-based activation observed in the brain. The design has been synthesized, simulated, and tested on Intel MAX10 Field-Programmable Gate Array (FPGA). The problem-based bio-inspired approach to SNN edge architectural design results in 25X reduction in average power compared to the state-of-the-art for a test with real-time context learning and 30 trials. Furthermore, 940x lower energy consumption is achieved due to improvement in the execution time.

Underwater target detection network based on differential routing assistance and bilateral attention synergy

Underwater target detection technology holds significant importance in both military and civilian applications of ocean exploration. However, due to the complex underwater environment, most targets are small and often obscured, leading to low detection accuracy and missed detections in existing target detection algorithms. To address these issues, we propose an underwater target detection algorithm that balances accuracy and speed. Specifically, we first propose the Differentiable Routing Assistance Sampling Network named (DRASN), where differentiable routing participates in training the sampling network but not in the inference process. It replaces the down-sampling network composed of Maxpool and convolution fusion in the backbone network, reducing the feature loss of small and occluded targets. Secondly, we proposed the Bilateral Attention Synergistic Network (BASN), which establishes connections between the backbone and neck with fine-grained information from both channel and spatial perspectives, thereby further enhancing the detection capability of targets in complex backgrounds. Finally, considering the characteristics of real frames, we proposed a scale approximation auxiliary loss function named (Aux-Loss) and modify the allocation strategy of positive and negative samples to enable the network to selectively learn high-quality anchors, thereby improving the convergence capability of the network. Compared with mainstream algorithms, our detection network achieves 82.9% in mAP@0.5 on the URPC2021 dataset, which is 9.5%, 5.7%, and 2.8% higher than YOLOv8s, RT-DETR, and SDBB respectively. The speed reaches 75 FPS and meets the requirements for real-time performance.

A Lightweight Real-Time Recognition Algorithm for Tomato Leaf Disease Based on Improved YOLOv8

To address the real-time detection challenge of deploying deep learning-based tomato leaf disease detection algorithms on embedded devices, an improved tomato leaf disease detection algorithm based on YOLOv8n is proposed in this paper. It is able to achieve the efficient, real-time detection of tomato leaf diseases while maintaining model’s lightweight requirements. The algorithm incorporated the LMSM (lightweight multi-scale module) and ALSA (Attention Lightweight Subsampling Module) to improve the ability to extract lightweight and multi-scale semantic information for the specific characteristics of tomato leaf disease, which include irregular spot size and lush tomato leaves. The head network was redesigned utilizing partial and group convolution along with a parameter-sharing method. Scalable auxiliary bounding box and loss function optimization strategies were introduced to further enhance performance. After undergoing the pruning technique, computation decreased by 61.7%, the model size decreased by 55.6%, and the FPS increased by 44.8%, all while a high level of accuracy was maintained. A detection speed of 19.70FPS on the Jetson Nano was obtained after undergoing TensorRT quantization, showing a 64.85% improvement compared to the initial detection speed. This method met the high real-time performance and small model size requirements for embedded tomato leaf disease detection systems, indirectly reducing the energy consumption of online detection. It provided an effective solution for the online detection of tomato leaf disease.

Design of ECG Circuit and Patient Remote Monitoring System Using MQTT Protocol for ECG Signals

In the healthcare sector, leveraging the Internet of Things (IoT) brings significant advantages, especially in remote monitoring of patients' health using electrocardiogram (ECG) signals. This study employs IoT technology to enable remote ECG monitoring, utilizing the MQTT (Message Queuing Telemetry Transport) protocol to establish a web server. The dedicated measuring device integrates the ADAS1000-3 IC with ESP32 for real-time ECG signal acquisition. In this study, we utilize the Pan-Tompkins algorithm to extract QRS complexes from electrocardiographic signals, count heartbeats, and early detect cardiovascular abnormalities. This solution was tested on a sample dataset from the MIT-BIH (Massachusetts Institute of Technology - Beth Israel Deaconess Medical Center) database. Finally, the article proposes to design and develop a device using advanced IC technologies such as IC ADAS1000-3 combined with ESP32. This device will be designed to meet the requirements of real-time processing and data transmission via webserver.

CIS: A Coral Instance Segmentation Network Model with Novel Upsampling, Downsampling, and Fusion Attention Mechanism

Coral segmentation poses unique challenges due to its irregular morphology and camouflage-like characteristics. These factors often result in low precision, large model parameters, and poor real-time performance. To address these issues, this paper proposes a novel coral instance segmentation (CIS) network model. Initially, we designed a novel downsampling module, ADown_HWD, which operates at multiple resolution levels to extract image features, thereby preserving crucial information about coral edges and textures. Subsequently, we integrated the bi-level routing attention (BRA) mechanism into the C2f module to form the C2f_BRA module within the neck network. This module effectively removes redundant information, enhancing the ability to distinguish coral features and reducing computational redundancy. Finally, dynamic upsampling, Dysample, was introduced into the CIS to better retain the rich semantic and key feature information of corals. Validation on our self-built dataset demonstrated that the CIS network model significantly outperforms the baseline YOLOv8n model, with improvements of 6.3% and 10.5% in PB and PM and 2.3% and 2.4% in mAP50B and mAP50M, respectively. Furthermore, the reduction in model parameters by 10.1% correlates with a notable 10.7% increase in frames per second (FPS) to 178.6, thus effectively meeting real-time operational requirements.

Dual-polarized transmissive metasurfaces for near-unitary-NA analog spatial computing empowered by impedance matching and mismatching.

In recent years, due to the increasing requirement for real-time and massive data processing, optical analog computation has arisen as a promising alternative to digital computation. Optical spatial differentiation plays a fundamentally important role in various emerging technologies, including augmented reality, autonomous driving, and object recognition. However, previous demonstrations encountered several limitations, such as the dependence on polarization states and a typically limited numerical aperture (NA) of about 0.5, especially in the transmission mode. Here, a new, to our knowledge, design strategy based on the evolution between impedance matching and mismatching in a metasurface is proposed to fill this gap, which can perform dual-polarized second-order derivative for image processing. Our scheme enables high transmission under dual polarization over an 85° incident angle range (NA = 0.996), resulting in more than twofold spatial resolution. Our work paves the way for polarization-insensitive high-resolution signal and image processing in the terahertz region.

FastUNet: Fast hierarchical multi-patch underwater enhancement network for industrial recirculating aquaculture

The industrialized aquaculture with recirculating water systems has gained increasing attention and rapid development in recent years. However, the existing underwater enhancement methods are time-consuming and severely hinder their online application in industrialized aquaculture due to the requirement for real-time processing. We found that their main limitations are the insufficient utilization of the hierarchical network approach and valuable prior information in underwater scenarios. To address this problem, we propose a novel architecture called FastUNet and introduce a visual enhancement benchmark dataset specifically designed for industrialized aquaculture with recirculating water systems. Surprisingly, by learning the fast multi-patch hierarchical module and employing prior constraints including color loss and homology loss, FastUNet achieves comparable or even better results than many deep learning methods. In terms of runtime, it is up to 30 times faster than current underwater enhancement methods. The processing time for images with a resolution of 5474 × 3653 is only 0.137s. The real application scenario of factory recirculating aquaculture is considered by FastUNet, and the embedding of prior information under the fast network framework is explored. In this paper, 14 state-of-the-art underwater enhancement methods are compared, and excellent results are obtained on 2 Full-reference datasets and 2 No-referenced datasets. The application test results of SIFT significant point detection, geometric rotation, edge detection, target segmentation and target detection were also presented, which strongly verified the feasibility of FastUNet in the field of factory recirculating aquaculture.

Power Generation Optimization for Next-Generation Cruise Ships with MVDC Architecture: A Dynamic Modeling and Simulation Approach

The cruise industry is obliged by economic and environmental initiatives to pursue fuel-efficient solutions and lower ship exhaust emissions. The medium voltage DC (MVDC) distribution with intelligent power management has become a concept for next-generation onboard power systems as its energy-saving feature is to eliminate the frequency constraint and simultaneously optimize engine loads and speed in response to load variations. The incentive for this transition lies on one hand in the fuel efficiency consideration and the reduction of power losses from serial conversion stages. On the other hand, the DC-based technology has been conceived as high-power density design, thus significantly increasing the payload. This study investigates such potential benefits focusing exclusively on large cruise vessels. A highly representative model of the integrated power platform that incorporates all dynamic interactions from the ship hull and essential machinery typically installed on board cruise ships is proposed. The power management strategy also takes account of actual sea conditions and real-time operation requirements. The simulation results demonstrate that the optimization-based MVDC system is able to maximize the opportunity of search agents in finding optimum fuel efficiency areas throughout the scenario time. An analysis of the system structure weight and space reduction of the MVDC architecture is also performed through the utilization of more compact electrical distribution devices and very high power-dense combustion turbines.

TLDM: An Enhanced Traffic Light Detection Model Based on YOLOv5

Traffic light detection and recognition are crucial for enhancing the security of unmanned systems. This study proposes a YOLOv5-based traffic light-detection algorithm to tackle the challenges posed by small targets and complex urban backgrounds. Initially, the Mosaic-9 method is employed to enhance the training dataset, thereby boosting the network’s ability to generalize and adapt to real-world scenarios. Furthermore, the Squeeze-and-Excitation (SE) attention mechanism is incorporated to improve the network. Moreover, the YOLOv5 algorithm’s loss function is optimized by substituting it with Efficient Intersection over Union loss (EIoU_loss), which addresses issues like missed detection and false alarms. Experimental results demonstrate that the model trained with this enhanced network achieves an mAP (mean average precision) of 99.4% on a custom dataset, which is 6.3% higher than that of the original YOLOv5, while maintaining a detection speed of 74 f/s. Therefore, this algorithm offers higher detection accuracy and effectively meets real-time operational requirements. The proposed method has a strong application potential, and can be widely used in the field of automatic driving, assisted driving, etc. Its application is not only of great significance in improving the accuracy and speed of traffic sign detection, but also can provide technical support for the development of intelligent transportation systems.

A lightweight and continuous dimensional emotion analysis system of facial expression recognition under complex background

Facial expression recognition technology has a brilliant prospect in applying the Internet of Things systems. Concerning the limited hardware computing capability and high real-time processing requirements, this paper proposes a lightweight emotion analysis system based on edge computing, which could be deployed on edge devices. To further improve the accuracy of dimensional emotion analysis, we propose a modified network structure of MobileNetV3 to measure the intensity of dimensional emotion. The optimization scheme includes introducing the improved efficient channel attention mechanism and the feature pyramid network, adjusting the structure of the model, and optimizing the loss function. Furthermore, the system uses Intel’s OpenVINO toolkit to make the model more suitable for stable operation and provides an operable human–computer interaction interface. The experimental results show that the system has the advantages of few parameters, high recognition accuracy, and low latency. The optimized network size is reduced by 67% compared with the original, the root mean square error of potency and activation is 0.413 and 0.389, and the latency is up to 9ms in Myriad. This work meets the requirements of practical applications and has essential significance with the demand for continuous dimensional emotion analysis. The source code is available at https://github.com/SCNU-RISLAB/Lightweight-and-Continuous-Dimensional-Emotion-Analysis-System-of-Facial-Expression-Recognition/.

Anomaly detection of power grid data based on storm cloud platform sliding window

Abstract To realize the monitoring and diagnosis of multi-source data in the power industry and meet the real-time processing requirements of the power system, the storm-distributed real-time computing platform is introduced to process the data. On this platform, the streaming data processing model based on a sliding window is deployed to realize the anomaly diagnosis of data flow. By calculating the predicted value of a point and the confidence interval of the predicted value of the point, we can judge whether the actual point is an abnormal value. By testing the data processing capacity of a cluster and a single machine, the experiment shows that in the cluster environment, reasonably setting the component parallelism can improve the throughput of flow computing.

Infrared Image Human Target Detection Method Based on Deep Learning

Introduction: In this study, we investigate human target detection techniques for infrared target recognition in complicated backdrops using deep learning approaches. This study uses the human target detection of infrared images in various scenes as its research object. Method: A test was conducted using MatlabR2010a and the real-time C language development platform. The findings indicate that, in the unoptimized case, the computing speed of the method was only 0.34 seconds. However, following optimization, its performance significantly increased to meet the real-time performance requirements. Result: The research findings presented in this work will be crucial to the relief and rescue efforts following engineering mishaps. Conclusion: This study is innovative in that it develops a deep learning-based model for human target detection in infrared photos and thoroughly examines and improves its functionality.

Multi source data security protection of smart grid based on edge computing

In order to cope with the continuous development of information technology, the data volume of edge devices in the power supply network is increasing rapidly, and the higher requirements for real-time data processing and transmission bandwidth are put forward, the author proposes the research on security protection of multi-source data in smart grid based on edge computing. Established a distribution network safety risk map, calculated and analyzed safety risks through potential functions, and obtained node potential values and key node data; The intelligent grid multi-source heterogeneous data monitoring model based on edge computing can realize the timely perception and real-time response of distribution network faults, shorten the outage time, and improve the power supply reliability and user satisfaction of the distribution network. The results indicate that: Taking a distribution network line in a certain area as an example, the highest potential value is node 9, followed by nodes 10 and 3. This indicates that in order to ensure the reliable operation of the distribution network and avoid power outages, it is necessary to improve the repair capacity, the proportion of old equipment, and the power supply radius exceeding the standard. ConclusionThe effectiveness of potential function in risk assessment of edge computing is verified, which can provide theoretical guidance for distribution network fault diagnosis.

Synchronising a stereoscopic surgical video stream using specular reflection.

A stereoscopic surgical video stream consists of left-right image pairs provided by a stereo endoscope. While the surgical display shows these image pairs synchronised, most capture cards cause de-synchronisation. This means that the paired left and right images may not correspond once used in downstream tasks such as stereo depth computation. The stereo synchronisation problem is to recover the corresponding left-right images. This is particularly challenging in the surgical setting, owing to the moist tissues, rapid camera motion, quasi-staticity and real-time processing requirement. Existing methods exploit image cues from the diffuse reflection component and are defeated by the above challenges. We propose to exploit the specular reflection. Specifically, we propose a powerful left-right comparison score (LRCS) using the specular highlights commonly occurring on moist tissues. We detect the highlights using a neural network, characterise them with invariant descriptors, match them, and use the number of matches to form the proposed LRCS. We perform evaluation against 147 existing LRCS in 44 challenging robotic partial nephrectomy and robotic-assisted hepatic resection video sequences with simulated and real de-synchronisation. The proposed LRCS outperforms, with an average and maximum offsets of 0.055 and 1 frames and 94.1±3.6% successfully synchronised frames. In contrast, the best existing LRCS achieves an average and maximum offsets of 0.3 and 3 frames and 81.2±6.4% successfully synchronised frames. The use of specular reflection brings a tremendous boost to the real-time surgical stereo synchronisation problem.

Real-Time Registration of Unmanned Aerial Vehicle Hyperspectral Remote Sensing Images Using an Acousto-Optic Tunable Filter Spectrometer

Differences in field of view may occur during unmanned aerial remote sensing imaging applications with acousto-optic tunable filter (AOTF) spectral imagers using zoom lenses. These differences may stem from image size deformation caused by the zoom lens, image drift caused by AOTF wavelength switching, and drone platform jitter. However, they can be addressed using hyperspectral image registration. This article proposes a new coarse-to-fine remote sensing image registration framework based on feature and optical flow theory, comparing its performance with that of existing registration algorithms using the same dataset. The proposed method increases the structure similarity index by 5.2 times, reduces the root mean square error by 3.1 times, and increases the mutual information by 1.9 times. To meet the real-time processing requirements of the AOTF spectrometer in remote sensing, a development environment using VS2023+CUDA+OPENCV was established to improve the demons registration algorithm. The registration algorithm for the central processing unit+graphics processing unit (CPU+GPU) achieved an acceleration ratio of ~30 times compared to that of a CPU alone. Finally, the real-time registration effect of spectral data during flight was verified. The proposed method demonstrates that AOTF hyperspectral imagers can be used in real-time remote sensing applications on unmanned aerial vehicles.

LID-Net: A lightweight image dehazing network for automatic driving vision systems

Visual system provides comprehensive road information for autonomous driving vehicles. Haze adversely affects the quality of driving images captured by onboard cameras, which poses a significant challenge to the safe operation of vehicles relying on pure vision-based autonomous driving solutions. To address the above issues, a lightweight image dehazing algorithm using a multi-scale architecture called LID-Net is proposed. LID-Net consists of Haze Extraction (HE) blocks and Haze Removal (HR) blocks. The HE block captures more haze features by employing a larger receptive field. The HR block introduces different attention weights to different regions to better remove haze of different concentrations. Furthermore, a Brightness Compensation unit is designed to address the issue of reduced brightness in images following dehazing. This unit compensates for the image's brightness without changing the color of the dehazed image. LID-Net is compared with other state-of-the-art methods on two real-world foggy weather datasets. The results indicate that LID-Net outperforms other methods in terms of dehazing effectiveness. LID-Net can efficiently process an image with a resolution of 1280 × 720 at 125 frames per second, which can fully meet the requirements of real-time processing of automatic driving systems.

A curved path extraction method using RGB-D multimodal data for single-edge guided navigation in irregularly shaped fields

Due to the undulating topography, the world contains numerous irregularly shaped small fields. Autonomous navigation along the curved edges of these fields is crucial for achieving complete unmanned operation in these small-field environments. Based on this, we proposed an accurate and robust curved path extraction method using RGB-D multimodal data for single-edge guided navigation in irregularly shaped fields. Leveraging the significant height difference between headland and farmland, we employed an efficient RGB-D multimodal semantic segmentation network to achieve high-precision segmentation of headland area. Meanwhile, a slope-based feature points adaptive selection algorithm and a feature points transformation algorithm fusing multimodal depth data were proposed innovatively for curved navigation path extraction. By utilizing the RGB-D segmentation model ESANet, we achieved a mean intersection over union (mIoU) of 96.94 %, surpassing other RGB segmentation models. The mean deviations between the ground truth path and the extracted path were less than 6.43 pixels on an image with a resolution of 640 × 480. The proposed method can finally achieve a processing efficiency of 23.80 FPS on images in 640 × 480 resolution, and the field experiments tested that the mean deviations of the agricultural vehicle navigating along the extracted curved navigation path were less than 0.188 m, which met the real-time performance requirement and accuracy requirement of autonomous navigation.

DFFNet: a lightweight approach for efficient feature-optimized fusion in steel strip surface defect detection

Steel surface defect detection is crucial in manufacturing, but achieving high accuracy and real-time performance with limited computing resources is challenging. To address this issue, this paper proposes DFFNet, a lightweight fusion network, for fast and accurate steel surface defect detection. Firstly, a lightweight backbone network called LDD is introduced, utilizing partial convolution to reduce computational complexity and extract spatial features efficiently. Then, PANet is enhanced using the Efficient Feature-Optimized Converged Network and a Feature Enhancement Aggregation Module (FEAM) to improve feature fusion. FEAM combines the Efficient Layer Aggregation Network and reparameterization techniques to extend the receptive field for defect perception, and reduce information loss for small defects. Finally, a WIOU loss function with a dynamic non-monotonic mechanism is designed to improve defect localization in complex scenes. Evaluation results on the NEU-DET dataset demonstrate that the proposed DFFNet achieves competitive accuracy with lower computational complexity, with a detection speed of 101 FPS, meeting real-time performance requirements in industrial settings. Furthermore, experimental results on the PASCAL VOC and MS COCO datasets demonstrate the strong generalization capability of DFFNet for object detection in diverse scenarios.

SpanEffiDet: Span-Scale and Span-Path Feature Fusion for Object Detection

Lower versions of EfficientDet (such as D0, D1) have smaller network structures and parameter sizes, but lower detection accuracy. Higher versions exhibit higher accuracy, but the increase in network complexity poses challenges for real-time processing and hardware requirements. To meet the higher accuracy requirements under limited computational resources, this paper introduces SpanEffiDet based on the channel adaptive frequency filter (CAFF) and the Span-Path Bidirectional Feature Pyramid structure. Firstly, the CAFF module proposed in this paper realizes the frequency domain transformation of channel information through Fourier transform and effectively extracts the key features through semantic adaptive frequency filtering, thus, eliminating channel redundant information of EfficientNet. Simultaneously, the module has the ability to compute the weights across the channels and at fine granularity, and capture the detailed information of element features. Secondly, a two-way characteristic pyramid network multi-level cross-BIFPN, which can achieve multi-layer and multi-nodes, is proposed to build cross-level information transmission to incorporate both semantic and positional information of the target. This design enables the network to more effectively detect objects with significant size differences in complex environments. Finally, by introducing generalized focal Loss V2, reliable localization quality estimation scores are predicted from the distribution statistics of bounding boxes, thereby improving localization accuracy. The experimental results indicate that on the MS COCO dataset, SpanEffiDet-D0 achieved an AP improvement of 3.3% compared to the original EfficientDet series algorithms. Similarly, on the PASCAL VOC2007 and 2012 datasets, the mAP of SpanEffiDet-D0 is respectively 1.66 and 2.65% higher than that of EfficientDet-D0.

Analysis and Construction of Hardware Accelerators for Calculating the Shortest Path in Real-Time Robot Route Planning

This study introduces an optimization approach for calculating the shortest path in mobile robot route planning. The proposed solution targets real-time processing requirements by offering a high-performance alternative. This is achieved by embedding in the dedicated hardware an architecture which emphasizes parallelism. Through improvements in parallel exploration techniques, our solution aims to present not only a boost in performance but also a dynamic adaptation to graph changes, accommodating randomly occurring edge insertions or deletions as environmental conditions fluctuate. We present the developed architecture alongside its results. Our method efficiently updates obstacle matrices, resulting in a remarkable 120-fold improvement for 1024-node graphs. When utilizing a cost-effective device like the Cyclone IV E, it achieves approximately 12 times the performance of software applications.