Jetson Nano Research Articles

APPLICATION OF VARIOUS DEEP LEARNING MODELS FOR AUTOMATIC TRAFFIC VIOLATION DETECTION USING EDGE COMPUTING

A rapid growth in the population and economic growth has resulted in an increasing number of vehicles on road every year. Traffic congestion is a big problem in every metropolitan city. To reach their destination faster and to avoid traffic, some people are violating traffic rules and regulations. Violation of traffic rules puts everyone in danger. Maintaining traffic rules manually has become difficult over the time due to the rapid increase in the population. This alarming situation has be taken care of at the earliest. To overcome this, we need a real-time violation detection system to help maintain the traffic rules. The approach is to detect traffic violations in real-time using edge computing, which reduces the time to detect. Different machine learning models and algorithms were applied to detect traffic violations like traveling without a helmet, line crossing, parking violation detection, violating the one-way rule etc. The model implemented gave an accuracy of around 85%, due to memory constraints of the edge device in this case NVIDIA Jetson Nano, as the fps is quite low.

A Lightweight Traffic Lights Detection and Recognition Method for Mobile Platform

Traffic lights detection and recognition (TLDR) is one of the necessary abilities of multi-type intelligent mobile platforms such as drones. Although previous TLDR methods have strong robustness in their recognition results, the feasibility of deployment of these methods is limited by their large model size and high requirements of computing power. In this paper, a novel lightweight TLDR method is proposed to improve its feasibility to be deployed on mobile platforms. The proposed method is a two-stage approach. In the detection stage, a novel lightweight YOLOv5s model is constructed to locate and extract the region of interest (ROI). In the recognition stage, the HSV color space is employed along with an extended twin support vector machines (TWSVMs) model to achieve the recognition of multi-type traffic lights including the arrow shapes. The dataset, collected in naturalistic driving experiments with an instrument vehicle, is utilized to train, verify, and evaluate the proposed method. The results suggest that compared with the previous YOLOv5s-based TLDR methods, the model size of the proposed lightweight TLDR method is reduced by 73.3%, and the computing power consumption of it is reduced by 79.21%. Meanwhile, the satisfied reasoning speed and recognition robustness are also achieved. The feasibility of the proposed method to be deployed on mobile platforms is verified with the Nvidia Jetson NANO platform.

The Performance Analysis of PSO-ResNet for the Fault Diagnosis of Vibration Signals Based on the Pipeline Robot.

In the context of pipeline robots, the timely detection of faults is crucial in preventing safety incidents. In order to ensure the reliability and safety of the entire application process, robots' fault diagnosis techniques play a vital role. However, traditional diagnostic methods for motor drive end-bearing faults in pipeline robots are often ineffective when the operating conditions are variable. An efficient solution for fault diagnosis is the application of deep learning algorithms. This paper proposes a rolling bearing fault diagnosis method (PSO-ResNet) that combines a Particle Swarm Optimization algorithm (PSO) with a residual network. A number of vibration signal sensors are placed at different locations in the pipeline robot to obtain vibration signals from different parts. The input to the PSO-ResNet algorithm is a two-bit image obtained by continuous wavelet transform of the vibration signal. The accuracy of this fault diagnosis method is compared with different types of fault diagnosis algorithms, and the experimental analysis shows that PSO-ResNet has higher accuracy. The algorithm was also deployed on an Nvidia Jetson Nano and a Raspberry Pi 4B. Through comparative experimental analysis, the proposed fault diagnosis algorithm was chosen to be deployed on the Nvidia Jetson Nano and used as the core fault diagnosis control unit of the pipeline robot for practical scenarios. However, the PSO-ResNet model needs further improvement in terms of accuracy, which is the focus of future research work.

Design of Fruit-Carrying Monitoring System for Monorail Transporter in Mountain Orchard

The real-time monitoring and detection of the fruit carrying for monorail transporter in the mountain orchard are significant for the transporter scheduling and safety. In this paper, we present a fruit carrying monitoring system, including the pan-tilt camera platform, AI edge computing platform, improved detection algorithm and the web client. The system used a pan-tilt camera to capture images of the truck body of the monorail transporter, realizing monitoring of fruit carrying. Besides, we present an improved fruit carrying detection algorithm based on YOLOv5s, taking the “basket”, “orange” and “fullbasket” as the object. We introduced the improved attention mechanism E-CBAM (Efficient-Convolutional Block Attention Module) based on CBAM, into the C3 module in the neck network of YOLOv5s. Focal loss was introduced to improve the classification and confidence loss to improve detection accuracy; to deploy the model on the embedded platform better, we compressed the model through the EagleEye pruning algorithm to reduce the parameters and improve the detection speed. The experiment was performed on the custom fruit-carrying datasets, the mAP was 91.5%, which was 9.6%, 9.9% and 12.0% higher than that of Faster-RCNN, RetinaNet-Res50 and YOLOv3-tiny, respectively, and detection speed at Jetson Nano was 72[Formula: see text]ms/img. The monitoring system and detection algorithm proposed in the paper can provide technical support for the safe transportation of monorail transporter and scheduling transportation equipment more efficiently.

R-PBFT: A secure and intelligent consensus algorithm for Internet of vehicles

Blockchain applications in the Internet of Vehicle (IoV) employ a secure and distributed consensus method, such as Practical Byzantine Fault Tolerance (PBFT) and Proof-of-Work (PoW), to reduce the load on the central server of IoV and the likelihood of single-node attacks. While traditional consensus algorithms based IoV improves consistency, efficiency, and eliminates single-node attacks, it has some limitations, including a delayed consensus process, restricted transaction frequency, and substantial block congestion. To address these limitations, we propose a fast and intelligent consensus mechanism called R-PBFT (Reputed PBFT). The R-PBFT improves the PBFT consensus process through the use of a reputation mechanism computed by logistic regression. We have built a testbed using the Raspberry Pi 4 and NVIDIA Jetson Nano platforms and experimented with the proposed scheme. The experiment result shows that the proposed R-PBFT outperforms state-of-the-art approaches by minimizing consensus delay by 50%, improving transaction frequency by 24% and throughput by 50%, reducing block congestion by 57%, and improves fairness by 15%. According to security analysis, R-PBFT is secure against Finney, Sybil, eclipse, routing, and impersonation attacks.

A Fast and Accurate Method for Classifying Tomato Plant Health Status Using Machine Learning and Image Processing

Agriculture is crucial to economic growth and development, and maintaining high-quality, disease-free plants is crucial to its success. Early detection of plant diseases, which can be caused by environmental factors, fungi, bacteria, and viruses, is essential to implement appropriate treatments. Tomatoes, which are one of the most vital food crops, are susceptible to diseases that can result in significant economic losses in agriculture. This study introduces a method to evaluate the health of tomato leaf using image processing techniques and machine learning algorithms. A dataset of 1,778 images of healthy and infected tomato leaves was collected from tomato planting areas in the Turkish provinces of Samsun and Mersin. Sixteen advanced machine learning algorithms were used for classification, and the optimal hyperparameters for each algorithm were determined using a grid search approach. The classifiers were executed on Jetson Nano and TX2 embedded systems. The experimental results indicate that the Random Forest classifier outperformed other algorithms, achieving approximately 99 % accuracy in detecting and classifying the health status of tomato leaves. The proposed system enables faster and more accurate detection, allowing farmers to classify plants as infected or healthy, ultimately improving decision-making on treatment and pest management strategies.

Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study.

The article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processes. We present a system designed to facilitate collaboration between humans and robots in manufacturing processes that require frequent revisions of the robot path and that allows direct definition of the waypoints, which differentiates our system from the existing ones. We introduce a novel and intuitive approach to human-robot cooperation through the use of simple gestures. As part of a robotic workspace, a proposed interface was developed and implemented utilising three RGB-D sensors for monitoring the operator's hand movements within the workspace. The system employs distributed data processing through multiple Jetson Nano units, with each unit processing data from a single camera. MediaPipe solution is utilised to localise the hand landmarks in the RGB image, enabling gesture recognition. We compare the conventional methods of defining robot trajectories with their developed gesture-based system through an experiment with 20 volunteers. The experiment involved verification of the system under realistic conditions in a real workspace closely resembling the intended industrial application. Data collected during the experiment included both objective and subjective parameters. The results indicate that the gesture-based interface enables users to define a given path objectively faster than conventional methods. We critically analyse the features and limitations of the developed system and suggest directions for future research. Overall, the experimental results indicate the usefulness of the developed system as it can speed up the definition of the robot's path.

Heterogeneous Fusion of Camera and mmWave Radar Sensor of Optimizing Convolutional Neural Networks for Parking Meter System.

In this article, a novel heterogeneous fusion of convolutional neural networks that combined an RGB camera and an active mmWave radar sensor for the smart parking meter is proposed. In general, the parking fee collector on the street outdoor surroundings by traffic flows, shadows, and reflections makes it an exceedingly tough task to identify street parking regions. The proposed heterogeneous fusion convolutional neural networks combine an active radar sensor and image input with specific geometric area, allowing them to detect the parking region against different tough conditions such as rain, fog, dust, snow, glare, and traffic flow. They use convolutional neural networks to acquire output results along with the individual training and fusion of RGB camera and mmWave radar data. To achieve real-time performance, the proposed algorithm has been implemented on a GPU-accelerated embedded platform Jetson Nano with a heterogeneous hardware acceleration methodology. The experimental results exhibit that the accuracy of the heterogeneous fusion method can reach up to 99.33% on average.

Study of convolutional neural network-based semantic segmentation methods on edge intelligence devices for field agricultural robot navigation line extraction

Smart agricultural machinery is emerging as the future trend for field robots, and the fully automatic robot has a great application prospect. However, it is a big challenge for robots to navigate in complex farmland environments. In this research, 5 deep learning-based computer vision methods under different field scenes for field navigation line extraction were studied and successfully deployed on an embedded system, which can be integrated into robots for automatic navigation in the future. The field road was segmented by the semantic segmentation algorithm at first, and then the navigation line is extracted from the segmented image by a polygon fitting method. Finally, all the models are transformed through the TensorRT library and deployed on the edge computing device Jetson Nano. In the experiment, five reprehensive semantic segmentation networks namely UNet, Deeplabv3+, BiseNetv1, BiseNetv2, and ENet networks were selected. Among the five networks, Deeplabv3+ is the most accurate. In five scenes, its average segmentation accuracy is 84.87 %, and the navigation line error is 9.59 pixels. Especially in the third scene with shadow and occlusion, it performs best, with only 8.34 pixel error, But the speed of Deeplabv3+ is only 9.7 FPS. ENet, BiseNetv1, and BiseNetv2 are lightweight networks. The speed of ENet is 16.8 FPS, BiseNetv2 is 17 FPS, and BiseNetv1 is 15.8 FPS. In segmentation accuracy and navigation line error, ENet performs better than BiseNet series networks, which are 84.94 % and 10.73 pixels, respectively. In the third scene with shadow and occlusion, it also performs slightly better than BiseNet series networks. In summary, deep learning-based semantic segmentation methods have strong robustness and stability in complex environment compared with previous research. Among all currently available neural networks, ENet has the best performance and good application potential in field navigation.

Ultrasonic testing of rivet in multilayer structure using a convolutional neural network on edge device.

Rivets are used to assemble layers in the air intakes, fuselages, and wings of an aircraft. After a long time of working under extreme conditions, pitting corrosion could appear in the rivets of the aircraft. The rivets could be broken down and thread the safety of the aircraft. In this paper, we proposed an ultrasonic testing method integrated with convolutional neural network (CNN) for the detection of corrosion in the rivets. The CNN model was designed to be lightweight enough to be able to run on edge devices. The CNN model was trained with a very limited sample of rivets, from 3 to 9 artificial pitting corrosive rivets. The results show that the proposed approach could detect up to 95.2% of pitting corrosion using experimental data with three training rivets. The detection accuracy could be improved to 99% by nine training rivets. The CNN model was implemented and ran on an edge device (Jetson Nano) in real-time with a small latency of 1.65 ms.

Bringing Online Egocentric Action Recognition Into the Wild

To enable a safe and effective human-robot cooperation, it is crucial to develop models for the identification of human activities. Egocentric vision seems to be a viable solution to solve this problem, and therefore many works provide deep learning solutions to infer human actions from first person videos. However, although very promising, most of these do not consider the major challenges that comes with a realistic deployment, such as the portability of the model, the need for real-time inference, and the robustness with respect to the novel domains (i.e., new spaces, users, tasks). With this letter, we set the boundaries that egocentric vision models should consider for realistic applications, defining a novel setting of egocentric action recognition in the wild, which encourages researchers to develop novel, applications-aware solutions. We also present a new model-agnostic technique that enables the rapid repurposing of existing architectures in this new context, demonstrating the feasibility to deploy a model on a tiny device (Jetson Nano) and to perform the task directly on the edge with very low energy consumption (2.4 W on average at 50 fps).

A Hand Hygiene Tracking System With LoRaWAN Network for the Abolition of Hospital-Acquired Infections

In healthcare settings, hand hygiene is a crucial step in preventing the spread of disease. Healthcare-associated infections, such as community-acquired pneumonia and hospital-acquired pneumonia, can be mitigated with the help of cutting-edge solutions and intelligent alcohol-based hand sanitizers. Specifically, this research presents a Long-Range network (LoRa) powered by the Internet of Things that can detect, monitor and track hand hygiene habits to reduce the spread of HAIs. The wearable Bluetooth Low Energy (BLE) device used by hospital personnel and patients will monitor and track the hygiene activities using LoRa and alcohol-based sanitizers deployed in ICUs, elevators, inpatient rooms, and other hospital facilities. The alcohol dispenser is equipped with a Jetson Nano Graphics Processing Unit (GPU) from NVIDIA, a hand sanitizer, and a door that opens once the user has washed their hands. Using a LoRa Wide Area Network (LoRaWAN) and short-range BLE technology, hand hygiene information, with a user ID and GPS location, has sent to a LoRa gateway and a cloud server. Various field tests in hospital settings and simulated environments are performed to effectively guarantee the clinical performance of the proposed IoT-enabled LoRa network. In conclusion, the total success percentage in real-world LoRa network conditions is 92.78 percent, whereas the success rate in laboratory testing is 98 percent.

Analysis of facial emotion recognition rate for real-time application using NVIDIA Jetson Nano in deep learning models

Detecting <span>facial emotion expression is a classic research problem in image processing. Face expression detection can be used to help human users monitor their stress levels. Perceiving an individual's failure to communicate specific looks might help analyze early psychological disorders. several issues like lighting changes, rotations, occlusions, and accessories persist. These are not simply traditional image processing issues, yet additionally, action units that make gathering activity of facial acknowledgment troublesome look information, and order of the demeanor. In this study, we use Xception taking into account Xception and convolution neural network (CNN), which is easy to focus on incredible parts like the face, and visual geometric group (VGG-19) used to extract the facial feature using the OpenCV framework classifying the image into any of the basic facial emotions. NVIDIA Jetson Nano has a high video handling outline rate. Accomplishing preferable precision over the recently evolved models on software. The average accuracies for standard data set CK+,” on NVIDIA Jetson Nano, the accuracy rate is 97.1% in the Xception model in the convolutional neural network, 98.4% in VGG-19, and real-time environment accuracy using OpenCV, accuracy rate is 95.6%.</span>

Deep Learning-based Bangla Sign Language Detection with an Edge Device

Sign language, often referred to as silent conversation, serves as a visual gesture-based primary communication medium for hearing-impaired individuals. People unable to speak or hear communicate with themselves and general persons with static and dynamic sign languages. To ensure that people who are deaf and unable to talk do not feel excluded from this fast-paced world, it is even more crucial that sign language is incorporated into our culture and that a cost-effective and straightforward method of detecting it be developed. With this goal in mind, an automatic Bangla sign language (BSL) detection system has been developed in this work using deep learning approaches and a Jetson Nano edge device. The deep learning models have been trained with Okkhornama, an open-source database and a custom author-curated dataset of 49 categories and 3,760 images have been used to validate these techniques. These images are preprocessed by auto-orienting and resizing into 416 × 416 pixels in the Roboflow framework. Next, we implemented Detectron2, EfficientDet-D0 with TensorFlow and the PyTorch-built YOLOv7 models. Jetson Nano, a portable and sufficiently dynamic NVIDIA microprocessor, is also utilized in this project to generate an object detection model and infer it from test images in real-time. The Dectectron2 model performed best concerning detection accuracy, with a mAP@.5 of 94.915 and an AP of 54.814. The YOLOv7 model achieved mAP@.5 accuracies in detecting various classes from 85 percent to 97 percent and mAP@.5-.95 from 41 percent to 53 percent. Finally, the YOLOv7 Tiny approach has been used in the Jetson Nano edge device for real-time Bangla sign language detection because of its lowest total training time and frame rate. The proposed system is expected to provide an effective solution for Bangladeshi people suffering from hearing disabilities and the inability to speak in a simple and affordable approach.

A LIGHTWEIGHT MULTI-PERSON POSE ESTIMATION SCHEME BASED ON JETSON NANO

As the basic technology of human action recognition, pose estimation is attracting more and more researchers' attention, while edge application scenarios pose a higher challenge. This paper proposes a lightweight multi-person pose estimation scheme to meet the needs of real-time human action recognition on the edge end. This scheme uses AlphaPose to extract human skeleton nodes, and adds ResNet and Dense Upsampling Revolution to improve its accuracy. Meanwhile, we use YOLO to enhance AlphaPose’s support for multi-person pose estimation, and optimize the proposed model with TensorRT. In addition, this paper sets Jetson Nano as the Edge AI deployment device of the proposed model and successfully realizes the model migration to the edge end. The experimental results show that the speed of the optimized object detection model can reach 20 FPS, and the optimized multi-person pose estimation model can reach 10 FPS. With the image resolution of 320×240, the model’s accuracy is 73.2%, which can meet the real-time requirements. In short, our scheme can provide a basis for lightweight multi-person action recognition scheme on the edge end.

Benchmarking of Contactless Heart Rate Measurement Systems in ARM-Based Embedded Platforms

Heart rate monitoring is especially important for aging individuals because it is associated with longevity and cardiovascular risk. Typically, this vital parameter can be measured using wearable sensors, which are widely available commercially. However, wearable sensors have some disadvantages in terms of acceptability, especially when used by elderly people. Thus, contactless solutions have increasingly attracted the scientific community in recent years. Camera-based photoplethysmography (also known as remote photoplethysmography) is an emerging method of contactless heart rate monitoring that uses a camera and a processing unit on the hardware side, and appropriate image processing methodologies on the software side. This paper describes the design and implementation of a novel pipeline for heart rate estimation using a commercial and low-cost camera as the input device. The pipeline's performance was tested and compared on a desktop PC, a laptop, and three different ARM-based embedded platforms (Raspberry Pi 4, Odroid N2+, and Jetson Nano). The results showed that the designed and implemented pipeline achieved an average accuracy of about 96.7% for heart rate estimation, with very low variance (between 1.5% and 2.5%) across processing platforms, user distances from the camera, and frame resolutions. Furthermore, benchmark analysis showed that the Odroid N2+ platform was the most convenient in terms of CPU load, RAM usage, and average execution time of the algorithmic pipeline.

Application of Machine Learning for Insect Monitoring in Grain Facilities

In this study, a basic insect detection system consisting of a manual-focus camera, a Jetson Nano—a low-cost, low-power single-board computer, and a trained deep learning model was developed. The model was validated through a live visual feed. Detecting, classifying, and monitoring insect pests in a grain storage or food facility in real time is vital to making insect control decisions. The camera captures the image of the insect and passes it to a Jetson Nano for processing. The Jetson Nano runs a trained deep-learning model to detect the presence and species of insects. With three different lighting situations: white LED light, yellow LED light, and no lighting condition, the detection results are displayed on a monitor. Validating using F1 scores and comparing the accuracy based on light sources, the system was tested with a variety of stored grain insect pests and was able to detect and classify adult cigarette beetles and warehouse beetles with acceptable accuracy. The results demonstrate that the system is an effective and affordable automated solution to insect detection. Such an automated insect detection system can help reduce pest control costs and save producers time and energy while safeguarding the quality of stored products.

Projection operator‐based robust adaptive control of an aerial robot with a manipulator

AbstractThis paper describes a quadcopter manipulator system, an aerial robot with an extended workspace, its controller design, and experimental validation. The aerial robot is based on a quadcopter with a three degree of freedom robotic arm connected to the base of the vehicle. The work aims to create a stable airborne robot with a robotic arm that can work above and below the airframe, regardless of where the arm is attached. Integrating a robotic arm into an underactuated, unstable system like a quadcopter can enhance the vehicle's functionality while increasing instability. To execute a mission with accuracy and reliability during a real‐time task, the system must overcome the inter‐coupling effects and external disturbances. This work presents a novel design for a robust adaptive feedback linearization controller with a model reference adaptive controller and hardware implementation of the quadcopter manipulator system with plant uncertainties. The closed‐loop stability of the aerial robot and the tracking error convergence with the robust controller is analyzed using Lyapunov stability analysis. The quadcopter manipulator system is custom developed in the lab with an off‐the‐shelf quadcopter and a 3D‐printed robotic arm. The robotic system architecture is implemented using a Jetson Nano companion computer for autonomous onboard flight. Experiments were conducted on quadcopter manipulator system to evaluate the autonomous aerial robot's stability and trajectory tracking with the proposed controller.

Pedestrian Detection with LiDAR Technology in Smart-City Deployments–Challenges and Recommendations

This paper describes a real case implementation of an automatic pedestrian-detection solution, implemented in the city of Aveiro, Portugal, using affordable LiDAR technology and open, publicly available, pedestrian-detection frameworks based on machine-learning algorithms. The presented solution makes it possible to anonymously identify pedestrians, and extract associated information such as position, walking velocity and direction in certain areas of interest such as pedestrian crossings or other points of interest in a smart-city context. All data computation (3D point-cloud processing) is performed at edge nodes, consisting of NVIDIA Jetson Nano and Xavier platforms, which ingest 3D point clouds from Velodyne VLP-16 LiDARs. High-performance real-time computation is possible at these edge nodes through CUDA-enabled GPU-accelerated computations. The MQTT protocol is used to interconnect publishers (edge nodes) with consumers (the smart-city platform). The results show that using currently affordable LiDAR sensors in a smart-city context, despite the advertising characteristics referring to having a range of up to 100 m, presents great challenges for the automatic detection of objects at these distances. The authors were able to efficiently detect pedestrians up to 15 m away, depending on the sensor height and tilt. Based on the implementation challenges, the authors present usage recommendations to get the most out of the used technologies.

Pedestrian detection model based on Tiny-Yolov3 architecture for wearable devices to visually impaired assistance.

Introduction: Wearable assistive devices for the visually impaired whose technology is based on video camera devices represent a challenge in rapid evolution, where one of the main problems is to find computer vision algorithms that can be implemented in low-cost embedded devices. Objectives and Methods: This work presents a Tiny You Only Look Once architecture for pedestrian detection, which can be implemented in low-cost wearable devices as an alternative for the development of assistive technologies for the visually impaired. Results: The recall results of the proposed refined model represent an improvement of 71% working with four anchor boxes and 66% with six anchor boxes compared to the original model. The accuracy achieved on the same data set shows an increase of 14% and 25%, respectively. The F1 calculation shows a refinement of 57% and 55%. The average accuracy of the models achieved an improvement of 87% and 99%. The number of correctly detected objects was 3098 and 2892 for four and six anchor boxes, respectively, whose performance is better by 77% and 65% compared to the original, which correctly detected 1743 objects. Discussion: Finally, the model was optimized for the Jetson Nano embedded system, a case study for low-power embedded devices, and in a desktop computer. In both cases, the graphics processing unit (GPU) and central processing unit were tested, and a documented comparison of solutions aimed at serving visually impaired people was performed. Conclusion: We performed the desktop tests with a RTX 2070S graphics card, and the image processing took about 2.8ms. The Jetson Nano board could process an image in about 110ms, offering the opportunity to generate alert notification procedures in support of visually impaired mobility.