NVIDIA Jetson Research Articles

A robust and real-time lane detection method in low-light scenarios to advanced driver assistance systems

Lane detection, which relies on front-view RGB cameras, is a crucial aspect of Advanced Driver Assistance Systems (ADAS), but its effectiveness is notably reduced in low-light conditions. This issue is exacerbated by the lack of specialized datasets and generalizable methods for such scenarios. To address this gap, we introduce NightLane, a comprehensive dataset tailored for low-light, multi-traffic lane detection. We adhere to stringent data annotation guidelines, ensuring reliable detection accuracy. Additionally, we propose the Fused Low-Light Enhancement Framework (FLLENet), which leverages modern detection networks and incorporates a low-light enhancement module and attention mechanisms. The enhancement module, based on zero-reference learning, improves image quality and channel richness, while the attention mechanisms effectively extract and utilize these features. Our extensive testing on NightLane and CULane datasets demonstrates superior performance in low-light lane detection, showcasing FLLENet’s robust generalizability and efficacy. Specifically, our approach achieves an F1 measure of 76.90 on CULane and 78.91 on NightLane, demonstrating its effectiveness against state-of-the-art methods. We also evaluate the real-time applicability of our framework on a low-power embedded lane detection system using NVIDIA Jetson AGX/Orin, achieving high accuracy and real-time performance. Our work offers a new approach and reference in the field of low-light lane detection, potentially aiding in the ongoing enhancement of ADAS (ADAS). Dateset are available at https://github.com/pengjingt/FLLENet.

Real-Time Ego-Lane Detection in a Low-Cost Embedded Platform using CUDA-Based Implementation

This work assesses the effectiveness of heterogeneous computing based on a CUDA implementation for real-time ego-lane detection using a typical low-cost embedded computer. We propose and evaluate a CUDA-optimized algorithm using a heterogeneous approach based on the extraction of features from an aerial perspective image. The method incorporates well-known algorithms optimized to achieve a very efficient solution with high detection rates and combines techniques to enhance markings and remove noise. The CUDA-based solution is compared to an OpenCV library and to a serial CPU implementation. Practical experiments using TuSimple's image datasets were conducted in an NVIDIA's Jetson Nano embedded computer. The algorithm detects up to 97.9% of the ego lanes with an accuracy of 99.0% in the best-evaluated scenario. Furthermore, the CUDA-optimized method performs at rates greater than 300 fps in the Jetson Nano embedded system, speeding up 25 and 140 times the OpenCV and CPU implementations at the same platform, respectively. These results show that more complex algorithms and solutions can be employed for better detection rates while maintaining real-time requirements in a typical low-power embedded computer using a CUDA implementation.

Optimizing Appearance-Based Localization with Catadioptric Cameras: Small-Footprint Models for Real-Time Inference on Edge Devices.

This paper considers the task of appearance-based localization: visual place recognition from omnidirectional images obtained from catadioptric cameras. The focus is on designing an efficient neural network architecture that accurately and reliably recognizes indoor scenes on distorted images from a catadioptric camera, even in self-similar environments with few discernible features. As the target application is the global localization of a low-cost service mobile robot, the proposed solutions are optimized toward being small-footprint models that provide real-time inference on edge devices, such as Nvidia Jetson. We compare several design choices for the neural network-based architecture of the localization system and then demonstrate that the best results are achieved with embeddings (global descriptors) yielded by exploiting transfer learning and fine tuning on a limited number of catadioptric images. We test our solutions on two small-scale datasets collected using different catadioptric cameras in the same office building. Next, we compare the performance of our system to state-of-the-art visual place recognition systems on the publicly available COLD Freiburg and Saarbrücken datasets that contain images collected under different lighting conditions. Our system compares favourably to the competitors both in terms of the accuracy of place recognition and the inference time, providing a cost- and energy-efficient means of appearance-based localization for an indoor service robot.

A Review of Embedded Machine Learning Based on Hardware, Application, and Sensing Scheme.

Machine learning is an expanding field with an ever-increasing role in everyday life, with its utility in the industrial, agricultural, and medical sectors being undeniable. Recently, this utility has come in the form of machine learning implementation on embedded system devices. While there have been steady advances in the performance, memory, and power consumption of embedded devices, most machine learning algorithms still have a very high power consumption and computational demand, making the implementation of embedded machine learning somewhat difficult. However, different devices can be implemented for different applications based on their overall processing power and performance. This paper presents an overview of several different implementations of machine learning on embedded systems divided by their specific device, application, specific machine learning algorithm, and sensors. We will mainly focus on NVIDIA Jetson and Raspberry Pi devices with a few different less utilized embedded computers, as well as which of these devices were more commonly used for specific applications in different fields. We will also briefly analyze the specific ML models most commonly implemented on the devices and the specific sensors that were used to gather input from the field. All of the papers included in this review were selected using Google Scholar and published papers in the IEEExplore database. The selection criterion for these papers was the usage of embedded computing systems in either a theoretical study or practical implementation of machine learning models. The papers needed to have provided either one or, preferably, all of the following results in their studies-the overall accuracy of the models on the system, the overall power consumption of the embedded machine learning system, and the inference time of their models on the embedded system. Embedded machine learning is experiencing an explosion in both scale and scope, both due to advances in system performance and machine learning models, as well as greater affordability and accessibility of both. Improvements are noted in quality, power usage, and effectiveness.

Embedded Federated Learning for VANET Environments

In the scope of smart cities, the sensors scattered throughout the city generate information that supplies intelligence mechanisms to learn the city’s mobility patterns. These patterns are used in machine learning (ML) applications, such as traffic estimation, that allow for improvement in the quality of experience in the city. Owing to the Internet-of-Things (IoT) evolution, the city’s monitoring points are always growing, and the transmission of the mass of data generated from edge devices to the cloud, required by centralized ML solutions, brings great challenges in terms of communication, thus negatively impacting the response time and, consequently, compromising the reaction in improving the flow of vehicles. In addition, when moving between the edge and the cloud, data are exposed, compromising privacy. Federated learning (FL) has emerged as an option for these challenges: (1) It has lower latency and communication overhead when performing most of the processing on the edge devices; (2) it improves privacy, as data do not travel over the network; and (3) it facilitates the handling of heterogeneous data sources and expands scalability. To assess how FL can effectively contribute to smart city scenarios, we present an FL framework, for which we built a testbed that integrated the components of the city infrastructure, where edge devices such as NVIDIA Jetson were connected to a cloud server. We deployed our lightweight container-based FL framework in this testbed, and we evaluated the performance of devices, the effectiveness of ML and aggregation algorithms, the impact on the communication between the edge and the server, and the consumption of resources. To carry out the evaluation, we opted for a scenario in which we estimated vehicle mobility inside and outside the city, using real data collected by the Aveiro Tech City Living Lab communication and sensing infrastructure in the city of Aveiro, Portugal.

Machine Learning Based Modeling for Real-Time Inferencer-in-the-Loop Hardware Emulation of High-Speed Rail Microgrid

The application of artificial intelligence (AI) technology in the field of power systems and power electronic devices is increasingly prevalent. With massive datasets generated by a wide range of equipment, AI-based modeling is promising in the future of hardware-in-the-loop emulation. This article studies and improves the machine learning based modeling approach for power electronic devices, and the inferencer-in-the-loop (IIL) system is proposed together with optimized neural network (NN) models. The high-speed rail microgrid, includes autotransformer rectifier unit subsystems, energy storage subsystems, two-level converter based permanent magnet synchronous motor propulsion subsystems, and modular multilevel converter based induction motor propulsion subsystems, serves as study cases to demonstrate the adaptability of this approach. Finally, to show high accuracy and versatility of the IIL real-time emulation system, the system-level (1 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\boldsymbol{\mu} \mathbf{s}$</tex-math></inline-formula> timestep) and device-level (50 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\text{ns}}$</tex-math></inline-formula> timestep) results are compared in three domains: the referencer system (C code simulation program in NVIDIA Jetson and offline SaberRD datasets), offline inferencer emulation on Xilinx VCU118 board, and online refined inferencer emulation on Xilinx VCU118 board.

Inference replication at edges via combinatorial multi-armed bandit

Inferences easily incur computation overload at the edge of the network, since they often consume plenty of resources and are often implemented by using deep neural networks (DNNs). Traditional approach via offloading those inference tasks to remote cloud is unsuitable, since the round-trip time is often a burden. As a result, offloading by using nearby idle edges is promising, which sacrifices the task replication overhead for speeding up the edge inference. Unfortunately, due to stochastic changes on both edge networks and edge inference, it is hard to determine the best suitable targets for replication, especially when those DNNs consist of multiple kernels for inference, the replication decision involves multiple edge candidates as the destinations, and the edges are further heterogeneous. In this paper, we propose to optimize the inference replication at edges, under the consideration of stochastic changes. We formulate related problem and design an online algorithm via combinatorial multi-armed bandit for the inference with minimum response time, which decides multiple destinations simultaneously for replication, upon both revealed feedback after the deployment and the offline profile. By rigorous proof, the sublinear regret is ensured, which measures the gap between our online decision and the offline optimum. Through extensive trace-driven experiments with Huawei Atlas and NVIDIA Jetson, the improvement earned by inference replication is confirmed, compared with other alternatives.

Real-time recognition system of soybean seed full-surface defects based on deep learning

Accurately sorting high-quality soybean seeds is a key task in increasing soybean yield in the breeding industry. At present, sorting systems based on machine vision focus on the recognition of only one side surface. This paper designs and develops a sorting system based on deep learning that can recognize the full surface of soybean seeds. An alternate circumrotating mechanism is used to expose the full surface feature information of the seeds, and a deep learning model is applied for accurate seed classification. We divide soybean seeds into six categories. Images are collected and masked in three brightness environments and six surfaces, and a defect scale of soybean seeds is quantified. We compare and test seven CNN models and improve the model with the best overall performance. Visualization technology is used to assess the recognition performance of different models for soybean seed defects, and the model is optimized based on the results to achieve accurate classification of seed defects at different scales. The testing process indicates that all the models have the highest accuracy under medium brightness conditions. The classification accuracy of the MobileNetV2-improved model reaches 97.84% in the masked dataset and has an inference speed of 35 FPS with NVIDIA's Jetson Nano board, realizing real-time recognition of the soybean full surface. The sorting system proposed in this paper can achieve high-precision and low-cost application, with a total sorting accuracy of 98.87% and a sorting speed of 222 seeds per minute. This method can be used as an effective tool for precise sorting of soybean seeds. Moreover, it provides an approach for full-surface detection of defective ellipsoid seeds of different scales.

The Untapped Social Impact of Artificial Intelligence for Breast Cancer Screening in Developing Countries: A Critical Commentary of DeepMind

The Untapped Social Impact of Artificial Intelligence for Breast Cancer Screening in Developing Countries: A Critical Commentary of DeepMind

Real-Time Human Detection Using Deep Learning on Embedded Platforms: A Review

The detection of an object such as a human is very important for image understanding in the field of computer vision. Human detection in images can provide essential information for a wide variety of applications in intelligent systems. In this paper, human detection is carried out using deep learning that has developed rapidly and achieved extraordinary success in various object detection implementations. Recently, several embedded systems have emerged as powerful computing boards to provide high processing capabilities using the graphics processing unit (GPU). This paper aims to provide a comprehensive survey of the latest achievements in this field brought about by deep learning techniques in the embedded platforms. NVIDIA Jetson was chosen as a low power system designed to accelerate deep learning applications. This review highlights the performance of human detection models such as PedNet, multiped, SSD MobileNet V1, SSD MobileNet V2, and SSD inception V2 on edge computing. This survey aims to provide an overview of these methods and compare their performance in accuracy and computation time for real-time applications. The experimental results show that the SSD MobileNet V2 model provides the highest accuracy with the fastest computation time compared to other models in our video datasets with several scenarios.

Real-Time Human Detection Using Deep Learning on Embedded Platforms: A Review

The detection of an object such as a human is very important for image understanding in the field of computer vision. Human detection in images can provide essential information for a wide variety of applications in intelligent systems. In this paper, human detection is carried out using deep learning that has developed rapidly and achieved extraordinary success in various object detection implementations. Recently, several embedded systems have emerged as powerful computing boards to provide high processing capabilities using the graphics processing unit (GPU). This paper aims to provide a comprehensive survey of the latest achievements in this field brought about by deep learning techniques in the embedded platforms. NVIDIA Jetson was chosen as a low power system designed to accelerate deep learning applications. This review highlights the performance of human detection models such as PedNet, multiped, SSD MobileNet V1, SSD MobileNet V2, and SSD inception V2 on edge computing. This survey aims to provide an overview of these methods and compare their performance in accuracy and computation time for real-time applications. The experimental results show that the SSD MobileNet V2 model provides the highest accuracy with the fastest computation time compared to other models in our video datasets with several scenarios.

Understanding the Perception of Road Segmentation and Traffic Light Detection u sing Machine Learning Techniques

Advanced Driving Assistance System (ADAS) has seen tremendous growth over the past 10 years. In recent times, luxury cars, as well as some newly emerging cars, come with ADAS application. From 2014, Because of the entry of the European new car assessment programme (EuroNCAP) [1] in the AEBS test, it helped gain momentum the introduction of ADAS in Europe [1]. Most OEMs and research institutes have already demonstrated on the self-driving cars [1]. So here, a focus is made on road segmentation where LiDAR sensor takes in the image of the surrounding and where the vehicle should know its path, it is fulfilled by processing a convolutional neural network called semantic segmentation on an FPGA board in 16.9ms [3]. Further, a traffic light detection model is also developed by using NVidia Jetson and 2 FPGA boards, collectively named as 'Driving brain' which acts as a super computer for such networks. The results are obtained at higher accuracy by processing the obtained traffic light images into the CNN classifier [5]. Overall, this paper gives a brief idea of the technical trend of autonomous driving which throws light on algorithms and for advanced driver-assistance systems used for road segmentation and traffic light detection.

Deep CNN-Based Real-Time Traffic Light Detector for Self-Driving Vehicles

Due to the unavailability of Vehicle-to-Infrastructure (V2I) communication in current transportation systems, Traffic Light Detection (TLD) is still considered an important module in autonomous vehicles and Driver Assistance Systems (DAS). To overcome low flexibility and accuracy of vision-based heuristic algorithms and high power consumption of deep learning-based methods, we propose a lightweight and real-time traffic light detector for the autonomous vehicle platform. Our model consists of a heuristic candidate region selection module to identify all possible traffic lights, and a lightweight Convolution Neural Network (CNN) classifier to classify the results obtained. Offline simulations on the GPU server with the collected dataset and several public datasets show that our model achieves higher average accuracy and less time consumption. By integrating our detector module on NVidia Jetson TX1/TX2, we conduct on-road tests on two full-scale self-driving vehicle platforms (a car and a bus) in normal traffic conditions. Our model can achieve an average detection accuracy of 99.3 percent (mRttld) and 99.7 percent (Rttld) at 10Hz on TX1 and TX2, respectively. The on-road tests also show that our traffic light detection module can achieve $ ± 1 . 5 m errors at stop lines when working with other self-driving modules.

Real-Time Apple Detection System Using Embedded Systems With Hardware Accelerators: An Edge AI Application

Real-time apple detection in orchards is one of the most effective ways of estimating apple yields, which helps in managing apple supplies more effectively. Traditional detection methods used highly computational machine learning algorithms with intensive hardware set up, which are not suitable for infield real-time apple detection due to their weight and power constraints. In this study, a real-time embedded solution inspired from "Edge AI" is proposed for apple detection with the implementation of YOLOv3-tiny algorithm on various embedded platforms such as Raspberry Pi 3 B+ in combination with Intel Movidius Neural Computing Stick (NCS), Nvidia's Jetson Nano and Jetson AGX Xavier. Data set for training were compiled using acquired images during field survey of apple orchard situated in the north region of Italy, and images used for testing were taken from widely used google data set by filtering out the images containing apples in different scenes to ensure the robustness of the algorithm. The proposed study adapts YOLOv3-tiny architecture to detect small objects. It shows the feasibility of deployment of the customized model on cheap and power-efficient embedded hardware without compromising mean average detection accuracy (83.64%) and achieved frame rate up to 30 fps even for the difficult scenarios such as overlapping apples, complex background, less exposure of apple due to leaves and branches. Furthermore, the proposed embedded solution can be deployed on the unmanned ground vehicles to detect, count, and measure the size of the apples in real-time to help the farmers and agronomists in their decision making and management skills.