NVIDIA TX2 Research Articles

Brain magnetic resonance imaging image classification for Alzheimer's disease and its hardware acceleration

<span lang="EN-US">Alzheimer's is a progressive neurodegenerative disorder and is considered the sixth leading cause of death after cancer and heart attack. Early detection and diagnosis provide individuals to go through a wider variety of clinical trials and get multiple medical benefits. Research on the application of deep learning and machine learning to the early detection of Alzheimer's disease has recently gained considerable attention. In this paper, we propose a deep learning classification framework to classify the individual with different progression stages of Alzheimer's disease such as mild cognitive impairment (MCI) and cognitive normal (CN). The dataset from Alzheimer’s disease neuroimaging initiative (ADNI) is considered in this paper which is a multisite having collection of Neuroimaging data for researchers. Structural magnetic resonance imaging (MRI) images are considered from the ADNI data set and feature extraction is done using a 2D discrete wavelet transform. 97% of data reduction is achieved during data pre-processing. The algorithm is trained and validated. The algorithm is accelerated in Nvidia Tx2 graphics processing unit (GPU) to get the better throughput. The result shows our algorithm outperforms the other deep learning algorithms with 91.56% accuracy. </span>

Sugar Beet Seed Classification for Production Quality Improvement by Using YOLO and NVIDIA Artificial Intelligence Boards

AbstractAll inputs are required for excellent and proper crop production, especially seed quality. In this way fewer disease and insect issues, increased seedling germination, uniform plant population and maturity, and better responsiveness to fertilizers and nutrients, leading to higher returns per unit area and profitability, and low labor costs could be possible. Because of this reason, NVIDIA Jetson Nano and TX2 artificial intelligence boards were used to test the efficiency of the YOLOv4 and YOLOv4-tiny models for sugar beet monogerm and multigerm seed classification for better production. YOLOv4-tiny outscored the other model based on FPS with 8.25–8.37 at NVIDIA Jetson Nano, 12.11–12.36 at NVIDIA TX2 artificial intelligence boards with accuracy 81–99% for monogerm seeds, and 89–99% for multigerm seeds at NVIDIA Jetson Nano, 88–99% for monogerm seeds, and 90–99% for multigerm at NVIDIA TX2 accuracy, respectively, implying that the YOLOv4 is more accurate but slow with based on FPS with 1.10–1.21 at NVIDIA Jetson Nano, 2.41–2.43 at NVIDIA TX2 artificial intelligence boards with 95–99% for monogerm seeds and 95–100% for multigerm seeds at NVIDIA Jetson Nano, 92–99% for monogerm seeds and 98–100% for multigerm seeds at NVIDIA TX2, respectively. As a result of the evaluations, NVIDIA Artificial Intelligence cards and YOLO deep learning model will be used effectively in classifying monogerm and multigerm sugar beet seeds, thus reducing seed loss with the help of NVIDIA Artificial Intelligence cards classification.

AeroNet: An efficient relative localization and object detection network for cooperative aerial-ground unmanned vehicles

This paper proposes an efficient relative localization and object detection network (AeroNet) based on incremental learning for minimalistic high-speed cooperative navigation of aerial-ground unmanned vehicles in cluttered environments. Due to highly limited computation capability and memory resources in micro-UAVs, YOLO series are applied as the baseline of object detection network, and a lightweight backbone is built based on the depthwise separable convolution. To improve the real-time performance, the detection head is formulated with broad learning system. Besides, 6D relative pose estimation is achieved via equation fitting of an elliptical cooperative mark. To verify the effectiveness of AeroNet, experiments are conducted on Intel NUC and NVIDIA TX2 with our self-collected dataset. Results show that AeroNet can progressively increase the accuracy of object detection to 89%, and the computational time is only 76ms on Intel NUC and 28ms on Nvidia TX2, respectively, which meet the need of real-time requirement of on-board calculation in micro-UAV avionics systems.

An Efficient Incremental Learning Algorithm for Sound Classification

This article proposes an efficient audio incremental learning method to reduce the computational complexity and catastrophic forgetting during the incremental addition of the audio data in Deep Neural Networks (DNN). The computational complexity is reduced by performing training of only fully-connected layers and catastrophic forgetting is reduced by sharing the knowledge from the old learned classes without using previously learned data. Our method has been evaluated extensively on UrbanSound8K, ESC-10, and TUT datasets where state-of-the-art accuracies have been achieved. Moreover, our method has been evaluated on Nvidia 1080-ti GPU, Nvidia TX-2, and Nvidia Xavier development boards to demonstrate the training time and energy consumption savings as compared to recent state-of-the-art methods.

A Tiny Model for Fast and Precise Ship Detection via Feature Channel Pruning

It is of great significance to accurately detect ships on the ocean. To obtain higher detection performance, many researchers use deep learning to identify ships from images instead of traditional detection methods. Nevertheless, the marine environment is relatively complex, making it quite difficult to determine features of ship targets. In addition, many detection models contain a large amount of parameters, which is not suitable to deploy in devices with limited computing resources. The two problems restrict the application of ship detection. In this paper, firstly, an SAR ship detection dataset is built based on several databases, solving the problem of a small number of ship samples. Then, we integrate the SPP, ASFF, and DIOU-NMS module into original YOLOv3 to improve the ship detection performance. SPP and ASFF help enrich semantic information of ship targets. DIOU-NMS can lower the false alarm. The improved YOLOv3 has 93.37% mAP, 4.11% higher than YOLOv3 on the self-built dataset. Then, we use the MCP method to compress the improved YOLOv3. Under the pruning ratio of 80%, the obtained compressed model has only 6.7 M parameters. Experiments show that MCP outperforms NS and ThiNet. With the size of 26.8 MB, the compact model can run at 15 FPS on an NVIDIA TX2 embedded development board, 4.3 times faster than the baseline model. Our work will contribute to the development and application of ship detection on the sea.

Using deep learning to identify maturity and 3D distance in pineapple fields

Pineapples are an important agricultural economic crop in Taiwan. Considerable human resources are required to protect pineapples from excessive solar radiation, which could otherwise lead to overheating and subsequent deterioration. Note that simple covering all of the fruit with a paper bag is not a viable solution, due to the fact that it makes it impossible to determine whether the fruit is ripe. This paper proposes a system by which to automate the detection of ripe pineapples. The proposed deep learning architecture enables detection regardless of lighting conditions, achieving accuracy of more than 99.27% with error of less than 2% at distances of 300 ~ 800 mm. This proposed system using an Nvidia TX2 is capable of 15 frames per second, thereby making it possible to mount the device on machines that move at walking speed.

Improved Real-Time Traffic Obstacle Detection and Classification Method Applied in Intelligent and Connected Vehicles in Mixed Traffic Environment

Mixed traffic is a common phenomenon in urban environment. For the mixed traffic situation, the detection of traffic obstacles, including motor vehicle, non-motor vehicle, and pedestrian, is an essential task for intelligent and connected vehicles (ICVs). In this paper, an improved YOLO model is proposed for traffic obstacle detection and classification. The YOLO network is used to accurately detect the traffic obstacles, while the Wasserstein distance-based loss is used to improve the misclassification in the detection that may cause serious consequences. A new established dataset containing four types of traffic obstacles including vehicles, bikes, riders, and pedestrians is collected under different time periods and different weather conditions in urban environment in Wuhan, China. Experiments are performed on the established dataset on Windows PC and NVIDIA TX2, respectively. From the experimental results, the improved YOLO model has higher mean average precision than the original YOLO model and can effectively reduce intolerable misclassifications. In addition, the improved YOLOv4-tiny model has a detection speed of 22.5928 fps on NVIDIA TX2, which can basically realize the real-time detection of traffic obstacles.

Position detection for electric vehicle DWCS using VI-SLAM method

The dynamic wireless charging system (DWCS) is developed to solve the problems of large battery volume and mileage anxiety of electric vehicles. However, the accurate position detection for electric vehicle DWCS is facing challenge. The traditional communication, detection and estimation methods are difficult to accurately obtain the position. To tackle this problem, the visual inertial simultaneous localization and mapping (VI-SLAM) method is applied to the electric vehicles DWCS. Firstly, the graph optimization based tight coupling method is used to integrate the monocular visual and inertial measurement unit (IMU) measurements. Secondly, the NVIDIA TX2 and MYNT VI-sensor suite are assembled, which the MTi300-IMU is treated as the ground truth system. Finally, the mobile vehicle is controlled to race on the simulated DWCS pathway. The experimental result shows that the method achieves great performance with the accuracy of centimeter level. In particular, the root mean square error (RMSE) of pose (i.e. position and orientation) in X, Y, Z directions are 0.086 m, 0.092 m, 0.102 m and 2.423°, 1.682°, 2.501°, respectively. Compared with the smooth variable structure filter (SVSF) based SLAM method, 0.106 m, 0.098 m, 0.130 m and 3.069°, 3.261°, 2.961°, the accuracy of ours is increased by 18.87%, 6.12%, 21.54% and 21.05%, 45.42%, 15.54%, respectively.

Embedded real-time infrared and visible image fusion for UAV surveillance

Infrared and visible image fusion is a beneficial processing task for Unmanned Aerial Vehicle (UAV) surveillance, which can improve visibility by combining the advantages of the infrared camera and the visible light camera. An embedded onboard solution is necessary for UAV-based surveillance missions because it reduces the amount of data that are transmitted to the ground. In this paper, we propose an infrared and visible light image fusion method and implement it on two platforms with commonly used HW accelerators for embedded vision applications: Zedboard (ARM + FPGA) and NVIDIA TX1 (ARM + GPU), and compare their performances. To verify the usefulness of image fusion, we carry out sufficient experiments to prove that image fusion can improve the target detection ability of a UAV in different scenes. The detection rate for target detection is up to 0.926 in our experiments. The execution times on the ZedBoard and the TX1 are, respectively, 205.3 FPS and 36.6 FPS (38 $$\times$$ and 6.7 $$\times$$ in comparison to an ARM Cortex-A9 processor). Our results also show that the ZedBoard achieves an energy/frame reduction ratio of 7.1 $$\times$$ and 18.9 $$\times$$ respectively compared to the TX1 and the ARM CPU. This work is based on a UAV platform designed by ourselves, and all image sets are real scenes that we have captured. This demonstrates that the proposed method is viable and reflects the actual needs of real UAV surveillance systems.

Position Detection for Electric Vehicle Dwcs Using Vi-Slam Method

The dynamic wireless charging system (DWCS) is developed to solve the problems of large battery volume and mileage anxiety of electric vehicles. However, the accurate position detection for electric vehicle DWCS is facing challenge. The traditional communication, detection and estimation methods are difficult to accurately obtain the position. To tackle this problem, the visual inertial simultaneous localization and mapping (VI-SLAM) method is applied to the electric vehicles DWCS. Firstly, the graph optimization based tight coupling method is used to integrate the monocular visual and IMU measurements. Secondly, the NVIDIA TX2 and MYNT VI-sensor suite are assembled, which the MTi300-IMU is treated as the ground truth system. Finally, the mobile vehicle is controlled to race on the simulated DWCS pathway. The experimental result shows that the method achieves great performance with the accuracy of centimeter level. In particular, the root mean square error (RMSE) in X, Y, Z directions are 0.086,0.092,0.102, respectively.

Enabling Latency-Aware Data Initialization for Integrated CPU/GPU Heterogeneous Platform

Nowadays, driven by the needs of autonomous driving and edge intelligence, integrated CPU/GPU heterogeneous platform has gained significant attention from both academia and industry. As the representative series, NVIDIA Jetson family perform well in terms of computation capability, power consumption, and mobile size. Even so, the integrated heterogeneous platform only contains one limited physical memory, which is shared by the CPU and GPU cores and can be the performance bottleneck of the mobile/edge applications. On the other hand, with the unified memory (UM) model introduced in GPU programming, not only the memory allocation is significantly reduced, which mitigates the memory bottleneck of the integrated platforms but also the memory management and programming are simplified. However, as a programming legacy, the UM model still follows the conventional copy-then-execute model, initializing data on the CPU side after allocating memory. This legacy programming mode not only causes significant initialization latency but also slows the execution of the following kernel. In this article, we propose a framework to enable the latency-aware data initialization on the integrated heterogeneous platform. The framework not only includes three data initialization modes, the CPU initialization, GPU initialization, and hybrid initialization, but also utilizes an affinity estimation model to wisely decide the best initialization mode for an application such that the initialization latency performance of the application can be optimized. We evaluate our design on NVIDIA TX2 and AGX platforms. The results demonstrate that the framework can accurately select a data initialization mode for a given application to significantly reduce the initialization latency. We envision this latency-aware data initialization framework being adopted in a full-version of autonomous solution (e.g., Autoware) in the future.

결함 검출을 위한 임베디드 딥러닝 시스템

A machine vision based industrial inspection requires little computation time and localizing defects robustly with high accuracy. Recent mobile and embedded systems require computationally efficient machine intelligence with a deep learning model. In order to improve detection performance and processing time, various network modification methods are proposed. The experiments for defect detection on the metal surfaces data are executed using the various YOLO networks on embedded GPU system Nvidia Tx-1. The results for detection performance and inspection time are compared and analysed. Among them, modified YOLOv2-tiny model shows a better performance in both detection rate and fps.

DAC-SDC Low Power Object Detection Challenge for UAV Applications.

The 55th Design Automation Conference (DAC) held its first System Design Contest (SDC) in 2018. SDC'18 features a lower power object detection challenge (LPODC) on designing and implementing novel algorithms based object detection in images taken from unmanned aerial vehicles (UAV). The dataset includes 95 categories and 150k images, and the hardware platforms include Nvidia's TX2 and Xilinx's PYNQ Z1. DAC-SDC'18 attracted more than 110 entries from 12 countries. This paper presents in detail the dataset and evaluation procedure. It further discusses the methods developed by some of the entries as well as representative results. The paper concludes with directions for future improvements.

Multimodal Fusion Object Detection System for Autonomous Vehicles

In order for autonomous vehicles to safely navigate the road ways, accurate object detection must take place before safe path planning can occur. Currently, general purpose object detection convolutional neural network (CNN) models have the highest detection accuracies of any method. However, there is a gap in the proposed detection frameworks. Specifically, those that provide high detection accuracy necessary for deployment but do not perform inference in realtime, and those that perform inference in realtime but detection accuracy is low. We propose multimodel fusion detection system (MFDS), a sensor fusion system that combines the speed of a fast image detection CNN model along with the accuracy of light detection and range (LiDAR) point cloud data through a decision tree approach. The primary objective is to bridge the tradeoff between performance and accuracy. The motivation for MFDS is to reduce the computational complexity associated with using a CNN model to extract features from an image. To improve efficiency, MFDS extracts complimentary features from the LiDAR point cloud in order to obtain comparable detection accuracy. MFDS is novel by not only using the image detections to aid three-dimensional (3D) LiDAR detection but also using the LiDAR data to jointly bolster the image detections and provide 3D detections. MFDS achieves 3.7% higher accuracy than the base CNN detection model and is able to operate at 10 Hz. Additionally, the memory requirement for MFDS is small enough to fit on the Nvidia Tx1 when deployed on an embedded device.

Broken Corn Detection Based on an Adjusted YOLO With Focal Loss

Corns may be broken during corn mechanical harvesting. The ratio of broken corns measures the quality of mechanical harvesting and should be monitored in real time. This paper presents a method of detecting both broken and non-broken corns at the conveyor belt of a corn harvester based on the YOLO. The network structure of the YOLO is adjusted here to obtain more robust features so that it can work well in the open working space of the corn harvesting. Moreover, we improve the loss function to ensure that the hard examples can catch more attention during training. As it is difficult to obtain many training data of broken corns, the simulated broken corn images are generated from the real images of corns by a simple synthetic method. The concerned corn detection network of the proposed YOLO-based method is first trained with plenty of simulated samples and then fine-tuned with the real corn images. The experiments on real corn data confirm that the proposed YOLO-based method can achieve good accuracy and fast speed on the NVIDIA TX2.

Cache-Aware Real-Time Virtualization for Clustered Multi-Core Platforms

With the increasing complexity of recent autonomous platforms, there is a strong demand to better utilize system resources while satisfying stringent real-time requirements. Embedded virtualization is an appealing technology to meet this demand. It enables the consolidation of real-time systems with different criticality levels on a single hardware platform by enforcing temporal isolation. On multi-core platforms, however, shared hardware resources, such as caches and memory buses, weaken this isolation. In particular, due to the resulting cache interference, a large last-level cache in recent processors can easily jeopardize the timing predictability of real-time tasks due to cache interference. While researchers in the real-time systems community have developed solutions to tackle this problem, existing cache management schemes reveal two major limitations when used in a clustered multi-core embedded system. The first is the cache co-partitioning problem, which can lead to wrong cache allocation and cache underutilization. The second is the cache interference of inter-virtual-machine (VM) communication because prior work has considered only independent tasks. This paper presents a cluster-aware real-time cache allocation scheme to address these problems. The proposed scheme takes into account the cluster information of the system, and finds the cache allocation that satisfies the timing and memory requirements of tasks. The scheme also maximizes slack time to meet task deadline, which brings flexibility and resilience to unexpected events. Tasks using inter-VM communication are also provided with guaranteed blocking time and cache isolation. We have implemented a prototype of our scheme on an Nvidia TX2 clustered multi-core platform and evaluated the effectiveness of our scheme over cluster-unaware approaches.

Real-Time Image Transmission Algorithm in WSN with Limited Bandwidth

In this paper, a real-time image transmission algorithm in WSN with limited bandwidth networks is studied. Firstly, a simple and effective monitoring network architecture is established, which allows multiple video monitoring nodes to access the network, and the data transmission is controlled by the synchronization mechanism without collision. Then, the image data is compressed locally at the monitoring nodes (over 85%), so that the image of each node can meet the needs of real-time data transmission, and the overall power consumption of the system is greatly reduced. Finally, based on NVIDIA TX1, four test nodes are constructed to test the algorithm cumulatively, which verifies the effectiveness of the system framework and compression algorithm.

Embedded Low-Power Processor for Personalized Stress Detection

Personal monitoring systems require sampling and processing on multiple streams of physiological signals to extract meaningful information. These systems require a large number of digital signal processing and machine learning kernels, which typically require significant amounts of power. However, to be used in a wearable environment, the processing system needs to be low-power, real-time, and light-weight. In this brief, we present a personalized stress monitoring processor that can meet these requirements. First, various physiological features are explored to maximize stress detection accuracy using two machine learning classifiers including support vector machine (SVM) and ${K}$ -nearest neighbors (KNN). Among different extracted features from four physiological sensors, heart rate and accelerometer features have 96.7% (SVM) and 95.8% (KNN) detection accuracy. In the second part, two fully flexible and multi-modal processing hardware designs are presented that consist of feature extraction and classification algorithms. We first demonstrate the ASIC post-layout implementation of both designs in 65-nm CMOS technology as well as the implementation on Artix-7 field-programmable gate array (FPGA). The proposed SVM and KNN processors on the ASIC platform occupy an area of 0.17 mm2 and 0.3 mm2 and dissipate 39.4 mW and 76.69 mW power, respectively. The ASIC implementation improves the energy efficiency by $42{\times }$ (SVM) and $12{\times }$ (KNN) over FPGA implementations. The entire stress monitoring system is further evaluated against a number of other platforms including Raspberry Pi 3B, NVIDIA TX1 GPU, and NVIDIA TX2 GPU. The experimental results indicate that ASIC and FPGA platforms have the highest throughput (decision/sec) as well as lowest power consumption over all other platforms. The ASIC/FPGA implementations improve the energy efficiency (throughput/power) by 6/5 and 5/4 order of magnitude compared to TX1 GPU and Raspberry pie ARM platforms, respectively.

Edge computing-based real-time passenger counting using a compact convolutional neural network

Crowd counting from low-resolution images is a challenging task, in particular in the edge computing system. An embedded equipment is commonly incompetent at patch-based crowd counting with real-time performance. This work develops a real-time method to count passengers in a bus by using Nvidia TX2. The videos of entry are recorded by a camera up ahead, and the data suffer from severe occlusion, which makes designing handcrafted features difficult. The counting is performed by summing up pixel values of the density map estimated using a compact convolutional neural network (CCNN), which is robust to scale variations by employing skip connections. A weighted Euclidean loss is proposed to handle cluttered backgrounds and blurry foregrounds. The loss increases the activations in dense regions, but can restrain the activations in background regions. The counting results are further improved by smoothing, which utilizes constraints between consecutive frames. Comparisons with existing counting approaches, including patch-based and whole image-based approaches, are made on two benchmarking datasets. The results indicate the accuracy of CCNN in counting dense crowds. Moreover, the evaluated bus datasets verify the feasibility of CCNN in counting passengers from low-resolution input images with real-time performance on TX2.

Vision-based Hand Recognition Based on ToF Depth Camera

The current gesture recognition methods mostly adopt the classification-based approaches such as Neural Network (NN), Support Vector Machine (SVM), Hidden Markov Model (HMM) etc. As for the input image features, most research studies combined the color and depth images (ex. RGB-D) to obtain more accurate information of hand area, and such techniques may cost high computational resources and energy consumptions. To provide a low-cost gesture recognition method for wearable devices, this thesis used merely the Time-of-Flight depth camera to achieve a lightweight gesture recognition method. In most traditional gesture recognition methods, users have to wear gloves or bracelets to let depth cameras being able to accurately capture hands areas, and so that the hand contours, palm’s distances, and angle feature can be obtained. Moreover, the Earth Mover’s Distance (EMD) algorithm, which is adopted in most gesture recognition approaches, costs high computational times. In this study, to avoid wearing gloves or bracelets, we propose a new algorithm that can compute the wrist cutting edges and capture the palm areas. In addition, this thesis proposes an efficient finger detection algorithm to judge the number of fingers, and significantly reduce the computing times. In the experimental results, our proposed method achieves a recognition rate of 90% and the performance has 5 frames per second on NVIDIA TX1 embedded platforms.