Real-time 3D Object Detection Research Articles

Real-Time 3D Object Detection, Recognition and Presentation Using a Mobile Device for Assistive Navigation

Real-Time 3D Object Detection, Recognition and Presentation Using a Mobile Device for Assistive Navigation

Efficient three-dimensional point cloud object detection based on improved Complex-YOLO.

Lidar-based 3D object detection and classification is a critical task for autonomous driving. However, inferencing from exceedingly sparse 3D data in real-time is a formidable challenge. Complex-YOLO solves the problem of point cloud disorder and sparsity by projecting it onto the bird's-eye view and realizes real-time 3D object detection based on LiDAR. However, Complex-YOLO has no object height detection, a shallow network depth, and poor small-size object detection accuracy. To address these issues, this paper has made the following improvements: (1) adds a multi-scale feature fusion network to improve the algorithm's capability to detect small-size objects; (2) uses a more advanced RepVGG as the backbone network to improve network depth and overall detection performance; and (3) adds an effective height detector to the network to improve the height detection. Through experiments, we found that our algorithm's accuracy achieved good performance on the KITTI dataset, while the detection speed and memory usage were very superior, 48FPS on RTX3070Ti and 20FPS on GTX1060, with a memory usage of 841Mib.

Real-Time 3D Object Detection and Classification in Autonomous Driving Environment Using 3D LiDAR and Camera Sensors

The rapid development of Autonomous Vehicles (AVs) increases the requirement for the accurate prediction of objects in the vicinity to guarantee safer journeys. For effectively predicting objects, sensors such as Three-Dimensional Light Detection and Ranging (3D LiDAR) and cameras can be used. The 3D LiDAR sensor captures the 3D shape of the object and produces point cloud data that describes the geometrical structure of the object. The LiDAR-only detectors may be subject to false detection or even non-detection over objects located at high distances. The camera sensor captures RGB images with sufficient attributes that describe the distinct identification of the object. The high-resolution images produced by the camera sensor benefit the precise classification of the objects. However, hindrances such as the absence of depth information from the images, unstructured point clouds, and cross modalities affect assertion and boil down the environmental perception. To this end, this paper proposes an object detection mechanism that fuses the data received from the camera sensor and the 3D LiDAR sensor (OD-C3DL). The 3D LiDAR sensor obtains point clouds of the object such as distance, position, and geometric shape. The OD-C3DL employs Convolutional Neural Networks (CNN) for further processing point clouds obtained from the 3D LiDAR sensor and the camera sensor to recognize the objects effectively. The point cloud of the LiDAR is enhanced and fused with the image space on the Regions of Interest (ROI) for easy recognition of the objects. The evaluation results show that the OD-C3DL can provide an average of 89 real-time objects for a frame and reduces the extraction time by a recall rate of 94%. The average processing time is 65ms, which makes the OD-C3DL model incredibly suitable for the AVs perception. Furthermore, OD-C3DL provides mean accuracy for identifying automobiles and pedestrians at a moderate degree of difficulty is higher than that of the previous models at 79.13% and 88.76%.

A Transformer Based Complex-YOLOv4-Trans for 3D Point Cloud Object Detection on Embedded Device

3D object detection is widely used in the field of autonomous driving. The embedded devices have few resources and low power consumption, so the existing 3D object detection algorithms are difficult to complete the detection task in the required time under some conditions. In this paper, the Complex-YOLOv4-Trans model is proposed to use point cloud data for 3D object detection. Firstly, a transformer encoder block is introduced to improve the structure of Backbone, which makes full use of its context information to improve detection accuracy. Secondly, the general convolution in Backbone is replaced with a depth-wise separable convolution, which effectively reduces the computational load of the model. Finally, Complete-IoU (CIoU) is used as the loss function, and then 8-bit model quantization is performed to speed up the inference. We select the KITTI dataset and deploy the model on NVIDIA Jetson Xavier NX by using TensorRT acceleration tools to test the methods. The mAP of Bird’s Eye View for Car is 71.79%, the mAP for 3D Detection is 15.82%, and the FPS on the NX device is 42 frames. Experimental results show that our Complex-YOLOv4-Trans can perform real-time 3D object detection tasks on low-energy embedded devices.

Lightweight convolutional neural network for real-time 3D object detection in road and railway environments

For smart mobility, and autonomous vehicles (AV), it is necessary to have a very precise perception of the environment to guarantee reliable decision-making, and to be able to extend the results obtained for the road sector to other areas such as rail. To this end, we introduce a new single-stage monocular real-time 3D object detection convolutional neural network (CNN) based on YOLOv5, dedicated to smart mobility applications for both road and rail environments. To perform the 3D parameter regression, we replace YOLOv5’s anchor boxes with our hybrid anchor boxes. Our method is available in different model sizes such as YOLOv5: small, medium, and large. The new model that we propose is optimized for real-time embedded constraints (lightweight, speed, and accuracy) that takes advantage of the improvement brought by split attention (SA) convolutions called small split attention model (Small-SA). To validate our CNN model, we also introduce a new virtual dataset for both road and rail environments by leveraging the video game Grand Theft Auto V (GTAV). We provide extensive results of our different models on both KITTI and our own GTAV datasets. Through our results, we show that our method is the fastest available 3D object detection with accuracy results close to state-of-the-art methods on the KITTI road dataset. We further demonstrate that the pre-training process on our GTAV virtual dataset improves the accuracy on real datasets such as KITTI, thus allowing our method to obtain an even greater accuracy than state-of-the-art approaches with 16.16% 3D average precision on hard car detection with inference time of 11.1 ms/image on an RTX 3080 GPU.

Real-time 3D Object Detection Using Improved Convolutional Neural Network Based on Image-driven Point Cloud

Background: While detecting the object directly from the 3D point cloud, the natural 3D patterns and invariance of 3D data are often obscure. Objective: In this work, we aimed at studying the 3D object detection from discrete, disordered and sparse 3D point clouds. Methods: The CNN comprises the frustum sequence module, 3D instance segmentation module SNET, 3D point cloud transformation module T-NET, and 3D boundary box estimation module ENET. The search space of the object is determined by the frustum sequence module. The instance segmentation of the point cloud is performed by the 3D instance segmentation module. The 3D coordinates of the object are confirmed by the transformation module and the 3D bounding box estimation module. Results: Evaluated on KITTI benchmark dataset, our method outperforms state of the art by remarkable margins while having real-time capability. Conclusion: We achieve real-time 3D object detection by proposing an improved Convolutional Neural Network (CNN) based on image-driven point clouds.

Real-Time Segmentation of Sparse 3D Point Clouds on a TX2

In this paper, we propose a real-time 3D object detection method, which from sparse 3D point clouds and based on NVIDIA Jetson TX2. This effective method first uses the external parameter matrix to transform the coordinates of the point clouds data, then supplements the missing data in the original data through the interpolation algorithm, and then uses the RANSAC algorithm to remove the ground, uses the improved DBSCAN clustering algorithm, which clustering segmentation is performed for the point clouds fusion partition distance threshold and angle threshold of different distance intervals. Finally, we regress the obstacle boundary to the mini-box by gradient descent, output for the result is visualized in the form of cuboid detection. Compared to traditional DBSCAN algorithm, our method reduces the missed detection rate and improves the real-time performance. Above all, we implements our approach in C++ and ROS, and the experimental results show that our method produces high quality object detection by using NVIDIA Jetson TX2 online.

Real-time 3D object proposal generation and classification using limited processing resources

The task of detecting 3D objects is important in various robotic applications. The existing deep learning-based detection techniques have achieved impressive performances. However, these techniques are limited to being run on a graphics processing unit (GPU) in a real-time environment. To achieve real-time 3D object detection with limited computational resources, we propose an efficient detection method based on 3D proposal generation and classification. The proposal generation is based mainly on point segmentation, while proposal classification is performed by a lightweight convolution neural network (CNN). KITTI datasets are then used to validate our method. It takes only 0.082 s for our method to process one point block with one core of the central processing unit (CPU). In addition to efficiency, the experimental results also demonstrate the capability of the proposed method of producing a competitive performance in object recall and classification.

SARPNET: Shape attention regional proposal network for liDAR-based 3D object detection

Real-time 3D object detection is a fundamental technique in numerous applications, such as autonomous driving, unmanned aerial vehicles (UAV) and robot vision. However, current LiDAR-based 3D object detection algorithms allocate inadequate attention to the inhomogeneity of LiDAR point clouds and the shape encoding capability of regional proposal schemes. This paper introduces a novel 3D object detection network called the Shape Attention Regional Proposal Network (SARPNET), which deploys a new low-level feature encoder to remedy the sparsity and inhomogeneity of LiDAR point clouds with an even sample method, and embodies a shape attention mechanism that learns the statistic 3D shape priors of objects and uses them to spatially enhance semantic embeddings. Experimental results show that the proposed one-stage method outperforms state-of-the-art one-stage and even two-stage methods on the KITTI 3D object detection benchmark. It achieved a BEV AP of (87.26%, 62.80%), 3D AP of (75.64%, 60.43%), and orientation AP of (88.86%, 71.01%) for the detection of cars and cyclists, respectively. Besides, the method is the third winner in the nuScenes 3D Detection challenge of CVPR2019 Workshop on Autonomous Driving (WAD).

SCNet: Subdivision Coding Network for Object Detection Based on 3D Point Cloud

High-precision real-time 3D object detection based on the LiDAR point cloud is an important task for autonomous driving. Most existing methods utilize grid-based convolutional networks to handle sparse and cluttered point clouds. However, the performance of object detection is limited by the coarse grid quantization and expensive computational cost. In this paper, we propose a more efficient representation of 3D point clouds and propose SCNet, a single-stage, end-to-end 3D subdivision coding network that learns finer feature representations for vertical grids. SCNet divides each grid into smaller sub-grids to preserve more point cloud information and converts points in the grid to a uniform feature representation through 2D convolutional neural networks. The 3D point cloud is encoded as the fine 2D sub-grid representation, which helps to reduce the computational cost. We validate our SCNet on the KITTI object benchmark in which we show that the proposed object detector produces state-of-the-art results with more than 20 FPS.

Learning Real-Time Perspective Patch Rectification

We propose two learning-based methods to patch rectification that are faster and more reliable than state-of-the-art affine region detection methods. Given a reference view of a patch, they can quickly recognize it in new views and accurately estimate the homography between the reference view and the new view. Our methods are more memory-consuming than affine region detectors, and are in practice currently limited to a few tens of patches. However, if the reference image is a fronto-parallel view and the internal parameters known, one single patch is often enough to precisely estimate an object pose. As a result, we can deal in real-time with objects that are significantly less textured than the ones required by state-of-the-art methods.The first method favors fast run-time performance while the second one is designed for fast real-time learning and robustness. However, they follow the same general approach: First, a classifier provides for every keypoint a first estimate of its transformation. Then, the estimate allows carrying out an accurate perspective rectification using linear predictors. The last step is a fast verification—made possible by the accurate perspective rectification—of the patch identity and its sub-pixel precision position estimation. We demonstrate the advantages of our approach on real-time 3D object detection and tracking applications.