3D Object Detection Algorithm Research Articles

VFL3D: A Single-Stage Fine-Grained Lightweight Point Cloud 3D Object Detection Algorithm Based on Voxels

VFL3D: A Single-Stage Fine-Grained Lightweight Point Cloud 3D Object Detection Algorithm Based on Voxels

Enhanced 3D object detection for autonomous driving: A spatial-temporal alignment approach in Bird's Eye View scenarios

This paper presents a novel 3D object detection algorithm designed for Bird's Eye View (BEV) scenarios, which significantly improves detection capabilities by integrating spatial and temporal features. The core of our approach is the spatial-temporal alignment module that efficiently processes information across different time steps and spatial locations, enhancing the precision and robustness of object detection. We employ a temporal self-attention mechanism to capture the motion information of objects over time, allowing the model to correlate features across various time steps for identifying and tracking moving objects. Additionally, a spatial cross-attention mechanism is utilized to focus on spatial features within regions of interest, promoting interactions between features extracted from camera views and BEV queries. Our method also implements temporal feature integration and multi-scale feature fusion to enhance detection stability and accuracy for fast-moving objects and to capture multi-scale context information, respectively. The model employs an enriched feature set post alignment for 3D bounding box prediction, ascertaining the position, dimensions, and orientation of objects. We conducted experiments on two public datasets for autonomous driving nuScenes and Waymo Open Dataset, demonstrating that our method outperforms previous BEVFormer and other state-of-the-art methods in terms of detection accuracy and robustness. The paper concludes with potential future directions for optimizing the BEVFormer model's performance and exploring its application in broader scenarios and tasks.

DopplerPTNet: Object Detection Network with Doppler Velocity Information for FMCW LiDAR Point Cloud

Abstract In the field of autonomous driving, LiDAR plays a crucial role in perception and detection. LiDAR based on Time-of-Flight (ToF) mode can only provide three-dimensional spatial coordinate information of point clouds. In point cloud object detection, the limited feature information of spatial coordinates to some extent restricts the further optimization and improvement of algorithm detection performance. However, LiDAR based on Frequency-Modulated Continuous-Wave (FMCW) mode can not only obtain the three-dimensional spatial coordinates of point clouds, but also directly measure the Doppler velocity information of points, effectively compensating for the limitation of relying solely on spatial coordinate information for object recognition. Therefore, based on the CARLA simulator, we construct the first FMCW LiDAR point cloud object detection simulation dataset, FMCWLidDet. What’s more, a novel 4D object detection algorithm, DopplerPTNet, is proposed based on the direct acquisition of point Doppler velocity information by FMCW LiDAR. The algorithm solves the problem of single spatial coordinate information feature in existing 3D object detection algorithms, which makes it difficult to further improve detection accuracy. The dataset is available at https://github.com/xilight123/FMCW-LiDAR-object-detection-dataset.

LiDAR-Based 3D Temporal Object Detection via Motion-Aware LiDAR Feature Fusion.

Recently, the growing demand for autonomous driving in the industry has led to a lot of interest in 3D object detection, resulting in many excellent 3D object detection algorithms. However, most 3D object detectors focus only on a single set of LiDAR points, ignoring their potential ability to improve performance by leveraging the information provided by the consecutive set of LIDAR points. In this paper, we propose a novel 3D object detection method called temporal motion-aware 3D object detection (TM3DOD), which utilizes temporal LiDAR data. In the proposed TM3DOD method, we aggregate LiDAR voxels over time and the current BEV features by generating motion features using consecutive BEV feature maps. First, we present the temporal voxel encoder (TVE), which generates voxel representations by capturing the temporal relationships among the point sets within a voxel. Next, we design a motion-aware feature aggregation network (MFANet), which aims to enhance the current BEV feature representation by quantifying the temporal variation between two consecutive BEV feature maps. By analyzing the differences and changes in the BEV feature maps over time, MFANet captures motion information and integrates it into the current feature representation, enabling more robust and accurate detection of 3D objects. Experimental evaluations on the nuScenes benchmark dataset demonstrate that the proposed TM3DOD method achieved significant improvements in 3D detection performance compared with the baseline methods. Additionally, our method achieved comparable performance to state-of-the-art approaches.

Sparse Embedded Convolution Based Dual Feature Aggregation 3D Object Detection Network

The algorithm design of compatible detection speed and accuracy based on LiDAR point clouds is a challenging issue in various practical applications of 3D object detection, including the field of autonomous driving. This paper designs a single-stage object detection algorithm that is lightweight and compatible with detection speed and accuracy for the above issue. To achieve these objectives, we propose a framework for a 3D object detection algorithm using a single-stage detection network as the backbone network. Firstly, we design a dual feature extraction module to reduce the occurrence of vehicle miss and error detection problems. Then, we use a multi-scale feature fusion scheme to fuse feature information with different scales. Furthermore, we design a data enhancement scheme suitable for this network architecture. Experimental results in the KITTI dataset show that the proposed method achieves improvement ratios of 38.5% for the detection speed and 2.88% ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document} 13.65% in terms of the average precision of vehicle detection compared to the existing algorithm based on single-stage object detection (SECOND).

BEV Space 3D Object Detection Algorithm Based on Fusion of Infrared Camera and LiDAR

BEV Space 3D Object Detection Algorithm Based on Fusion of Infrared Camera and LiDAR

Multiattention Mechanism 3D Object Detection Algorithm Based on RGB and LiDAR Fusion for Intelligent Driving.

This paper proposes a multimodal fusion 3D target detection algorithm based on the attention mechanism to improve the performance of 3D target detection. The algorithm utilizes point cloud data and information from the camera. For image feature extraction, the ResNet50 + FPN architecture extracts features at four levels. Point cloud feature extraction employs the voxel method and FCN to extract point and voxel features. The fusion of image and point cloud features is achieved through regional point fusion and voxel fusion methods. After information fusion, the Coordinate and SimAM attention mechanisms extract fusion features at a deep level. The algorithm's performance is evaluated using the DAIR-V2X dataset. The results show that compared to the Part-A2 algorithm; the proposed algorithm improves the mAP value by 7.9% in the BEV view and 7.8% in the 3D view at IOU = 0.5 (cars) and IOU = 0.25 (pedestrians and cyclists). At IOU = 0.7 (cars) and IOU = 0.5 (pedestrians and cyclists), the mAP value of the SECOND algorithm is improved by 5.4% in the BEV view and 4.3% in the 3D view, compared to other comparison algorithms.

YOLOv7-3D: A Monocular 3D Traffic Object Detection Method from a Roadside Perspective

Current autonomous driving systems predominantly focus on 3D object perception from the vehicle’s perspective. However, the single-camera 3D object detection algorithm in the roadside monitoring scenario provides stereo perception of traffic objects, offering more accurate collection and analysis of traffic information to ensure reliable support for urban traffic safety. In this paper, we propose the YOLOv7-3D algorithm specifically designed for single-camera 3D object detection from a roadside viewpoint. Our approach utilizes various information, including 2D bounding boxes, projected corner keypoints, and offset vectors relative to the center of the 2D bounding boxes, to enhance the accuracy of 3D object bounding box detection. Additionally, we introduce a 5-layer feature pyramid network (FPN) structure and a multi-scale spatial attention mechanism to improve feature saliency for objects of different scales, thereby enhancing the detection accuracy of the network. Experimental results demonstrate that our YOLOv7-3D network achieved significantly higher detection accuracy on the Rope3D dataset while reducing computational complexity by 60%.

2D and 3D object detection algorithms from images: A Survey

2D and 3D object detection algorithms from images: A Survey

Pseudo-Mono for Monocular 3D Object Detection in Autonomous Driving

Current monocular 3D object detection algorithms generally suffer from inaccurate depth estimation, which leads to reduction of detection accuracy. The depth error from image-to-image generation for the stereo view is insignificant compared with the gap in single-image generation. Therefore, a novel pseudo-monocular 3D object detection framework is proposed, which is called Pseudo-Mono. Particularly, stereo images are brought into monocular 3D detection. Firstly, stereo images are taken as input, then a lightweight depth predictor is used to generate the depth map of input images. Secondly, the left input images obtained from stereo camera are used as subjects, which generate enhanced visual feature and multi-scale depth feature by depth indexing and feature matching probabilities, respectively. Finally, sparse anchors set by the foreground probability maps and the multi-scale feature maps are used as reference points to find the suitable initialization approach of object query. The encoded visual feature is adopted to enhance object query for enabling deep interaction between visual feature and depth feature. Compared with popular monocular 3D object detection methods, Pseudo-Mono is able to achieve richer fine-grained information without additional data input. Extensive experimental results on the datasets of KITTI, NuScenes, and MS-COCO demonstrate the generalizability and portability of the proposed method. The effectiveness and efficiency of Pseudo-Mono have been demonstrated by extensive ablation experiments. Experiments on a real vehicle platform have shown that the proposed method maintains high performance in complex real-world environments.

An efficient 3D object detection method based on Fast Guided Anchor Stereo RCNN

In most binocular 3D detection algorithms, a large number of anchor points need to be selected, which leads to the problem of slow feature extraction. To solve this problem, an anchor-guided 3D object detection algorithm for autonomous driving is proposed based on Stereo Recurrent Convolutional Neutral Network (Stereo RCNN), which is called Fast Guided Anchored Stereo RCNN (FGAS RCNN). The proposed FGAS framework is divided into two stages. In the first stage, a probability map is generated for the left and right input images to determine the foreground position. Sparse anchor points and corresponding sparse anchor boxes are generated from the prior information. Left and right anchors are used as a whole to generate a 2D preselection box. In the second stage, a Feature Pyramid Network (FPN) based on key point generation network is used to generate key points, which are combined with stereo regression to generate 3D preselected boxes. Finally, instance-level disparity estimation is proposed to solve the problem of pixel-level information loss in the original image. Instance-level disparity is combined with instance segmentation masks to improve the accuracy of center depth on the 3D bounding box. Extensive experiments on the challenging Kitti dataset and NuScences dataset show that the proposed method reduces the computational cost while maintaining a high regression rate without any depth information and prior information of position. Compared to other methods, the proposed method has higher efficiency, better robustness and stronger generalization ability.

3D object detection algorithm based on multi-sensor segmental fusion of frustum association for autonomous driving

3D object detection algorithm based on multi-sensor segmental fusion of frustum association for autonomous driving

3D Point Cloud Object Detection Algorithm Based on Temporal Information Fusion and Uncertainty Estimation

In autonomous driving, LiDAR (light detection and ranging) data are acquired over time. Most existing 3D object detection algorithms propose the object bounding box by processing each frame of data independently, which ignores the temporal sequence information. However, the temporal sequence information is usually helpful to detect the object with missing shape information due to long distance or occlusion. To address this problem, we propose a temporal sequence information fusion 3D point cloud object detection algorithm based on the Ada-GRU (adaptive gated recurrent unit). In this method, the feature of each frame for the LiDAR point cloud is extracted through the backbone network and is fed to the Ada-GRU together with the hidden features of the previous frames. Compared to the traditional GRU, the Ada-GRU can adjust the gating mechanism adaptively during the training process by introducing the adaptive activation function. The Ada-GRU outputs the temporal sequence fusion features to predict the 3D object in the current frame and transmits the hidden features of the current frame to the next frame. At the same time, the label uncertainty of the distant and occluded objects affects the training effect of the model. For this problem, this paper proposes a probability distribution model of 3D bounding box coordinates based on the Gaussian distribution function and designs the corresponding bounding box loss function to enable the model to learn and estimate the uncertainty of the positioning of the bounding box coordinates, so as to remove the bounding box with large positioning uncertainty in the post-processing stage to reduce the false positive rate. Finally, the experiments show that the methods proposed in this paper improve the accuracy of the object detection without significantly increasing the complexity of the algorithm.

Multi-Modal 3D Object Detection in Autonomous Driving: A Survey

The past decade has witnessed the rapid development of autonomous driving systems. However, it remains a daunting task to achieve full autonomy, especially when it comes to understanding the ever-changing, complex driving scenes. To alleviate the difficulty of perception, self-driving vehicles are usually equipped with a suite of sensors (e.g., cameras, LiDARs), hoping to capture the scenes with overlapping perspectives to minimize blind spots. Fusing these data streams and exploiting their complementary properties is thus rapidly becoming the current trend. Nonetheless, combining data that are captured by different sensors with drastically different ranging/ima-ging mechanisms is not a trivial task; instead, many factors need to be considered and optimized. If not careful, data from one sensor may act as noises to data from another sensor, with even poorer results by fusing them. Thus far, there has been no in-depth guidelines to designing the multi-modal fusion based 3D perception algorithms. To fill in the void and motivate further investigation, this survey conducts a thorough study of tens of recent deep learning based multi-modal 3D detection networks (with a special emphasis on LiDAR-camera fusion), focusing on their fusion stage (i.e., when to fuse), fusion inputs (i.e., what to fuse), and fusion granularity (i.e., how to fuse). These important design choices play a critical role in determining the performance of the fusion algorithm. In this survey, we first introduce the background of popular sensors used for self-driving, their data properties, and the corresponding object detection algorithms. Next, we discuss existing datasets that can be used for evaluating multi-modal 3D object detection algorithms. Then we present a review of multi-modal fusion based 3D detection networks, taking a close look at their fusion stage, fusion input and fusion granularity, and how these design choices evolve with time and technology. After the review, we discuss open challenges as well as possible solutions. We hope that this survey can help researchers to get familiar with the field and embark on investigations in the area of multi-modal 3D object detection.

Monocular 3D Object Detection Based on Pseudo Multimodal Information Extraction and Keypoint Estimation

Three-dimensional object detection is an essential and fundamental task in the field of computer vision which can be widely used in various scenarios such as autonomous driving and visual navigation. In view of the current insufficient utilization of image information in current monocular camera-based 3D object detection algorithms, we propose a monocular 3D object detection algorithm based on pseudo-multimodal information extraction and keypoint estimation. We utilize the original image to generate pseudo-lidar and a bird’s-eye view, and then feed the fused data of the original image and pseudo-lidar to the keypoint-based network for an initial 3D box estimation, finally using the bird’s-eye view to refine the initial 3D box. The experimental performance of our method exceeds state-of-the-art algorithms under the evaluation criteria of 3D object detection and localization on the KITTI dataset, achieving the best experimental performance so far.

3D object detection for autonomous driving from image: a survey——benchmarks, constraints and error analysis

从高分辨率图像中获取周边目标的精准3D位置和尺寸信息是实现自动驾驶控制和行为决策的基础，因此基于图像的3D目标检测是自动驾驶领域中的研究热点。已有学者对该领域方法论及成果进行了比较详细的综述，但对于导致现有方法检测精度不尽如意的制约因素未能进行深入系统的分析。考虑自动驾驶领域在工程应用方面的要求高，且现有方法以数据驱动类型为主，本文从常用数据集和评价基准、数据影响、方法论的制约因素和误差等角度，对学术界和产业界在3D目标检测方面的研究成果及行业应用进行较为系统的阐述。首先，从学术界探索成果以及自动驾驶行业的应用角度进行概要介绍。然后，从数据采集设备、数据精度和标注信息3方面详细分析总结了KITTI等4个通用数据集，并对这些数据集提出的主要评价指标进行对比分析。接着，从数据和方法论方面分析制约算法性能的主要因素及由此造成的误差影响。在数据方面，制约因素主要是数据精度、样本差异、标注数据量和标注规范；在方法论方面，制约因素主要包括先验几何关系、深度预测误差和数据模态等。最后，对国内外研究现状进行总结，并在数据集、评价指标和目标深度预测等方面提出了未来需要重点关注的研究方向。;Autonomous driving-oriented accurate perception and measurement of the three-dimensional(3D) spatial position and scale can be as the basis for realizing the control ability and decision-making level. Sensing technology-driven autonomous vehicles are equipped with high-resolution camera, light detection and ranging (LiDAR), radar, global positioning system (GPS)/inertial measurement unit (IMU) and other related sensors. Current LiDAR or multi-modal data-based 3D object detection algorithms are challenged for its deployment and application because of the shortcomings of LiDAR sensors like high price, limited sensing range, and sparse point clouds data. In contrast, such high-resolution cameras are commonly-used and featured by its lower price, and it can obtain high-resolution spatial information, richer shape, and appearance details as well. The emerging image-based 3D object detection is focused on further. At present, constraints of detection accuracy of the existing methods are still to be analyzed thoroughly and systematically. We summary the research results and industrial applications in relevance to such 1) perspectives of commonly used datasets and evaluation criteria, 2) data impact, 3) methodological constraints and prediction errors. First, a brief introduction is linked to perspective of academic domain and application of autonomous driving industry. We briefly review latest growths of Baidu Apollo, Google Waymo, Tesla and other related autonomous driving companies, and the thread of 3D object detection methods for autonomous driving. Then, we analyzed and summarized four popular datasets like KITTI, nuScenes, Waymo open dataset, and DAIR-V2X dataset from three aspects of:1) data acquisition/sensors, data accuracy and data label information;2) key evaluation standards proposed by these data sets, and 3) pros/cons and applicability of these evaluation standards. Third, main constraints of the image-based 3D object detection algorithm and the errors are derived from two sides of:data and methodology. Such main data constraints are originated from their data accuracy, sample difference, data volume, and data annotation. The data accuracy is mainly limited by equipment performance. The sample difference is mainly restricted by such image processing problems in related to object distance difference, angle difference, occlusion, and truncation. Data volume is affected by variety of 3D data types and high difficulty of labeling. The volume of 3D object detection data set is much smaller in comparison with the 2D object detection data set. Data annotation is mainly focused on 3D bounding box labeling, the labeling details, and quality of the dataset, especially for image annotation used in image-based 3D object detection. For non-rigid objects like pedestrians, the annotation error is larger, and there are some optimal for improving the labeling method. The general framework of image-based 3D object detection can be classified as one-stage methods and two-stage methods, and the limitations consists of 1) the prior geometric relationship, 2) depth prediction accuracy, and 3) data modality. The prior geometric relationship is focused on 2D-3D geometric constraints for 2D images-projected 3D objects and objects-between position relationships. The image-based 3D object detection methods face such problems as:prior 2D-3D geometric constraints and occluded and truncated objects. The prediction of depth information from 2D images is an ill conditioned problem, and dimension collapse will cause depth prediction error-relevant loss of depth information in the image. On the one hand, the depth prediction is often not accurate due to the influence of projection relationship. On the other hand, the performance of continuous depth prediction is often poor at the depth mutation of the image(such as edge of objects). When the prediction depth is discretized, there is a problem that the classification of depth is relatively rough, and the accuracy classification cannot be arbitrarily divided. The limitation of single image-based data modality is mainly reflected via large error of depth prediction. The detection performance of the algorithm can be optimized by 1) simulating the stereo signal and LiDAR point clouds, or 2) using stereo image as the aided input, or 3) leveraging point clouds data with accurate 3D information as supervision signal. In addition, video data can be adopted to improve the detection accuracy to a certain extent. Forth, current research situation is summarized and compared from academic and industrial domain. Finally, some future research directions are predicted in terms of such factors of datasets, evaluation indicators, and depth prediction.

X-view: Non-egocentric Multi-View 3D Object Detector.

3D object detection algorithms for autonomous driving reason about 3D obstacles either from 3D birds-eye view or perspective view or both. Recent works attempt to improve the detection performance via mining and fusing from multiple egocentric views. Although the egocentric perspective view alleviates some weaknesses of the birds-eye view, the sectored grid partition becomes so coarse in the distance that the targets and surrounding context mix together, which makes the features less discriminative. In this paper, we generalize the research on 3D multi-view learning and propose a novel multi-view-based 3D detection method, named X-view, to overcome the drawbacks of the multi-view methods. Specifically, X-view breaks through the traditional limitation about the perspective view whose original point must be consistent with the 3D Cartesian coordinate. X-view is designed as a general paradigm that can be applied on almost any 3D detectors based on LiDAR with only little increment of running time, no matter it is voxel/grid-based or raw-point-based. We conduct experiments on KITTI [1] and NuScenes [2] datasets to demonstrate the robustness and effectiveness of our proposed X-view. The results show that X-view obtains consistent improvements when combined with mainstream state-of-the-art 3D methods.

Anti-Noise 3D Object Detection of Multimodal Feature Attention Fusion Based on PV-RCNN.

3D object detection methods based on camera and LiDAR fusion are susceptible to environmental noise. Due to the mismatch of physical characteristics of the two sensors, the feature vectors encoded by the feature layer are in different feature spaces. This leads to the problem of feature information deviation, which has an impact on detection performance. To address this problem, a point-guided feature abstract method is presented to fuse the camera and LiDAR at first. The extracted image features and point cloud features are aggregated to keypoints for enhancing information redundancy. Second, the proposed multimodal feature attention (MFA) mechanism is used to achieve adaptive fusion of point cloud features and image features with information from multiple feature spaces. Finally, a projection-based farthest point sampling (P-FPS) is proposed to downsample the raw point cloud, which can project more keypoints onto the close object and improve the sampling rate of the point-guided image features. The 3D bounding boxes of the object is obtained by the region of interest (ROI) pooling layer and the fully connected layer. The proposed 3D object detection algorithm is evaluated on three different datasets, and the proposed algorithm achieved better detection performance and robustness when the image and point cloud data contain rain noise. The test results on a physical test platform further validate the effectiveness of the algorithm.

A Transformer Based Complex-YOLOv4-Trans for 3D Point Cloud Object Detection on Embedded Device

3D object detection is widely used in the field of autonomous driving. The embedded devices have few resources and low power consumption, so the existing 3D object detection algorithms are difficult to complete the detection task in the required time under some conditions. In this paper, the Complex-YOLOv4-Trans model is proposed to use point cloud data for 3D object detection. Firstly, a transformer encoder block is introduced to improve the structure of Backbone, which makes full use of its context information to improve detection accuracy. Secondly, the general convolution in Backbone is replaced with a depth-wise separable convolution, which effectively reduces the computational load of the model. Finally, Complete-IoU (CIoU) is used as the loss function, and then 8-bit model quantization is performed to speed up the inference. We select the KITTI dataset and deploy the model on NVIDIA Jetson Xavier NX by using TensorRT acceleration tools to test the methods. The mAP of Bird’s Eye View for Car is 71.79%, the mAP for 3D Detection is 15.82%, and the FPS on the NX device is 42 frames. Experimental results show that our Complex-YOLOv4-Trans can perform real-time 3D object detection tasks on low-energy embedded devices.

User Experience Adaptation of Complex Game Interface for User Behaviour Modeling Using RNN

In a video game review, the main focus is the narratives, characters, graphics, and mechanics in the gameplay. Some recent research mentions the user interface only when it comes into light as a creative platform for simple interactive narratives from a technical point of view; this narrative is mainly a software tool that requires traditionally modernized inputs from the user. The user needs to interact with the navigational controls or menus in order to start a basic game play. A complex game interface as stimulus is generally considered as having a feeling of immersion that allows for visual tracking of user behavioural patterns and use it to predict the next strategy of the user using robust computational models. A number of users have limited sensory perception in a gameplay and hence rely on complex game stimulus and an adaptive model is paramount when considering behavioural expectations that place the user in a digital environment with more expressive perceptions. We developed a custom based eye tracking and 3D object detection algorithm which was utilised by recruiting users to interact with visual 3D objects and trace their eye movement behaviour to generated data. We then applied the use of recurrent neural network (RNN) for direct tracing of user behavioural activities in a sequential manner to predict their behaviour for interface adaptation. Result indicates that redundant user attributes are flexible and flawless for identifying predicted response of the user in a controlled environment. This would lead to prototypical representation of user behavioural analytics as an embedded platform in the confined digital environment. One of the limitations of the project is its inability to basically specify the 3D gaze point at the inner boundaries of the visual field. Data visualisation is strictly based on combined object flow detection. The originality of the work is its ability to redefine fixation point to a rendered cascaded 3D gaze point and space-defined saccade which is indicated by the distance between one gaze points to the other. The 3D gaze point would be well suited for fixation generalisation on 3D as well as on 2D digital oriented environment.