3D Object Detection Algorithm Research Articles

An Adaptive Multimodal Fusion 3D Object Detection Algorithm for Unmanned Systems in Adverse Weather

Unmanned systems encounter challenging weather conditions during obstacle removal tasks. Researching stable, real-time, and accurate environmental perception methods under such conditions is crucial. Cameras and LiDAR sensors provide different and complementary data. However, the integration of disparate data presents challenges such as feature mismatches and the fusion of sparse and dense information, which can degrade algorithmic performance. Adverse weather conditions, like rain and snow, introduce noise that further reduces perception accuracy. To address these issues, we propose a novel weather-adaptive bird’s-eye view multi-level co-attention fusion 3D object detection algorithm (BEV-MCAF). This algorithm employs an improved feature extraction network to obtain more effective features. A multimodal feature fusion module has been constructed with BEV image feature generation and a co-attention mechanism for better fusion effects. A multi-scale multimodal joint domain adversarial network (M2-DANet) is proposed to enhance adaptability to adverse weather conditions. The efficacy of BEV-MCAF has been validated on both the nuScenes and Ithaca365 datasets, confirming its robustness and good generalization capability in a variety of bad weather conditions. The findings indicate that our proposed algorithm performs better than the benchmark, showing improved adaptability to harsh weather conditions and enhancing the robustness of UVs, ensuring reliable perception under challenging conditions.

Research on Traffic Scene Element Recognition for Autonomous Driving Based on Deep Learning

This paper introduces the overall research status in the field of autonomous driving, followed by an overview of the research status of multi-lane line recognition algorithms, 3D object detection algorithms, and multi-target tracking algorithms in autonomous driving. However, the road scene in the real environment is complex and changeable, and there is still room for improvement in the real-time performance and accuracy of existing traffic sign detection and recognition methods.

Multi-Scale Candidate Fusion and Optimization-Based 3D Object Detection Algorithm

To address the issues of target omission and the inclusion of a large number of background points in keypoint sampling for point cloud-based object detection, an improved algorithm based on the PV-RCNN network is introduced. This approach employs both a regional proposal fusion network and weighted Non-Maximum Suppression (NMS) to merge proposals generated at various scales while eliminating redundancy. A segmentation network is utilized to segment foreground points from the original point cloud, and object center points are identified based on these proposals. Gaussian density functions are employed for regional density estimation, which assigns different sampling weights to solve the problem of difficult sampling in sparse areas. Experimental evaluations on the KITTI dataset indicate that the algorithm enhances the average precision at medium difficulty levels by 0.39%, 1.31%, and 0.63%for cars, pedestrians, and cyclists, respectively. Generalization experiments were also conducted on the Waymo dataset. The results suggest that the introduced algorithm achieves higher accuracy compared to most of the existing 3D object detection networks.

VFL3D: A Single-Stage Fine-Grained Lightweight Point Cloud 3D Object Detection Algorithm Based on Voxels

VFL3D: A Single-Stage Fine-Grained Lightweight Point Cloud 3D Object Detection Algorithm Based on Voxels

Enhanced 3D object detection for autonomous driving: A spatial-temporal alignment approach in Bird's Eye View scenarios

This paper presents a novel 3D object detection algorithm designed for Bird's Eye View (BEV) scenarios, which significantly improves detection capabilities by integrating spatial and temporal features. The core of our approach is the spatial-temporal alignment module that efficiently processes information across different time steps and spatial locations, enhancing the precision and robustness of object detection. We employ a temporal self-attention mechanism to capture the motion information of objects over time, allowing the model to correlate features across various time steps for identifying and tracking moving objects. Additionally, a spatial cross-attention mechanism is utilized to focus on spatial features within regions of interest, promoting interactions between features extracted from camera views and BEV queries. Our method also implements temporal feature integration and multi-scale feature fusion to enhance detection stability and accuracy for fast-moving objects and to capture multi-scale context information, respectively. The model employs an enriched feature set post alignment for 3D bounding box prediction, ascertaining the position, dimensions, and orientation of objects. We conducted experiments on two public datasets for autonomous driving nuScenes and Waymo Open Dataset, demonstrating that our method outperforms previous BEVFormer and other state-of-the-art methods in terms of detection accuracy and robustness. The paper concludes with potential future directions for optimizing the BEVFormer model's performance and exploring its application in broader scenarios and tasks.

DopplerPTNet: Object Detection Network with Doppler Velocity Information for FMCW LiDAR Point Cloud

Abstract In the field of autonomous driving, LiDAR plays a crucial role in perception and detection. LiDAR based on Time-of-Flight (ToF) mode can only provide three-dimensional spatial coordinate information of point clouds. In point cloud object detection, the limited feature information of spatial coordinates to some extent restricts the further optimization and improvement of algorithm detection performance. However, LiDAR based on Frequency-Modulated Continuous-Wave (FMCW) mode can not only obtain the three-dimensional spatial coordinates of point clouds, but also directly measure the Doppler velocity information of points, effectively compensating for the limitation of relying solely on spatial coordinate information for object recognition. Therefore, based on the CARLA simulator, we construct the first FMCW LiDAR point cloud object detection simulation dataset, FMCWLidDet. What’s more, a novel 4D object detection algorithm, DopplerPTNet, is proposed based on the direct acquisition of point Doppler velocity information by FMCW LiDAR. The algorithm solves the problem of single spatial coordinate information feature in existing 3D object detection algorithms, which makes it difficult to further improve detection accuracy. The dataset is available at https://github.com/xilight123/FMCW-LiDAR-object-detection-dataset.

LiDAR-Based 3D Temporal Object Detection via Motion-Aware LiDAR Feature Fusion.

Recently, the growing demand for autonomous driving in the industry has led to a lot of interest in 3D object detection, resulting in many excellent 3D object detection algorithms. However, most 3D object detectors focus only on a single set of LiDAR points, ignoring their potential ability to improve performance by leveraging the information provided by the consecutive set of LIDAR points. In this paper, we propose a novel 3D object detection method called temporal motion-aware 3D object detection (TM3DOD), which utilizes temporal LiDAR data. In the proposed TM3DOD method, we aggregate LiDAR voxels over time and the current BEV features by generating motion features using consecutive BEV feature maps. First, we present the temporal voxel encoder (TVE), which generates voxel representations by capturing the temporal relationships among the point sets within a voxel. Next, we design a motion-aware feature aggregation network (MFANet), which aims to enhance the current BEV feature representation by quantifying the temporal variation between two consecutive BEV feature maps. By analyzing the differences and changes in the BEV feature maps over time, MFANet captures motion information and integrates it into the current feature representation, enabling more robust and accurate detection of 3D objects. Experimental evaluations on the nuScenes benchmark dataset demonstrate that the proposed TM3DOD method achieved significant improvements in 3D detection performance compared with the baseline methods. Additionally, our method achieved comparable performance to state-of-the-art approaches.

Sparse Embedded Convolution Based Dual Feature Aggregation 3D Object Detection Network

The algorithm design of compatible detection speed and accuracy based on LiDAR point clouds is a challenging issue in various practical applications of 3D object detection, including the field of autonomous driving. This paper designs a single-stage object detection algorithm that is lightweight and compatible with detection speed and accuracy for the above issue. To achieve these objectives, we propose a framework for a 3D object detection algorithm using a single-stage detection network as the backbone network. Firstly, we design a dual feature extraction module to reduce the occurrence of vehicle miss and error detection problems. Then, we use a multi-scale feature fusion scheme to fuse feature information with different scales. Furthermore, we design a data enhancement scheme suitable for this network architecture. Experimental results in the KITTI dataset show that the proposed method achieves improvement ratios of 38.5% for the detection speed and 2.88% ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document} 13.65% in terms of the average precision of vehicle detection compared to the existing algorithm based on single-stage object detection (SECOND).

BEV Space 3D Object Detection Algorithm Based on Fusion of Infrared Camera and LiDAR

BEV Space 3D Object Detection Algorithm Based on Fusion of Infrared Camera and LiDAR

Multiattention Mechanism 3D Object Detection Algorithm Based on RGB and LiDAR Fusion for Intelligent Driving.

This paper proposes a multimodal fusion 3D target detection algorithm based on the attention mechanism to improve the performance of 3D target detection. The algorithm utilizes point cloud data and information from the camera. For image feature extraction, the ResNet50 + FPN architecture extracts features at four levels. Point cloud feature extraction employs the voxel method and FCN to extract point and voxel features. The fusion of image and point cloud features is achieved through regional point fusion and voxel fusion methods. After information fusion, the Coordinate and SimAM attention mechanisms extract fusion features at a deep level. The algorithm's performance is evaluated using the DAIR-V2X dataset. The results show that compared to the Part-A2 algorithm; the proposed algorithm improves the mAP value by 7.9% in the BEV view and 7.8% in the 3D view at IOU = 0.5 (cars) and IOU = 0.25 (pedestrians and cyclists). At IOU = 0.7 (cars) and IOU = 0.5 (pedestrians and cyclists), the mAP value of the SECOND algorithm is improved by 5.4% in the BEV view and 4.3% in the 3D view, compared to other comparison algorithms.

YOLOv7-3D: A Monocular 3D Traffic Object Detection Method from a Roadside Perspective

Current autonomous driving systems predominantly focus on 3D object perception from the vehicle’s perspective. However, the single-camera 3D object detection algorithm in the roadside monitoring scenario provides stereo perception of traffic objects, offering more accurate collection and analysis of traffic information to ensure reliable support for urban traffic safety. In this paper, we propose the YOLOv7-3D algorithm specifically designed for single-camera 3D object detection from a roadside viewpoint. Our approach utilizes various information, including 2D bounding boxes, projected corner keypoints, and offset vectors relative to the center of the 2D bounding boxes, to enhance the accuracy of 3D object bounding box detection. Additionally, we introduce a 5-layer feature pyramid network (FPN) structure and a multi-scale spatial attention mechanism to improve feature saliency for objects of different scales, thereby enhancing the detection accuracy of the network. Experimental results demonstrate that our YOLOv7-3D network achieved significantly higher detection accuracy on the Rope3D dataset while reducing computational complexity by 60%.

2D and 3D object detection algorithms from images: A Survey

2D and 3D object detection algorithms from images: A Survey

Pseudo-Mono for Monocular 3D Object Detection in Autonomous Driving

Current monocular 3D object detection algorithms generally suffer from inaccurate depth estimation, which leads to reduction of detection accuracy. The depth error from image-to-image generation for the stereo view is insignificant compared with the gap in single-image generation. Therefore, a novel pseudo-monocular 3D object detection framework is proposed, which is called Pseudo-Mono. Particularly, stereo images are brought into monocular 3D detection. Firstly, stereo images are taken as input, then a lightweight depth predictor is used to generate the depth map of input images. Secondly, the left input images obtained from stereo camera are used as subjects, which generate enhanced visual feature and multi-scale depth feature by depth indexing and feature matching probabilities, respectively. Finally, sparse anchors set by the foreground probability maps and the multi-scale feature maps are used as reference points to find the suitable initialization approach of object query. The encoded visual feature is adopted to enhance object query for enabling deep interaction between visual feature and depth feature. Compared with popular monocular 3D object detection methods, Pseudo-Mono is able to achieve richer fine-grained information without additional data input. Extensive experimental results on the datasets of KITTI, NuScenes, and MS-COCO demonstrate the generalizability and portability of the proposed method. The effectiveness and efficiency of Pseudo-Mono have been demonstrated by extensive ablation experiments. Experiments on a real vehicle platform have shown that the proposed method maintains high performance in complex real-world environments.

An efficient 3D object detection method based on Fast Guided Anchor Stereo RCNN

In most binocular 3D detection algorithms, a large number of anchor points need to be selected, which leads to the problem of slow feature extraction. To solve this problem, an anchor-guided 3D object detection algorithm for autonomous driving is proposed based on Stereo Recurrent Convolutional Neutral Network (Stereo RCNN), which is called Fast Guided Anchored Stereo RCNN (FGAS RCNN). The proposed FGAS framework is divided into two stages. In the first stage, a probability map is generated for the left and right input images to determine the foreground position. Sparse anchor points and corresponding sparse anchor boxes are generated from the prior information. Left and right anchors are used as a whole to generate a 2D preselection box. In the second stage, a Feature Pyramid Network (FPN) based on key point generation network is used to generate key points, which are combined with stereo regression to generate 3D preselected boxes. Finally, instance-level disparity estimation is proposed to solve the problem of pixel-level information loss in the original image. Instance-level disparity is combined with instance segmentation masks to improve the accuracy of center depth on the 3D bounding box. Extensive experiments on the challenging Kitti dataset and NuScences dataset show that the proposed method reduces the computational cost while maintaining a high regression rate without any depth information and prior information of position. Compared to other methods, the proposed method has higher efficiency, better robustness and stronger generalization ability.

3D object detection algorithm based on multi-sensor segmental fusion of frustum association for autonomous driving

3D object detection algorithm based on multi-sensor segmental fusion of frustum association for autonomous driving

3D Point Cloud Object Detection Algorithm Based on Temporal Information Fusion and Uncertainty Estimation

In autonomous driving, LiDAR (light detection and ranging) data are acquired over time. Most existing 3D object detection algorithms propose the object bounding box by processing each frame of data independently, which ignores the temporal sequence information. However, the temporal sequence information is usually helpful to detect the object with missing shape information due to long distance or occlusion. To address this problem, we propose a temporal sequence information fusion 3D point cloud object detection algorithm based on the Ada-GRU (adaptive gated recurrent unit). In this method, the feature of each frame for the LiDAR point cloud is extracted through the backbone network and is fed to the Ada-GRU together with the hidden features of the previous frames. Compared to the traditional GRU, the Ada-GRU can adjust the gating mechanism adaptively during the training process by introducing the adaptive activation function. The Ada-GRU outputs the temporal sequence fusion features to predict the 3D object in the current frame and transmits the hidden features of the current frame to the next frame. At the same time, the label uncertainty of the distant and occluded objects affects the training effect of the model. For this problem, this paper proposes a probability distribution model of 3D bounding box coordinates based on the Gaussian distribution function and designs the corresponding bounding box loss function to enable the model to learn and estimate the uncertainty of the positioning of the bounding box coordinates, so as to remove the bounding box with large positioning uncertainty in the post-processing stage to reduce the false positive rate. Finally, the experiments show that the methods proposed in this paper improve the accuracy of the object detection without significantly increasing the complexity of the algorithm.

Multi-Modal 3D Object Detection in Autonomous Driving: A Survey

The past decade has witnessed the rapid development of autonomous driving systems. However, it remains a daunting task to achieve full autonomy, especially when it comes to understanding the ever-changing, complex driving scenes. To alleviate the difficulty of perception, self-driving vehicles are usually equipped with a suite of sensors (e.g., cameras, LiDARs), hoping to capture the scenes with overlapping perspectives to minimize blind spots. Fusing these data streams and exploiting their complementary properties is thus rapidly becoming the current trend. Nonetheless, combining data that are captured by different sensors with drastically different ranging/ima-ging mechanisms is not a trivial task; instead, many factors need to be considered and optimized. If not careful, data from one sensor may act as noises to data from another sensor, with even poorer results by fusing them. Thus far, there has been no in-depth guidelines to designing the multi-modal fusion based 3D perception algorithms. To fill in the void and motivate further investigation, this survey conducts a thorough study of tens of recent deep learning based multi-modal 3D detection networks (with a special emphasis on LiDAR-camera fusion), focusing on their fusion stage (i.e., when to fuse), fusion inputs (i.e., what to fuse), and fusion granularity (i.e., how to fuse). These important design choices play a critical role in determining the performance of the fusion algorithm. In this survey, we first introduce the background of popular sensors used for self-driving, their data properties, and the corresponding object detection algorithms. Next, we discuss existing datasets that can be used for evaluating multi-modal 3D object detection algorithms. Then we present a review of multi-modal fusion based 3D detection networks, taking a close look at their fusion stage, fusion input and fusion granularity, and how these design choices evolve with time and technology. After the review, we discuss open challenges as well as possible solutions. We hope that this survey can help researchers to get familiar with the field and embark on investigations in the area of multi-modal 3D object detection.

Monocular 3D Object Detection Based on Pseudo Multimodal Information Extraction and Keypoint Estimation

Three-dimensional object detection is an essential and fundamental task in the field of computer vision which can be widely used in various scenarios such as autonomous driving and visual navigation. In view of the current insufficient utilization of image information in current monocular camera-based 3D object detection algorithms, we propose a monocular 3D object detection algorithm based on pseudo-multimodal information extraction and keypoint estimation. We utilize the original image to generate pseudo-lidar and a bird’s-eye view, and then feed the fused data of the original image and pseudo-lidar to the keypoint-based network for an initial 3D box estimation, finally using the bird’s-eye view to refine the initial 3D box. The experimental performance of our method exceeds state-of-the-art algorithms under the evaluation criteria of 3D object detection and localization on the KITTI dataset, achieving the best experimental performance so far.

3D object detection for autonomous driving from image: a survey——benchmarks, constraints and error analysis

从高分辨率图像中获取周边目标的精准3D位置和尺寸信息是实现自动驾驶控制和行为决策的基础，因此基于图像的3D目标检测是自动驾驶领域中的研究热点。已有学者对该领域方法论及成果进行了比较详细的综述，但对于导致现有方法检测精度不尽如意的制约因素未能进行深入系统的分析。考虑自动驾驶领域在工程应用方面的要求高，且现有方法以数据驱动类型为主，本文从常用数据集和评价基准、数据影响、方法论的制约因素和误差等角度，对学术界和产业界在3D目标检测方面的研究成果及行业应用进行较为系统的阐述。首先，从学术界探索成果以及自动驾驶行业的应用角度进行概要介绍。然后，从数据采集设备、数据精度和标注信息3方面详细分析总结了KITTI等4个通用数据集，并对这些数据集提出的主要评价指标进行对比分析。接着，从数据和方法论方面分析制约算法性能的主要因素及由此造成的误差影响。在数据方面，制约因素主要是数据精度、样本差异、标注数据量和标注规范；在方法论方面，制约因素主要包括先验几何关系、深度预测误差和数据模态等。最后，对国内外研究现状进行总结，并在数据集、评价指标和目标深度预测等方面提出了未来需要重点关注的研究方向。;Autonomous driving-oriented accurate perception and measurement of the three-dimensional(3D) spatial position and scale can be as the basis for realizing the control ability and decision-making level. Sensing technology-driven autonomous vehicles are equipped with high-resolution camera, light detection and ranging (LiDAR), radar, global positioning system (GPS)/inertial measurement unit (IMU) and other related sensors. Current LiDAR or multi-modal data-based 3D object detection algorithms are challenged for its deployment and application because of the shortcomings of LiDAR sensors like high price, limited sensing range, and sparse point clouds data. In contrast, such high-resolution cameras are commonly-used and featured by its lower price, and it can obtain high-resolution spatial information, richer shape, and appearance details as well. The emerging image-based 3D object detection is focused on further. At present, constraints of detection accuracy of the existing methods are still to be analyzed thoroughly and systematically. We summary the research results and industrial applications in relevance to such 1) perspectives of commonly used datasets and evaluation criteria, 2) data impact, 3) methodological constraints and prediction errors. First, a brief introduction is linked to perspective of academic domain and application of autonomous driving industry. We briefly review latest growths of Baidu Apollo, Google Waymo, Tesla and other related autonomous driving companies, and the thread of 3D object detection methods for autonomous driving. Then, we analyzed and summarized four popular datasets like KITTI, nuScenes, Waymo open dataset, and DAIR-V2X dataset from three aspects of:1) data acquisition/sensors, data accuracy and data label information;2) key evaluation standards proposed by these data sets, and 3) pros/cons and applicability of these evaluation standards. Third, main constraints of the image-based 3D object detection algorithm and the errors are derived from two sides of:data and methodology. Such main data constraints are originated from their data accuracy, sample difference, data volume, and data annotation. The data accuracy is mainly limited by equipment performance. The sample difference is mainly restricted by such image processing problems in related to object distance difference, angle difference, occlusion, and truncation. Data volume is affected by variety of 3D data types and high difficulty of labeling. The volume of 3D object detection data set is much smaller in comparison with the 2D object detection data set. Data annotation is mainly focused on 3D bounding box labeling, the labeling details, and quality of the dataset, especially for image annotation used in image-based 3D object detection. For non-rigid objects like pedestrians, the annotation error is larger, and there are some optimal for improving the labeling method. The general framework of image-based 3D object detection can be classified as one-stage methods and two-stage methods, and the limitations consists of 1) the prior geometric relationship, 2) depth prediction accuracy, and 3) data modality. The prior geometric relationship is focused on 2D-3D geometric constraints for 2D images-projected 3D objects and objects-between position relationships. The image-based 3D object detection methods face such problems as:prior 2D-3D geometric constraints and occluded and truncated objects. The prediction of depth information from 2D images is an ill conditioned problem, and dimension collapse will cause depth prediction error-relevant loss of depth information in the image. On the one hand, the depth prediction is often not accurate due to the influence of projection relationship. On the other hand, the performance of continuous depth prediction is often poor at the depth mutation of the image(such as edge of objects). When the prediction depth is discretized, there is a problem that the classification of depth is relatively rough, and the accuracy classification cannot be arbitrarily divided. The limitation of single image-based data modality is mainly reflected via large error of depth prediction. The detection performance of the algorithm can be optimized by 1) simulating the stereo signal and LiDAR point clouds, or 2) using stereo image as the aided input, or 3) leveraging point clouds data with accurate 3D information as supervision signal. In addition, video data can be adopted to improve the detection accuracy to a certain extent. Forth, current research situation is summarized and compared from academic and industrial domain. Finally, some future research directions are predicted in terms of such factors of datasets, evaluation indicators, and depth prediction.

X-view: Non-egocentric Multi-View 3D Object Detector.

3D object detection algorithms for autonomous driving reason about 3D obstacles either from 3D birds-eye view or perspective view or both. Recent works attempt to improve the detection performance via mining and fusing from multiple egocentric views. Although the egocentric perspective view alleviates some weaknesses of the birds-eye view, the sectored grid partition becomes so coarse in the distance that the targets and surrounding context mix together, which makes the features less discriminative. In this paper, we generalize the research on 3D multi-view learning and propose a novel multi-view-based 3D detection method, named X-view, to overcome the drawbacks of the multi-view methods. Specifically, X-view breaks through the traditional limitation about the perspective view whose original point must be consistent with the 3D Cartesian coordinate. X-view is designed as a general paradigm that can be applied on almost any 3D detectors based on LiDAR with only little increment of running time, no matter it is voxel/grid-based or raw-point-based. We conduct experiments on KITTI [1] and NuScenes [2] datasets to demonstrate the robustness and effectiveness of our proposed X-view. The results show that X-view obtains consistent improvements when combined with mainstream state-of-the-art 3D methods.