3D Object Detection Method Research Articles

BRTPillar: boosting real-time 3D object detection based point cloud and RGB image fusion in autonomous driving

PurposeIn autonomous driving, the inherent sparsity of point clouds often limits the performance of object detection, while existing multimodal architectures struggle to meet the real-time requirements for 3D object detection. Therefore, the main purpose of this paper is to significantly enhance the detection performance of objects, especially the recognition capability for small-sized objects and to address the issue of slow inference speed. This will improve the safety of autonomous driving systems and provide feasibility for devices with limited computing power to achieve autonomous driving.Design/methodology/approachBRTPillar first adopts an element-based method to fuse image and point cloud features. Secondly, a local-global feature interaction method based on an efficient additive attention mechanism was designed to extract multi-scale contextual information. Finally, an enhanced multi-scale feature fusion method was proposed by introducing adaptive spatial and channel interaction attention mechanisms, thereby improving the learning of fine-grained features.FindingsExtensive experiments were conducted on the KITTI dataset. The results showed that compared with the benchmark model, the accuracy of cars, pedestrians and cyclists on the 3D object box improved by 3.05, 9.01 and 22.65%, respectively; the accuracy in the bird’s-eye view has increased by 2.98, 10.77 and 21.14%, respectively. Meanwhile, the running speed of BRTPillar can reach 40.27 Hz, meeting the real-time detection needs of autonomous driving.Originality/valueThis paper proposes a boosting multimodal real-time 3D object detection method called BRTPillar, which achieves accurate location in many scenarios, especially for complex scenes with many small objects, while also achieving real-time inference speed.

A monocular three-dimensional object detection model based on uncertainty-guided depth combination for autonomous driving

Three-Dimensional (3D) object detection is a crucial task for enhancing safety and efficiency in autonomous driving. However, estimating depth from monocular images remains a challenging task. Most existing monocular 3D object detection methods rely on additional auxiliary data sources to compensate for the lack of spatial information in monocular images. Nevertheless, these methods bring substantial computational overhead and time-consuming preprocessing steps. To address this issue, we propose a novel depth estimation method for monocular images that does not rely on any auxiliary information. Leveraging both the texture and geometric cues of detected objects, our method generates two depth estimates for each object based on the extracted Region of Interest (RoI) features: a direct depth estimate and a height-based depth estimate with uncertainty modeling. Our model dynamically assigns weights to these depth estimates based on their respective uncertainties and combines them to obtain the final depth. During the training process, the model assigns higher weights to depth branches with higher uncertainties, as these estimates exhibit greater tolerance to errors. As the combined depth network introduces increased complexity, we utilize Group Normalization (GN) to better capture spatial information in the prediction branch outputs. Furthermore, we leverage the Two-Dimensional (2D) information of objects to predict the residual in the 2D center after downsampling, aiding in the regression of 3D center. On the KITTI benchmark, our model achieves an average precision (AP) of 16.65 % and 23.19 % on 3D and bird's-eye view (BEV) detection for the moderate category, surpassing the state-of-the-art (SOTA) models in each category.

MS3D: A Multi-Scale Feature Fusion 3D Object Detection Method for Autonomous Driving Applications

With advancements in autonomous driving, LiDAR has become central to 3D object detection due to its precision and interference resistance. However, challenges such as point cloud sparsity and unstructured data persist. This study introduces MS3D (Multi-Scale Feature Fusion 3D Object Detection Method), a novel approach to 3D object detection that leverages the architecture of a 2D Convolutional Neural Network (CNN) as its core framework. It integrates a Second Feature Pyramid Network to enhance multi-scale feature representation and contextual integration. The Adam optimizer is employed for efficient adaptive parameter tuning, significantly improving detection performance. On the KITTI dataset, MS3D achieves average precisions of 93.58%, 90.91%, and 88.46% in easy, moderate, and hard scenarios, respectively, surpassing state-of-the-art models like VoxelNet, SECOND, and PointPillars.

MonoFG: Monocular 3D Object Detection with Knowledge Distillation for Human-Centric Autonomous Driving Systems

Monocular 3D object detection is essential for identifying objects in road images, thus offering valuable environmental perception data that are crucial for human-centric autonomous driving systems. However, due to the inherent limitations of camera imaging, obtaining precise depth information from images alone is challenging, which hampers the accuracy of within-scene object localization. In this paper, we introduce a monocular 3D object detection method called MonoFG that uses knowledge distillation with separated foreground and background components to improve the accuracy of object localization. First, detached foreground and background distillation processes can strategically leverage the distinct positional information acquired from each location to optimize the produced global distillation effects. This step serves as the foundation for the subsequent feature and response distillation process, which focuses on the distilled foreground and background rather than isolated object distillation. Second, triple attention mechanism-based feature distillation intensifies the feature imitation and feature representation capabilities of the student network. Spatial and channel attention mechanisms encourage the student network to capture crucial pixels and channels from the teacher network, whereas a self-attention mechanism globally transfers the learned relationships between pixels. Third, localization error-based response distillation facilitates a clearer transfer of positional information from the teacher network to the student network. Only when the positioning ability of the teacher network exceeds that of the student network can knowledge be comprehensively distilled across both the foreground and background. Therefore, the distillation process is constrained to specific content, which is delineated by positioning errors that serve as the boundaries. Finally, experiments conducted on the KITTI benchmark dataset demonstrate that our method outperforms many well-known baseline methods in several representative evaluation tasks (e.g., 3D object detection and bird's-eye view (BEV) detection) involving human-centric autonomous driving systems.

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

DeployFusion: A Deployable Monocular 3D Object Detection with Multi-Sensor Information Fusion in BEV for Edge Devices.

To address the challenges of suboptimal remote detection and significant computational burden in existing multi-sensor information fusion 3D object detection methods, a novel approach based on Bird's-Eye View (BEV) is proposed. This method utilizes an enhanced lightweight EdgeNeXt feature extraction network, incorporating residual branches to address network degradation caused by the excessive depth of STDA encoding blocks. Meantime, deformable convolution is used to expand the receptive field and reduce computational complexity. The feature fusion module constructs a two-stage fusion network to optimize the fusion and alignment of multi-sensor features. This network aligns image features to supplement environmental information with point cloud features, thereby obtaining the final BEV features. Additionally, a Transformer decoder that emphasizes global spatial cues is employed to process the BEV feature sequence, enabling precise detection of distant small objects. Experimental results demonstrate that this method surpasses the baseline network, with improvements of 4.5% in the NuScenes detection score and 5.5% in average precision for detection objects. Finally, the model is converted and accelerated using TensorRT tools for deployment on mobile devices, achieving an inference time of 138 ms per frame on the Jetson Orin NX embedded platform, thus enabling real-time 3D object detection.

PE-MCAT: Leveraging Image Sensor Fusion and Adaptive Thresholds for Semi-Supervised 3D Object Detection.

Existing 3D object detection frameworks in sensor-based applications heavily rely on large-scale annotated data to achieve optimal performance. However, obtaining such annotations from sensor data-like LiDAR or image sensors-is both time-consuming and costly. Semi-supervised learning offers an efficient solution to this challenge and holds significant potential for sensor-driven artificial intelligence (AI) applications. While it reduces the need for labeled data, semi-supervised learning still depends on a small amount of labeled samples for training. In the initial stages, relying on such limited samples can adversely affect the effective training of student-teacher networks. In this paper, we propose PE-MCAT, a semi-supervised 3D object detection method that generates high-precision pseudo-labels. First, to address the challenges of insufficient local feature capture and poor robustness in point cloud data, we introduce a point enrichment module. This module incorporates information from image sensors and combines multiple feature fusion methods of local and self-features to directly enhance the quality of point clouds and pseudo-labels, compensating for the limitations posed by using only a few labeled samples. Second, we explore the relationship between the teacher network and the pseudo-labels it generates. We propose a multi-class adaptive threshold strategy to initially filter and create a high-quality pseudo-label set. Furthermore, a joint variable threshold strategy is introduced to refine this set further, enhancing the selection of superior pseudo-labels.Extensive experiments demonstrate that PE-MCAT consistently outperforms recent state-of-the-art methods across different datasets. Specifically, on the KITTI dataset and using only 2% of labeled samples, our method improved the mean Average Precision (mAP) by 0.7% for cars, 3.7% for pedestrians, and 3.0% for cyclists.

MonoCAPE: Monocular 3D object detection with coordinate-aware position embeddings

3D monocular detection remains to be a focal point of research, particularly due to its capacity to deliver available precision under conditions of low cost and simplified configurations, making it especially valuable in fields like autonomous driving. Current 3D object detection methods often overlook the spatial information missing from images, which is critical to spatial perception, and optimize bounding box attributes separately, failing to meet the requirements of autonomous driving. We introduce MonoCAPE, a novel 3D detection framework addressing these issues by encoding spatial information and co-optimizing attributes through a Coordinate-Aware Position Encoding (CAPE) Generator and a Task Co-optimization Strategy (TCS). The CAPE Generator produces sparse positional embeddings, enabling spatial awareness with low computational cost, while the TCS utilizes Gaussian modeling to prevent suboptimal outputs. In this way, our framework comprehensively takes into account what existing approaches ignore. Extensive experiments on the KITTI dataset demonstrate MonoCAPE significantly improves AP3D and APBEV metrics compared to existing advanced methods.

Three-Dimensional Outdoor Object Detection in Quadrupedal Robots for Surveillance Navigations

Quadrupedal robots are confronted with the intricate challenge of navigating dynamic environments fraught with diverse and unpredictable scenarios. Effectively identifying and responding to obstacles is paramount for ensuring safe and reliable navigation. This paper introduces a pioneering method for 3D object detection, termed viewpoint feature histograms, which leverages the established paradigm of 2D detection in projection. By translating 2D bounding boxes into 3D object proposals, this approach not only enables the reuse of existing 2D detectors but also significantly increases the performance with less computation required, allowing for real-time detection. Our method is versatile, targeting both bird’s eye view objects (e.g., cars) and frontal view objects (e.g., pedestrians), accommodating various types of 2D object detectors. We showcase the efficacy of our approach through the integration of YOLO3D, utilizing LiDAR point clouds on the KITTI dataset, to achieve real-time efficiency aligned with the demands of autonomous vehicle navigation. Our model selection process, tailored to the specific needs of quadrupedal robots, emphasizes considerations such as model complexity, inference speed, and customization flexibility, achieving an accuracy of up to 99.93%. This research represents a significant advancement in enabling quadrupedal robots to navigate complex and dynamic environments with heightened precision and safety.

KPTr: Key point transformer for LiDAR-based 3D object detection

Accurate 3D environmental perception is crucial for ensuring the safety of autonomous vehicles. However, many existing point-based 3D object detection methods rely only on independent point features to extract spatial information and ignore the dependencies between points. Meanwhile, these methods usually use max-pooling operations to aggregate neighboring point features without considering the stability of local geometric information. To address these issues, this paper proposes a novel two-stage LiDAR-based key point transformer that consists of a region proposal network module and a region-based transformer module to enhance 3D detection performance. Firstly, unprocessed raw point clouds are taken as the input of the model to preserve detailed spatial structural information. Secondly, in the region proposal network an enhanced PointNet++ is used to segment foreground points and generate high-quality first-stage proposals. Then, the representation of proposals is improved by the region-based transformer module, which constructs long-range key point cloud relationship features within the regions of interest and diffuses spatial information into the feature channel layers. Additionally experiments on the KITTI dataset show that detection performance of the proposed method surpasses most mainstream 3D object detection methods and achieves a detection speed of 14.28 Hz, striking a good balance between detection accuracy and speed. Finally, the proposed method is deployed on real vehicle hardware platforms for online detection, thereby proving its industrial value.

AEPF: Attention-Enabled Point Fusion for 3D Object Detection.

Current state-of-the-art (SOTA) LiDAR-only detectors perform well for 3D object detection tasks, but point cloud data are typically sparse and lacks semantic information. Detailed semantic information obtained from camera images can be added with existing LiDAR-based detectors to create a robust 3D detection pipeline. With two different data types, a major challenge in developing multi-modal sensor fusion networks is to achieve effective data fusion while managing computational resources. With separate 2D and 3D feature extraction backbones, feature fusion can become more challenging as these modes generate different gradients, leading to gradient conflicts and suboptimal convergence during network optimization. To this end, we propose a 3D object detection method, Attention-Enabled Point Fusion (AEPF). AEPF uses images and voxelized point cloud data as inputs and estimates the 3D bounding boxes of object locations as outputs. An attention mechanism is introduced to an existing feature fusion strategy to improve 3D detection accuracy and two variants are proposed. These two variants, AEPF-Small and AEPF-Large, address different needs. AEPF-Small, with a lightweight attention module and fewer parameters, offers fast inference. AEPF-Large, with a more complex attention module and increased parameters, provides higher accuracy than baseline models. Experimental results on the KITTI validation set show that AEPF-Small maintains SOTA 3D detection accuracy while inferencing at higher speeds. AEPF-Large achieves mean average precision scores of 91.13, 79.06, and 76.15 for the car class's easy, medium, and hard targets, respectively, in the KITTI validation set. Results from ablation experiments are also presented to support the choice of model architecture.

Spatial awareness enhancement based single-stage anchor-free 3D object detection for autonomous driving

The real-time and accurate detection of three-dimensional (3D) objects based on LiDAR is a focal problem in the field of autonomous driving environment perception. Compared to two-stage and anchor-based 3D object detection methods that suffer from inference latency challenges, single-stage anchor-free 3D object detection approaches are more suitable for deployment in autonomous driving vehicles with the strict real-time requirement. However, they face the issue of insufficient spatial awareness, which can result in detection errors such as false positives and false negatives, thereby increasing the potential risks of autonomous driving. In response to this, we focus on enhancing the spatial awareness of CenterPoint, a widely used single-stage anchor-free 3D object detector in the industry. Considering the limited allocation of computational resources and the performance bottleneck caused by pillar encoder, we propose an efficient SSDCM backbone to strengthen feature representation and extraction. Furthermore, a simple BGC neck is devised to weight and exchange contextual information in order to deeply fuse multi-scale features. Combining improved backbone and neck networks, we construct a single-stage anchor-free 3D object detection model with spatial awareness enhancement, named CenterPoint-Spatial Awareness Enhancement (CenterPoint-SAE). We evaluate CenterPoint-SAE on two large-scale and challenging autonomous driving datasets, nuScenes and Waymo. It achieves 53.3% mAP and 62.5% NDS on nuScenes detection benchmark, and runs inference at a speed of 11.1 FPS. Compared to the baseline, the upgraded networks deliver a performance improvement of 1.6% mAP and 1.2% NDS at minor cost. Notably, on Waymo dataset, our method achieves competitive detection performance compared to two-stage and point-based methods.

Small Object Detection with lightweight PointNet Based on Attention Mechanisms

Abstract In response to the issue of subpar small object detection outcomes, the improved 3D object detection method for Point Net is proposed. First, multi-scale fusion method is added to the channel attention mechanism into the feature extraction stage of Point Net; then, the loss function is adjusted to optimize the backbone network and reduce the number of model parameters while improving the accuracy. The experimental results show that in the dataset, the proposed method has higher detection accuracy, the improved algorithm average accuracy improves 3.9%, the model size decreases 20%, effectively improves the detection effect of small objects, and realizes the lightweight of the network.

A Systematic Survey of Transformer-Based 3D Object Detection for Autonomous Driving: Methods, Challenges and Trends

In recent years, with the continuous development of autonomous driving technology, 3D object detection has naturally become a key focus in the research of perception systems for autonomous driving. As the most crucial component of these systems, 3D object detection has gained significant attention. Researchers increasingly favor the deep learning framework Transformer due to its powerful long-term modeling ability and excellent feature fusion advantages. A large number of excellent Transformer-based 3D object detection methods have emerged. This article divides the methods based on data sources. Firstly, we analyze different input data sources and list standard datasets and evaluation metrics. Secondly, we introduce methods based on different input data and summarize the performance of some methods on different datasets. Finally, we summarize the limitations of current research, discuss future directions and provide some innovative perspectives.

Boundary points guided 3D object detection for point clouds

3D object detection in LiDAR point clouds is crucial for computer vision tasks such as autonomous driving. The two-stage approach with point cloud completion achieves remarkable performance by generating semantic surface points for foreground objects or learning bird’s eye view shape heat map labels. However, these methods require additional completion datasets, leading to substantial computation and memory demands. In this context, we propose a Boundary Points Guided 3D (BPG3D) object detection method that complements point cloud boundary information without the need for additional data. Specifically, we generate Region of Interest (RoI) boundary points to aggregate the neighbor voxel information at the RoI boundary during the refinement stage to complement the missing boundary information. Meanwhile, we design a Dual Feature Selection (DFS) module to adaptively fuse RoI grid point features and RoI boundary point features for bounding box refinement with negligible computational cost. Additionally, inspired by tensor decomposition theory, we use low-rank tensors to reconstruct high-rank tensors in the point cloud feature encoder to enhance contextual semantic information. The proposed method achieves 65.81% mAP on KITTI Test Set, obtaining a good trade-off between accuracy and efficiency.

MS23D: A 3D object detection method using multi-scale semantic feature points to construct 3D feature layer

LiDAR point clouds can effectively depict the motion and posture of objects in three-dimensional space. Many studies accomplish the 3D object detection by voxelizing point clouds. However, in autonomous driving scenarios, the sparsity and hollowness of point clouds create some difficulties for voxel-based methods. The sparsity of point clouds makes it challenging to describe the geometric features of objects. The hollowness of point clouds poses difficulties for the aggregation of 3D features. We propose a two-stage 3D object detection framework, called MS23D. (1) We propose a method using voxel feature points from multi-branch to construct the 3D feature layer. Using voxel feature points from different branches, we construct a relatively compact 3D feature layer with rich semantic features. Additionally, we propose a distance-weighted sampling method, reducing the loss of foreground points caused by downsampling and allowing the 3D feature layer to retain more foreground points. (2) In response to the hollowness of point clouds, we predict the offsets between deep-level feature points and the object’s centroid, making them as close as possible to the object’s centroid. This enables the aggregation of these feature points with abundant semantic features. For feature points from shallow-level, we retain them on the object’s surface to describe the geometric features of the object. To validate our approach, we evaluated its effectiveness on both the KITTI and ONCE datasets.

LiDAR-Based 3D Temporal Object Detection via Motion-Aware LiDAR Feature Fusion.

Recently, the growing demand for autonomous driving in the industry has led to a lot of interest in 3D object detection, resulting in many excellent 3D object detection algorithms. However, most 3D object detectors focus only on a single set of LiDAR points, ignoring their potential ability to improve performance by leveraging the information provided by the consecutive set of LIDAR points. In this paper, we propose a novel 3D object detection method called temporal motion-aware 3D object detection (TM3DOD), which utilizes temporal LiDAR data. In the proposed TM3DOD method, we aggregate LiDAR voxels over time and the current BEV features by generating motion features using consecutive BEV feature maps. First, we present the temporal voxel encoder (TVE), which generates voxel representations by capturing the temporal relationships among the point sets within a voxel. Next, we design a motion-aware feature aggregation network (MFANet), which aims to enhance the current BEV feature representation by quantifying the temporal variation between two consecutive BEV feature maps. By analyzing the differences and changes in the BEV feature maps over time, MFANet captures motion information and integrates it into the current feature representation, enabling more robust and accurate detection of 3D objects. Experimental evaluations on the nuScenes benchmark dataset demonstrate that the proposed TM3DOD method achieved significant improvements in 3D detection performance compared with the baseline methods. Additionally, our method achieved comparable performance to state-of-the-art approaches.

Snow-CLOCs: Camera-LiDAR Object Candidate Fusion for 3D Object Detection in Snowy Conditions.

Although existing 3D object-detection methods have achieved promising results on conventional datasets, it is still challenging to detect objects in data collected under adverse weather conditions. Data distortion from LiDAR and cameras in such conditions leads to poor performance of traditional single-sensor detection methods. Multi-modal data-fusion methods struggle with data distortion and low alignment accuracy, making accurate target detection difficult. To address this, we propose a multi-modal object-detection algorithm, Snow-CLOCs, specifically for snowy conditions. In image detection, we improved the YOLOv5 algorithm by integrating the InceptionNeXt network to enhance feature extraction and using the Wise-IoU algorithm to reduce dependency on high-quality data. For LiDAR point-cloud detection, we built upon the SECOND algorithm and employed the DROR filter to remove noise, enhancing detection accuracy. We combined the detection results from the camera and LiDAR into a unified detection set, represented using a sparse tensor, and extracted features through a 2D convolutional neural network to achieve object detection and localization. Snow-CLOCs achieved a detection accuracy of 86.61% for vehicle detection in snowy conditions.

Intracranial aneurysm detection based on 3D point cloud object detection method

Intracranial aneurysm detection based on 3D point cloud object detection method

Impact of LiDAR point cloud compression on 3D object detection evaluated on the KITTI dataset

The rapid growth on the amount of generated 3D data, particularly in the form of Light Detection And Ranging (LiDAR) point clouds (PCs), poses very significant challenges in terms of data storage, transmission, and processing. Point cloud (PC) representation of 3D visual information has shown to be a very flexible format with many applications ranging from multimedia immersive communication to machine vision tasks in the robotics and autonomous driving domains. In this paper, we investigate the performance of four reference 3D object detection techniques, when the input PCs are compressed with varying levels of degradation. Compression is performed using two MPEG standard coders based on 2D projections and octree decomposition, as well as two coding methods based on Deep Learning (DL). For the DL coding methods, we used a Joint Photographic Experts Group (JPEG) reference PC coder, that we adapted to accept LiDAR PCs in both Cartesian and cylindrical coordinate systems. The detection performance of the four reference 3D object detection methods was evaluated using both pre-trained models and models specifically trained using degraded PCs reconstructed from compressed representations. It is shown that LiDAR PCs can be compressed down to 6 bits per point with no significant degradation on the object detection precision. Furthermore, employing specifically trained detection models improves the detection capabilities even at compression rates as low as 2 bits per point. These results show that LiDAR PCs can be coded to enable efficient storage and transmission, without significant object detection performance loss.