3D Object Detection Research Articles

CO-Net++: A Cohesive Network for Multiple Point Cloud Tasks at Once With Two-Stage Feature Rectification.

We present CO-Net++, a cohesive framework that optimizes multiple point cloud tasks collectively across heterogeneous dataset domains with a two-stage feature rectification strategy. The core of CO-Net++ lies in optimizing task-shared parameters to capture universal features across various tasks while discerning task-specific parameters tailored to encapsulate the unique characteristics of each task. Specifically, CO-Net++ develops a two-stage feature rectification strategy (TFRS) that distinctly separates the optimization processes for task-shared and task-specific parameters. At the first stage, TFRS configures all parameters in backbone as task-shared, which encourages CO-Net++ to thoroughly assimilate universal attributes pertinent to all tasks. In addition, TFRS introduces a sign-based gradient surgery to facilitate the optimization of task-shared parameters, thus alleviating conflicting gradients induced by various dataset domains. In the second stage, TFRS freezes task-shared parameters and flexibly integrates task-specific parameters into the network for encoding specific characteristics of each dataset domain. CO-Net++ prominently mitigates conflicting optimization caused by parameter entanglement, ensuring the sufficient identification of universal and specific features. Extensive experiments reveal that CO-Net++ realizes exceptional performances on both 3D object detection and 3D semantic segmentation tasks. Moreover, CO-Net++ delivers an impressive incremental learning capability and prevents catastrophic amnesia when generalizing to new point cloud tasks.

UADA3D: Unsupervised Adversarial Domain Adaptation for 3D Object Detection With Sparse LiDAR and Large Domain Gaps

UADA3D: Unsupervised Adversarial Domain Adaptation for 3D Object Detection With Sparse LiDAR and Large Domain Gaps

Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection.

The Detection Transformer (DETR) has revolutionized the design of CNN-based object detection systems, showcasing impressive performance. However, its potential in the domain of multi-frame 3D object detection remains largely unexplored. In this paper, we present STEMD, a novel end-to-end framework that enhances the DETR-like paradigm for multi-frame 3D object detection by addressing three key aspects specifically tailored for this task. First, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network, which represents queries as nodes in a graph and enables effective modeling of object interactions within a social context. To solve the problem of missing hard cases in the proposed output of the encoder in the current frame, we incorporate the output of the previous frame to initialize the query input of the decoder. Finally, it poses a challenge for the network to distinguish between the positive query and other highly similar queries that are not the best match. And similar queries are insufficiently suppressed and turn into redundant prediction boxes. To address this issue, our proposed IoU regularization term encourages similar queries to be distinct during the refinement. Through extensive experiments, we demonstrate the effectiveness of our approach in handling challenging scenarios, while incurring only a minor additional computational overhead.

Contextual Attribution Maps-Guided Transferable Adversarial Attack for 3D Object Detection

The study of LiDAR-based 3D object detection and its robustness under adversarial attacks has achieved great progress. However, existing adversarial attack methods mainly focus on the targeted object, which destroys the integrity of the object and makes the attack easy to perceive. In this work, we propose a novel adversarial attack against deep 3D object detection models named the contextual attribution maps-guided attack (CAMGA). Based on the combinations of subregions in the context area and their impact on the prediction results, contextual attribution maps can be generated. An attribution map exposes the influence of individual subregions in the context area on the detection results and narrows down the scope of the adversarial attack. Subsequently, perturbations are generated under the guidance of a dual loss, which is proposed to suppress the detection results and maintain visual imperception simultaneously. The experimental results proved that the CAMGA method achieved an attack success rate of over 68% on three large-scale datasets and 83% on the KITTI dataset. Meanwhile, the CAMGA has a transfer attack success rate of at least 50% against all four victim detectors, as they all overly rely on contextual information.

A Component-based Systematic Review of Pool-based Active Learning for 2D Object Detection

Active learning can help reduce the labeling burden in training deep object detection models by strategically selecting the most informative data points for labeling from unlabeled data. Different approaches to this task have been proposed in recent years, with differences often found in their approaches to the specific components. Consequently, this study breaks down active learning methods into their individual components and examines the diverse strategies associated with each intending to shed light on areas that have not yet been explored in depth. Doing so, the study offers valuable insights into potential research directions in this area.

Research on Traffic Scene Element Recognition for Autonomous Driving Based on Deep Learning

This paper introduces the overall research status in the field of autonomous driving, followed by an overview of the research status of multi-lane line recognition algorithms, 3D object detection algorithms, and multi-target tracking algorithms in autonomous driving. However, the road scene in the real environment is complex and changeable, and there is still room for improvement in the real-time performance and accuracy of existing traffic sign detection and recognition methods.

Sec-CLOCs: Multimodal Back-End Fusion-Based Object Detection Algorithm in Snowy Scenes

LiDAR and cameras, often regarded as the “eyes” of intelligent driving vehicles, are vulnerable to adverse weather conditions like haze, rain, and snow, compromising driving safety. In order to solve this problem and enhance the environmental sensing capability under severe weather conditions, this paper proposes a multimodal back-end fusion object detection method, Sec-CLOCs, which is specifically optimized for vehicle detection under heavy snow. This method achieves object detection by integrating an improved YOLOv8s 2D detector with a SECOND 3D detector. First, the quality of image data is enhanced through the Two-stage Knowledge Learning and Multi-contrastive Regularization (TKLMR) image processing algorithm. Additionally, the DyHead detection head and Wise-IOU loss function are introduced to optimize YOLOv8s and improve 2D detection performance.The LIDROR algorithm preprocesses point cloud data for the SECOND detector, yielding 3D object detection results. The CLOCs back-end fusion algorithm is then employed to merge the 2D and 3D detection outcomes, thereby enhancing overall object detection capabilities. The experimental results show that the Sec-CLOCs algorithm achieves a vehicle detection accuracy of 82.34% in moderate mode (30–100 m) and 81.76% in hard mode (more than 100 m) under heavy snowfall, which demonstrates the algorithm’s high detection performance and robustness.

PLC-Fusion: Perspective-Based Hierarchical and Deep LiDAR Camera Fusion for 3D Object Detection in Autonomous Vehicles

Accurate 3D object detection is essential for autonomous driving, yet traditional LiDAR models often struggle with sparse point clouds. We propose perspective-aware hierarchical vision transformer-based LiDAR-camera fusion (PLC-Fusion) for 3D object detection to address this. This efficient, multi-modal 3D object detection framework integrates LiDAR and camera data for improved performance. First, our method enhances LiDAR data by projecting them onto a 2D plane, enabling the extraction of object perspective features from a probability map via the Object Perspective Sampling (OPS) module. It incorporates a lightweight perspective detector, consisting of interconnected 2D and monocular 3D sub-networks, to extract image features and generate object perspective proposals by predicting and refining top-scored 3D candidates. Second, it leverages two independent transformers—CamViT for 2D image features and LidViT for 3D point cloud features. These ViT-based representations are fused via the Cross-Fusion module for hierarchical and deep representation learning, improving performance and computational efficiency. These mechanisms enhance the utilization of semantic features in a region of interest (ROI) to obtain more representative point features, leading to a more effective fusion of information from both LiDAR and camera sources. PLC-Fusion outperforms existing methods, achieving a mean average precision (mAP) of 83.52% and 90.37% for 3D and BEV detection, respectively. Moreover, PLC-Fusion maintains a competitive inference time of 0.18 s. Our model addresses computational bottlenecks by eliminating the need for dense BEV searches and global attention mechanisms while improving detection range and precision.

MS3D: A Multi-Scale Feature Fusion 3D Object Detection Method for Autonomous Driving Applications

With advancements in autonomous driving, LiDAR has become central to 3D object detection due to its precision and interference resistance. However, challenges such as point cloud sparsity and unstructured data persist. This study introduces MS3D (Multi-Scale Feature Fusion 3D Object Detection Method), a novel approach to 3D object detection that leverages the architecture of a 2D Convolutional Neural Network (CNN) as its core framework. It integrates a Second Feature Pyramid Network to enhance multi-scale feature representation and contextual integration. The Adam optimizer is employed for efficient adaptive parameter tuning, significantly improving detection performance. On the KITTI dataset, MS3D achieves average precisions of 93.58%, 90.91%, and 88.46% in easy, moderate, and hard scenarios, respectively, surpassing state-of-the-art models like VoxelNet, SECOND, and PointPillars.

Advanced Point Cloud Techniques for Improved 3D Object Detection: A Study on DBSCAN, Attention, and Downsampling

To address the challenges of limited detection precision and insufficient segmentation of small to medium-sized objects in dynamic and complex scenarios, such as the dense intermingling of pedestrians, vehicles, and various obstacles in urban environments, we propose an enhanced methodology. Firstly, we integrated a point cloud processing module utilizing the DBSCAN clustering algorithm to effectively segment and extract critical features from the point cloud data. Secondly, we introduced a fusion attention mechanism that significantly improves the network’s capability to capture both global and local features, thereby enhancing object detection performance in complex environments. Finally, we incorporated a CSPNet downsampling module, which substantially boosts the network’s overall performance and processing speed while reducing computational costs through advanced feature map segmentation and fusion techniques. The proposed method was evaluated using the KITTI dataset. Under moderate difficulty, the BEV mAP for detecting cars, pedestrians, and cyclists achieved 87.74%, 55.07%, and 67.78%, reflecting improvements of 1.64%, 5.84%, and 5.53% over PointPillars. For 3D mAP, the detection accuracy for cars, pedestrians, and cyclists reached 77.90%, 49.22%, and 62.10%, with improvements of 2.91%, 5.69%, and 3.03% compared to PointPillars.

Efficient indoor 3D object detection in point clouds using the Kinect sensor

Abstract Only using the point clouds collected by a Kinect sensor to perform 3D object detection is a very challenging task for indoor sense understanding. In this paper, we propose a novel point clouds based 3D object detection framework named Weighted VoteNet (W-VoteNet), which systematically enhances seed points, voting points, and proposal objects to varying degree, thereby enhancing the detection accuracy of indoor scene point clouds. Firstly, a new neighborhood feature enhancement module was designed to enhance the features of the seed point, which can better utilize the neighborhood feature of the seed point. Secondly, a weighted voting module is implemented to increase the accuracy of voting, allowing more prospect seed points to participate in voting. Finally, a new semantic relation reasoning module is proposed to get the semantic relationship features of the proposal object, which can further decrease false alarm rate. The three modules we proposed all contribute to more accurate voting and more effective proposals. The experimental results on two benchmark indoor 3D object detection datasets, SUN RGB-D and ScanNet V2, demonstrate the effectiveness of our approach. The source code is available at: https://github.com/Guo-JQ/W-VoteNet.

MonoFG: Monocular 3D Object Detection with Knowledge Distillation for Human-Centric Autonomous Driving Systems

Monocular 3D object detection is essential for identifying objects in road images, thus offering valuable environmental perception data that are crucial for human-centric autonomous driving systems. However, due to the inherent limitations of camera imaging, obtaining precise depth information from images alone is challenging, which hampers the accuracy of within-scene object localization. In this paper, we introduce a monocular 3D object detection method called MonoFG that uses knowledge distillation with separated foreground and background components to improve the accuracy of object localization. First, detached foreground and background distillation processes can strategically leverage the distinct positional information acquired from each location to optimize the produced global distillation effects. This step serves as the foundation for the subsequent feature and response distillation process, which focuses on the distilled foreground and background rather than isolated object distillation. Second, triple attention mechanism-based feature distillation intensifies the feature imitation and feature representation capabilities of the student network. Spatial and channel attention mechanisms encourage the student network to capture crucial pixels and channels from the teacher network, whereas a self-attention mechanism globally transfers the learned relationships between pixels. Third, localization error-based response distillation facilitates a clearer transfer of positional information from the teacher network to the student network. Only when the positioning ability of the teacher network exceeds that of the student network can knowledge be comprehensively distilled across both the foreground and background. Therefore, the distillation process is constrained to specific content, which is delineated by positioning errors that serve as the boundaries. Finally, experiments conducted on the KITTI benchmark dataset demonstrate that our method outperforms many well-known baseline methods in several representative evaluation tasks (e.g., 3D object detection and bird's-eye view (BEV) detection) involving human-centric autonomous driving systems.

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

Point cloud segmentation method based on an image mask and its application verification

Abstract Accurately perceiving three-dimensional (3D) environments or objects is crucial for the advancement of artificial intelligence interaction technologies. Currently, various types of sensors are employed to obtain point cloud data for 3D object detection or segmentation tasks. While this multi-sensor approach provides more precise 3D data than monocular or stereo cameras, it is also more expensive. The advent of RGB-D cameras, which provide both RGB images and depth information, addresses this issue. In this study, we propose a point cloud segmentation method based on image masks. By using an RGB-D camera to capture color and depth images, we generate image masks through object recognition and segmentation. Given the mapping relationship between RGB image pixels and point clouds, these image masks can be further used to extract the point cloud data of the target objects. The experimental results revealed that the average accuracy of target segmentation was 84.78%, which was close to that of PointNet++. Compared with three traditional segmentation algorithms, the accuracy was improved by nearly 23.97%. The running time of our algorithm is reduced by 95.76% compared to the PointNet++ algorithm, which has the longest running time; and by 15.65% compared to the LCCP algorithm, which has the shortest running time among traditional methods. Compared with PointNet++, the segmentation accuracy was improved. This method addressed the issues of low robustness and excessive reliance on manual feature extraction in traditional point cloud segmentation methods, providing valuable support and reference for the accurate segmentation of 3D point clouds.

Exploring Diversity-Based Active Learning for 3D Object Detection in Autonomous Driving

Exploring Diversity-Based Active Learning for 3D Object Detection in Autonomous Driving

DeployFusion: A Deployable Monocular 3D Object Detection with Multi-Sensor Information Fusion in BEV for Edge Devices.

To address the challenges of suboptimal remote detection and significant computational burden in existing multi-sensor information fusion 3D object detection methods, a novel approach based on Bird's-Eye View (BEV) is proposed. This method utilizes an enhanced lightweight EdgeNeXt feature extraction network, incorporating residual branches to address network degradation caused by the excessive depth of STDA encoding blocks. Meantime, deformable convolution is used to expand the receptive field and reduce computational complexity. The feature fusion module constructs a two-stage fusion network to optimize the fusion and alignment of multi-sensor features. This network aligns image features to supplement environmental information with point cloud features, thereby obtaining the final BEV features. Additionally, a Transformer decoder that emphasizes global spatial cues is employed to process the BEV feature sequence, enabling precise detection of distant small objects. Experimental results demonstrate that this method surpasses the baseline network, with improvements of 4.5% in the NuScenes detection score and 5.5% in average precision for detection objects. Finally, the model is converted and accelerated using TensorRT tools for deployment on mobile devices, achieving an inference time of 138 ms per frame on the Jetson Orin NX embedded platform, thus enabling real-time 3D object detection.

CWGA-Net: Center-Weighted Graph Attention Network for 3D object detection from point clouds

The precision of 3D object detection from unevenly distributed outdoor point clouds is critical in autonomous driving perception systems. Current point-based detectors employ self-attention and graph convolution to establish contextual relationships between point clouds; however, they often introduce weakly correlated redundant information, leading to blurred geometric details and false detections. To address this issue, a novel Center-weighted Graph Attention Network (CWGA-Net) has been proposed to fuse geometric and semantic similarities for weighting cross-attention scores, thereby capturing precise fine-grained geometric features. CWGA-Net initially constructs and encodes local graphs between foreground points, establishing connections between point clouds from geometric and semantic dimensions. Subsequently, center-weighted cross-attention is utilized to compute the contextual relationships between vertices within the graph, and geometric and semantic similarities between vertices are fused to weight attention scores, thereby extracting strongly related geometric shape features. Finally, a cross-feature fusion Module is introduced to deeply fuse high and low-resolution features to compensate for the information loss during downsampling. Experiments conducted on the KITTI and Waymo datasets demonstrate that the network achieves superior detection capabilities, outperforming state-of-the-art point-based single-stage methods in terms of average precision metrics while maintaining good speed.

PE-MCAT: Leveraging Image Sensor Fusion and Adaptive Thresholds for Semi-Supervised 3D Object Detection.

Existing 3D object detection frameworks in sensor-based applications heavily rely on large-scale annotated data to achieve optimal performance. However, obtaining such annotations from sensor data-like LiDAR or image sensors-is both time-consuming and costly. Semi-supervised learning offers an efficient solution to this challenge and holds significant potential for sensor-driven artificial intelligence (AI) applications. While it reduces the need for labeled data, semi-supervised learning still depends on a small amount of labeled samples for training. In the initial stages, relying on such limited samples can adversely affect the effective training of student-teacher networks. In this paper, we propose PE-MCAT, a semi-supervised 3D object detection method that generates high-precision pseudo-labels. First, to address the challenges of insufficient local feature capture and poor robustness in point cloud data, we introduce a point enrichment module. This module incorporates information from image sensors and combines multiple feature fusion methods of local and self-features to directly enhance the quality of point clouds and pseudo-labels, compensating for the limitations posed by using only a few labeled samples. Second, we explore the relationship between the teacher network and the pseudo-labels it generates. We propose a multi-class adaptive threshold strategy to initially filter and create a high-quality pseudo-label set. Furthermore, a joint variable threshold strategy is introduced to refine this set further, enhancing the selection of superior pseudo-labels.Extensive experiments demonstrate that PE-MCAT consistently outperforms recent state-of-the-art methods across different datasets. Specifically, on the KITTI dataset and using only 2% of labeled samples, our method improved the mean Average Precision (mAP) by 0.7% for cars, 3.7% for pedestrians, and 3.0% for cyclists.

MSSA: Multi-Representation Semantics-Augmented Set Abstraction for 3D Object Detection

Accurate recognition and localization of 3D objects is a fundamental research problem in 3D computer vision. Benefiting from transformation-free point cloud processing and flexible receptive fields, point-based methods have become accurate in 3D point cloud modeling, but still fall behind voxel-based competitors in 3D detection. We observe that the set abstraction module, commonly utilized by point-based methods for downsampling points, tends to retain excessive irrelevant background information, thus hindering the effective learning of features for object detection tasks. To address this issue, we propose MSSA, a Multi-representation Semantics-augmented Set Abstraction for 3D object detection. Specifically, we first design a backbone network to encode different representation features of point clouds, which extracts point-wise features through PointNet to preserve fine-grained geometric structure features, and adopts VoxelNet to extract voxel features and BEV features to enhance the semantic features of key points. Second, to efficiently fuse different representation features of keypoints, we propose a Point feature-guided Voxel feature and BEV feature fusion (PVB-Fusion) module to adaptively fuse multi-representation features and remove noise. At last, a novel Multi-representation Semantic-guided Farthest Point Sampling (MS-FPS) algorithm is designed to help set abstraction modules progressively downsample point clouds, thereby improving instance recall and detection performance with more important foreground points. We evaluate MSSA on the widely used KITTI dataset and the more challenging nuScenes dataset. Experimental results show that compared to PointRCNN, our method improves the AP of “moderate” level for three classes of objects by 7.02%, 6.76%, and 5.44%, respectively. Compared to the advanced point-voxel-based method PV-RCNN, our method improves the AP of “moderate” level by 1.23%, 2.84%, and 0.55% for the three classes, respectively.

MonoCAPE: Monocular 3D object detection with coordinate-aware position embeddings

3D monocular detection remains to be a focal point of research, particularly due to its capacity to deliver available precision under conditions of low cost and simplified configurations, making it especially valuable in fields like autonomous driving. Current 3D object detection methods often overlook the spatial information missing from images, which is critical to spatial perception, and optimize bounding box attributes separately, failing to meet the requirements of autonomous driving. We introduce MonoCAPE, a novel 3D detection framework addressing these issues by encoding spatial information and co-optimizing attributes through a Coordinate-Aware Position Encoding (CAPE) Generator and a Task Co-optimization Strategy (TCS). The CAPE Generator produces sparse positional embeddings, enabling spatial awareness with low computational cost, while the TCS utilizes Gaussian modeling to prevent suboptimal outputs. In this way, our framework comprehensively takes into account what existing approaches ignore. Extensive experiments on the KITTI dataset demonstrate MonoCAPE significantly improves AP3D and APBEV metrics compared to existing advanced methods.