3D Object Detection Research Articles

Semi-Supervised Online Continual Learning for 3D Object Detection in Mobile Robotics

Continual learning addresses the challenge of acquiring and retaining knowledge over time across multiple tasks and environments. Previous research primarily focuses on offline settings where models learn through increasing tasks from samples paired with ground truth annotations. In this work, we focus on an unsolved, challenging, yet practical scenario, specifically, the semi-supervised online continual learning in autonomous driving and mobile robotics. In our settings, models are tasked with learning new distributions from streaming unlabeled samples and performing 3D object detection as soon as the LiDAR point cloud arrives. Additionally, we conducted experiments on both the KITTI dataset, our newly built IUSL dataset and Canadian Adverse Driving Conditions (CADC) Dataset. The results indicate that our method achieves a balance between rapid adaptation and knowledge retention, showcasing its effectiveness in the dynamic and complex environment of autonomous driving and mobile robotics. The developed ROS packages and IUSL dataset will be publicly available at: https://github.com/npu-ius-lab/OCL3D.

Multi-feature enhancement based on sparse networks for single-stage 3D object detection

In the field of autonomous driving, the accuracy and real-time requirements for 3D object detection technology continue to improve, which is directly related to the commercialization process and market popularity of autonomous vehicles. Despite the efficiency of pillar-based coding for onboard systems, it falls short in terms of accuracy and the reduction of incorrect positives. In this paper, we will examine how to solve the problem of high incorrect rate and low accuracy of existing methods. Firstly, a MAP coding module is introduced to optimize previous point cloud feature coding modules, allowing for the efficient extraction of fine-grained features from point cloud data. Then, we introduce an innovative sparse dual attention (SDA) to efficiently filter out irrelevant details in feature extraction, thereby improving the pertinence and efficiency of information extraction. Finally, to address the potential loss of information from single local feature extraction, a local and global fusion module (CTGC) is introduced. Our method has proactively demonstrated its efficiency and accuracy through rigorous experimentation across diverse datasets. Analysis of the results leads to the conclusion that our solutions provide accurate and robust detection results. Code will be available at https://github.com/lcy199905/MyOpenPCDet.git.

Three-Dimensional Outdoor Object Detection in Quadrupedal Robots for Surveillance Navigations

Quadrupedal robots are confronted with the intricate challenge of navigating dynamic environments fraught with diverse and unpredictable scenarios. Effectively identifying and responding to obstacles is paramount for ensuring safe and reliable navigation. This paper introduces a pioneering method for 3D object detection, termed viewpoint feature histograms, which leverages the established paradigm of 2D detection in projection. By translating 2D bounding boxes into 3D object proposals, this approach not only enables the reuse of existing 2D detectors but also significantly increases the performance with less computation required, allowing for real-time detection. Our method is versatile, targeting both bird’s eye view objects (e.g., cars) and frontal view objects (e.g., pedestrians), accommodating various types of 2D object detectors. We showcase the efficacy of our approach through the integration of YOLO3D, utilizing LiDAR point clouds on the KITTI dataset, to achieve real-time efficiency aligned with the demands of autonomous vehicle navigation. Our model selection process, tailored to the specific needs of quadrupedal robots, emphasizes considerations such as model complexity, inference speed, and customization flexibility, achieving an accuracy of up to 99.93%. This research represents a significant advancement in enabling quadrupedal robots to navigate complex and dynamic environments with heightened precision and safety.

AMVFNet: Attentive Multi-View Fusion Network for 3D Object Detection

Pillar-based method is significant in the field of LiDAR-based 3D object detection which could directly make use of efficient 2D backbone and save computational resources during reference. Existing methods usually sequentially project the original point clouds into the cylindrical view or the bird-eye view for feature extraction. However, the former suffers from obscured problems and the scales of instances vary greatly with distance, while the latter leads to considerable confusion problems due to the loss of semantic information caused by the sparsity of the projected point cloud. In this paper, we present a novel and efficient two-stage point-pillar hybrid architecture named Attentive Multi-View Fusion Network (AMVFNet), in which we abstract features from all cylindrical view, bird-eye view, and raw point clouds. Rather than designing more complex modules to solve the problems inherent in the single-view approach, our multi-view fusion architecture effectively combines the strengths of multiple perspectives to improve performance at a more fundamental level. Besides, to compensate for quantization distortion caused by projection operations, we propose attentive feature enhancement layers to further improve the capability of contextual information capturing. Extensive experiments on the KITTI detection benchmark illustrate that our proposed AMVFNet achieves competitive performance compared with other SOTA 3D object detectors.

Rethinking the Non-Maximum Suppression Step in 3D Object Detection from a Bird’s-Eye View

In camera-based bird’s-eye view (BEV) 3D object detection, non-maximum suppression (NMS) plays a crucial role. However, traditional NMS methods become ineffective in BEV scenarios where the predicted bounding boxes of small object instances often have no overlapping areas. To address this issue, this paper proposes a BEV intersection over union (IoU) computation method based on relative position and absolute spatial information, referred to as B-IoU. Additionally, a BEV circular search method, called B-Grouping, is introduced to handle prediction boxes of varying scales. Utilizing these two methods, a novel NMS strategy called BEV-NMS is developed to handle the complex prediction boxes in BEV perspectives. This BEV-NMS strategy is implemented in several existing algorithms. Based on the results from the nuScenes validation set, there was an average increase of 7.9% in mAP when compared to the strategy without NMS. The NDS also showed an average increase of 7.9% under the same comparison. Furthermore, compared to the Scale-NMS strategy, the mAP increased by an average of 3.4%, and the NDS saw an average improvement of 3.1%.

Dense projection fusion for 3D object detection

Fusing information from LiDAR and cameras can effectively enhance the overall perceptivity of autonomous vehicles in various scenarios. Despite the relatively good results achieved by point-wise fusion and Bird’s-Eye-View (BEV) fusion, they still cannot fully leverage the image information and lack of effective depth information. For any fusion methods, the multi-modal features first need to be concatenated along the channel, and then the fused features are extracted using convolutional layers. However, this type of fusion methods is effective, but too coarse which causes that the fused features cannot pay more attention to the regions with important features and suffer from severe noise. To tackle these issues, we propose in this paper a Dense Projection Fusion (DPFusion) approach. It consists of two new modules: dense depth map guided BEV transform (DGBT) module and multi-modal feature adaptive fusion (MFAF) module. The DGBT module first quickly estimates the depth of each pixel and then projects all image features to the BEV space, making full use of the image information. The MFAF module computes the image weights and point cloud weights for each channel in each BEV grid and then adaptively weights and fuses the image BEV features with the point cloud BEV features. It is worth pointing out that the MFAF module makes the fused features pay more attention to background outline and object outline. Our proposed DPFusion demonstrates competitive results in 3D object detection, achieving a mean Average Precision (mAP) of 70.4 and a nuScenes detection score (NDS) of 72.3 on the nuScenes validation set.

Multi-Scale Candidate Fusion and Optimization-Based 3D Object Detection Algorithm

To address the issues of target omission and the inclusion of a large number of background points in keypoint sampling for point cloud-based object detection, an improved algorithm based on the PV-RCNN network is introduced. This approach employs both a regional proposal fusion network and weighted Non-Maximum Suppression (NMS) to merge proposals generated at various scales while eliminating redundancy. A segmentation network is utilized to segment foreground points from the original point cloud, and object center points are identified based on these proposals. Gaussian density functions are employed for regional density estimation, which assigns different sampling weights to solve the problem of difficult sampling in sparse areas. Experimental evaluations on the KITTI dataset indicate that the algorithm enhances the average precision at medium difficulty levels by 0.39%, 1.31%, and 0.63%for cars, pedestrians, and cyclists, respectively. Generalization experiments were also conducted on the Waymo dataset. The results suggest that the introduced algorithm achieves higher accuracy compared to most of the existing 3D object detection networks.

EFMF-pillars: 3D object detection based on enhanced features and multi-scale fusion

As unmanned vehicle technology advances rapidly, obstacle recognition and target detection are crucial links, which directly affect the driving safety and efficiency of unmanned vehicles. In response to the inaccurate localization of small targets such as pedestrians in current object detection tasks and the problem of losing local features in the PointPillars, this paper proposes a three-dimensional object detection method based on improved PointPillars. Firstly, addressing the issue of lost spatial and local information in the PointPillars, the feature encoding part of the PointPillars is improved, and a new pillar feature enhancement extraction module, CSM-Module, is proposed. Channel encoding and spatial encoding are introduced in the new pillar feature enhancement extraction module, fully considering the spatial information and local detailed geometric information of each pillar, thereby enhancing the feature representation capability of each pillar. Secondly, based on the fusion of CSPDarknet and SENet, a new backbone network CSE-Net is designed in this paper, enabling the extraction of rich contextual semantic information and multi-scale global features, thereby enhancing the feature extraction capability. Our method achieves higher detection accuracy when validated on the KITTI dataset. Compared to the original network, the improved algorithm’s average detection accuracy is increased by 3.42%, it shows that the method is reasonable and valuable.

Bi-Interfusion: A bidirectional cross-fusion framework with semantic-guided transformers in LiDAR-camera fusion

Multi-sensor modal fusion has shown significant advantages in 3D object detection tasks. However, existing methods for fusing multi-modal features into the bird’s eye view (BEV) space often encounter challenges such as feature misalignment, underutilization of semantic information, and inaccurate depth estimation on the Z-axis, resulting in suboptimal performance. To address these issues, we propose Bi-Interfusion, a novel multi-modal fusion framework based on transformers. Bi-Interfusion incorporates a bidirectional fusion architecture, including components such as Pixel-wise Semantic Painting, Gaussian Depth Prior Distribution module, and Semantic Guidance Align module, to overcome the limitations of traditional fusion methods. Specifically, Bi-Interfusion employs a bidirectional cross-fusion strategy to merge image and LiDAR features, enabling the generation of multi-sensor BEV features. This approach leverages a refined Gaussian Depth Prior Distribution generated from LiDAR points, thereby improving the precision of view transformation. Additionally, we apply a pixel-wise semantic painting technique to embed image semantic information into LiDAR point clouds, facilitating a more comprehensive scene understanding. Furthermore, a transformer-based model is utilized to establish soft correspondences among multi-sensor BEV features, capturing positional dependencies and fully exploiting semantic information for alignment. Through experiments on nuScenes benchmark dataset, Bi-Interfusion demonstrates notable performance improvements, achieving a competitive performance of 72.6% mAP and 75.4% NDS in the 3D object detection task.

KPTr: Key point transformer for LiDAR-based 3D object detection

Accurate 3D environmental perception is crucial for ensuring the safety of autonomous vehicles. However, many existing point-based 3D object detection methods rely only on independent point features to extract spatial information and ignore the dependencies between points. Meanwhile, these methods usually use max-pooling operations to aggregate neighboring point features without considering the stability of local geometric information. To address these issues, this paper proposes a novel two-stage LiDAR-based key point transformer that consists of a region proposal network module and a region-based transformer module to enhance 3D detection performance. Firstly, unprocessed raw point clouds are taken as the input of the model to preserve detailed spatial structural information. Secondly, in the region proposal network an enhanced PointNet++ is used to segment foreground points and generate high-quality first-stage proposals. Then, the representation of proposals is improved by the region-based transformer module, which constructs long-range key point cloud relationship features within the regions of interest and diffuses spatial information into the feature channel layers. Additionally experiments on the KITTI dataset show that detection performance of the proposed method surpasses most mainstream 3D object detection methods and achieves a detection speed of 14.28 Hz, striking a good balance between detection accuracy and speed. Finally, the proposed method is deployed on real vehicle hardware platforms for online detection, thereby proving its industrial value.

Local-to-Global Semantic Learning for Multi-View 3D Object Detection From Point Cloud

Local-to-Global Semantic Learning for Multi-View 3D Object Detection From Point Cloud

Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

Point-Based Fusion for Multimodal 3D Detection in Autonomous Driving

In the broader field of mechanical technology, and specifically within the domain of self-driving vehicles, cameras and LIDAR are crucial sensor modalities that provide complementary information, offering significant potential for sensor fusion. However, directly merging multi-sensor data through point projection can lead to information loss due to quantization, and managing the differences between data formats from multiple sensors remains a challenge. To address these issues, we propose an new fusion method that leverages continuous convolution, point-pooling, and a learned MLP to achieve superior detection performance. Our approach integrates the segmentation mask with raw LIDAR points instead of using projected points, thereby avoiding quantization loss. We conduct neighbor searches on the points and retrieve corresponding semantic features from images to concatenate image and LIDAR data. Subsequently, we apply continuous convolution, point-pooling, and a learned MLP to obtain the fused output. The pooling and aggregation operations, as extensions of convolution, are specifically designed to handle the disparities in data formats. Our detection network is divided into two stages: in the first stage, preliminary proposals and segmentation features are generated; in the second stage, the fusion result with the segmentation mask is refined to produce the final prediction. Our method aims to achieve precise object detection in 3D environments by enhancing LIDAR point data with semantic features from images, allowing for the flexibility to alternate segmentation sub-algorithms as needed. Extensive experiments on the KITTI dataset demonstrate the effectiveness of our approach, which achieves high precision and robust performance in 3D object detection tasks.

3D Object Detection via Residual SqueezeDet

Three-dimensional object detection is a critical task in computer vision with applications in autonomous driving, robotics, and augmented reality. This paper introduces Residual SqueezeDet, a novel network architecture that enhances the performance of 3D object detection on the KITTI dataset. Building upon the efficient SqueezeDet framework, we propose the Residual Fire module, which incorporates skip connections inspired by ResNet architectures into the original Fire module. This innovation improves gradient flow, enhances feature propagation, and allows for more effective training of deeper networks. Our method leverages point cloud and image-based features, employing a Residual SqueezeDet to effectively capture local and global context. Extensive experiments on the KITTI dataset demonstrate that Residual SqueezeDet significantly outperforms the original SqueezeDet, with particularly notable improvements in challenging scenarios. The proposed model maintains computational efficiency while achieving state-of-the-art performance, making it well-suited for real-time applications in autonomous driving. Our work contributes to the field by providing a more accurate and robust solution for 3D object detection, paving the way for improved perception systems in dynamic environments.

AEPF: Attention-Enabled Point Fusion for 3D Object Detection.

Current state-of-the-art (SOTA) LiDAR-only detectors perform well for 3D object detection tasks, but point cloud data are typically sparse and lacks semantic information. Detailed semantic information obtained from camera images can be added with existing LiDAR-based detectors to create a robust 3D detection pipeline. With two different data types, a major challenge in developing multi-modal sensor fusion networks is to achieve effective data fusion while managing computational resources. With separate 2D and 3D feature extraction backbones, feature fusion can become more challenging as these modes generate different gradients, leading to gradient conflicts and suboptimal convergence during network optimization. To this end, we propose a 3D object detection method, Attention-Enabled Point Fusion (AEPF). AEPF uses images and voxelized point cloud data as inputs and estimates the 3D bounding boxes of object locations as outputs. An attention mechanism is introduced to an existing feature fusion strategy to improve 3D detection accuracy and two variants are proposed. These two variants, AEPF-Small and AEPF-Large, address different needs. AEPF-Small, with a lightweight attention module and fewer parameters, offers fast inference. AEPF-Large, with a more complex attention module and increased parameters, provides higher accuracy than baseline models. Experimental results on the KITTI validation set show that AEPF-Small maintains SOTA 3D detection accuracy while inferencing at higher speeds. AEPF-Large achieves mean average precision scores of 91.13, 79.06, and 76.15 for the car class's easy, medium, and hard targets, respectively, in the KITTI validation set. Results from ablation experiments are also presented to support the choice of model architecture.

High‐order multilayer attention fusion network for3Dobject detection

AbstractThree‐dimensional object detection based on the fusion of 2D image data and 3D point clouds has become a research hotspot in the field of 3D scene understanding. However, different sensor data have discrepancies in spatial position, scale, and alignment, which severely impact detection performance. Inappropriate fusion methods can lead to the loss and interference of valuable information. Therefore, we propose the High‐Order Multi‐Level Attention Fusion Network (HMAF‐Net), which takes camera images and voxelized point clouds as inputs for 3D object detection. To enhance the expressive power between different modality features, we introduce a high‐order feature fusion module that performs multi‐level convolution operations on the element‐wise summed features. By incorporating filtering and non‐linear activation, we extract deep semantic information from the fused multi‐modal features. To maximize the effectiveness of the fused salient feature information, we introduce an attention mechanism that dynamically evaluates the importance of pooled features at each level, enabling adaptive weighted fusion of significant and secondary features. To validate the effectiveness of HMAF‐Net, we conduct experiments on the KITTI dataset. In the “Car,” “Pedestrian,” and “Cyclist” categories, HMAF‐Net achieves mAP performances of 81.78%, 60.09%, and 63.91%, respectively, demonstrating more stable performance compared to other multi‐modal methods. Furthermore, we further evaluate the framework's effectiveness and generalization capability through the KITTI benchmark test, and compare its performance with other published detection methods on the 3D detection benchmark and BEV detection benchmark for the “Car” category, showing excellent results. The code and model will be made available onhttps://github.com/baowenzhang/High‐order‐Multilayer‐Attention‐Fusion‐Network‐for‐3D‐Object‐Detection.

VIDF-Net: A Voxel-Image Dynamic Fusion method for 3D object detection

In recent years, multi-modal fusion methods have shown excellent performance in the field of 3D object detection, which select the voxel centers and globally fuse with image features across the scene. However, these approaches exist two issues. First, The distribution of voxel density is highly heterogeneous due to the discrete volumes. Additionally, there are significant differences in the features between images and point clouds. Global fusion does not take into account the correspondence between these two modalities, which leads to the insufficient fusion. In this paper, we propose a new multi-modal fusion method named Voxel-Image Dynamic Fusion (VIDF). Specifically, VIDF-Net is composed of the Voxel Centroid Mapping module (VCM) and the Deformable Attention Fusion module (DAF). The Voxel Centroid Mapping module is used to calculate the centroid of voxel features and map them onto the image plane, which can locate the position of voxel features more effectively. We then use the Deformable Attention Fusion module to dynamically calculates the offset of each voxel centroid from the image position and combine these two modalities. Furthermore, we propose Region Proposal Network with Channel-Spatial Aggregate to combine channel and spatial attention maps for improved multi-scale feature interaction. We conduct extensive experiments on the KITTI dataset to demonstrate the outstanding performance of proposed VIDF network. In particular, significant improvements have been observed in the Hard categories of Cars and Pedestrians, which shows the significant effectiveness of our approach in dealing with complex scenarios.

Spatial awareness enhancement based single-stage anchor-free 3D object detection for autonomous driving

The real-time and accurate detection of three-dimensional (3D) objects based on LiDAR is a focal problem in the field of autonomous driving environment perception. Compared to two-stage and anchor-based 3D object detection methods that suffer from inference latency challenges, single-stage anchor-free 3D object detection approaches are more suitable for deployment in autonomous driving vehicles with the strict real-time requirement. However, they face the issue of insufficient spatial awareness, which can result in detection errors such as false positives and false negatives, thereby increasing the potential risks of autonomous driving. In response to this, we focus on enhancing the spatial awareness of CenterPoint, a widely used single-stage anchor-free 3D object detector in the industry. Considering the limited allocation of computational resources and the performance bottleneck caused by pillar encoder, we propose an efficient SSDCM backbone to strengthen feature representation and extraction. Furthermore, a simple BGC neck is devised to weight and exchange contextual information in order to deeply fuse multi-scale features. Combining improved backbone and neck networks, we construct a single-stage anchor-free 3D object detection model with spatial awareness enhancement, named CenterPoint-Spatial Awareness Enhancement (CenterPoint-SAE). We evaluate CenterPoint-SAE on two large-scale and challenging autonomous driving datasets, nuScenes and Waymo. It achieves 53.3% mAP and 62.5% NDS on nuScenes detection benchmark, and runs inference at a speed of 11.1 FPS. Compared to the baseline, the upgraded networks deliver a performance improvement of 1.6% mAP and 1.2% NDS at minor cost. Notably, on Waymo dataset, our method achieves competitive detection performance compared to two-stage and point-based methods.

PesRec: A parametric estimation method for indoor semantic scene reconstruction from a single image

Reconstructing semantic indoor scenes is a challenging task in augmented and virtual reality. The quality of scene reconstruction is limited by the complexity and occlusion of indoor scenes. This is due to the difficulty in estimating the spatial structure of the scene and insufficient learning for object location inference. To address these challenges, we have developed PesRec, an end-to-end multi-task scene reconstruction network for parameterizing indoor semantic information. PesRec incorporates a newly designed spatial layout estimator and a 3D object detector to effectively learn scene parameter features from RGB images. We modify an object mesh generator to enhance the robustness of reconstructing indoor occluded objects through point cloud optimization in PesRec. Using the analyzed scene parameters and spatial structure, the proposed PesRec reconstructs an indoor scene by placing object meshes scaled to 3D detection boxes in an estimated layout cuboid. Extensive experiments on two benchmark datasets demonstrate that PesRec performs exceptionally well for object reconstruction with an average chamfer distance of 5.24 × 10-3 on the Pix3D dataset including 53.61 % mAP for 3D object detection and 79.7 % 3D IoU for the estimation of layout on the commonly-used SUN RGB-D datasets. The proposed computing network breaks through the limitations caused by complex indoor scenes and occlusions, showing optimization results that improve the quality of reconstruction in the fields of augmented reality and virtual reality.

VFL3D: A Single-Stage Fine-Grained Lightweight Point Cloud 3D Object Detection Algorithm Based on Voxels

VFL3D: A Single-Stage Fine-Grained Lightweight Point Cloud 3D Object Detection Algorithm Based on Voxels