Three-dimensional Object Detection Research Articles

Vehicle Behavior Discovery and Three-Dimensional Object Detection and Tracking Based on Spatio-Temporal Dependency Knowledge and Artificial Fish Swarm Algorithm

In complex traffic environments, 3D target tracking and detection are often occluded by various stationary and moving objects. When the target is occluded, its apparent characteristics change, resulting in a decrease in the accuracy of tracking and detection. In order to solve this problem, we propose to learn the vehicle behavior from the driving data, predict and calibrate the vehicle trajectory, and finally use the artificial fish swarm algorithm to optimize the tracking results. The experiments show that compared with the CenterTrack method, the proposed method improves the key indicators of MOTA (Multi-Object Tracking Accuracy) in 3D object detection and tracking on the nuScenes dataset, and the frame rate is 26 fps.

SOA: Seed point offset attention for indoor 3D object detection in point clouds

Three-dimensional object detection plays a pivotal role in scene understanding and holds significant importance in various indoor perception applications. Traditional methods based on Hough voting are susceptible to interference from background points or neighboring objects when casting votes for the target’s center from each seed point. Moreover, fixed-size set abstraction modules may result in the loss of structural information for large objects. To address these challenges, this paper proposes a three-dimensional object detection model based on seed point offset attention. The objective of this model is to enhance the model’s resilience to voting noise interference and alleviate feature loss for large-scale objects. Specifically, a seed point offset tensor is first defined, and then the offset tensor self-attention network is employed to learn the weights between votes, thereby establishing a correlation between the voting semantic features and the object structural information. Furthermore, an object surface perception module is introduced, which incorporates detailed features of local object surfaces into global feature representations through vote backtracking and surface mapping. Experimental results indicate that the model achieved excellent performance on the ScanNet-V2 (mAP@0.5, 60.3%) and SUN RGB-D (mAP@0.5, 64.0%) datasets, respectively improving by 2.6% (mAP@0.5) and 5.4% (mAP@0.5) compared to VoteNet.

A Multi-Sensor 3D Detection Method for Small Objects

In response to the limited accuracy of current three-dimensional (3D) object detection algorithms for small objects, this paper presents a multi-sensor 3D small object detection method based on LiDAR and a camera. Firstly, the LiDAR point cloud is projected onto the image plane to obtain a depth image. Subsequently, we propose a cascaded image fusion module comprising multi-level pooling layers and multi-level convolution layers. This module extracts features from both the camera image and the depth image, addressing the issue of insufficient depth information in the image feature. Considering the non-uniform distribution characteristics of the LiDAR point cloud, we introduce a multi-scale voxel fusion module composed of three sets of VFE (voxel feature encoder) layers. This module partitions the point cloud into grids of different sizes to improve detection ability for small objects. Finally, the multi-level fused point features are associated with the corresponding scale’s initial voxel features to obtain the fused multi-scale voxel features, and the final detection results are obtained based on this feature. To evaluate the effectiveness of this method, experiments are conducted on the KITTI dataset, achieving a 3D AP (average precision) of 73.81% for the hard level of cars and 48.03% for the hard level of persons. The experimental results demonstrate that this method can effectively achieve 3D detection of small objects.

NeXtFusion: Attention-Based Camera-Radar Fusion Network for Improved Three-Dimensional Object Detection and Tracking

Accurate perception is crucial for autonomous vehicles (AVs) to navigate safely, especially in adverse weather and lighting conditions where single-sensor networks (e.g., cameras or radar) struggle with reduced maneuverability and unrecognizable targets. Deep Camera-Radar fusion neural networks offer a promising solution for reliable AV perception under any weather and lighting conditions. Cameras provide rich semantic information, while radars act like an X-ray vision, piercing through fog and darkness. This work proposes a novel, efficient Camera-Radar fusion network called NeXtFusion for robust AV perception with an improvement in object detection accuracy and tracking. Our proposed approach of utilizing an attention module enhances crucial feature representation for object detection while minimizing information loss from multi-modal data. Extensive experiments on the challenging nuScenes dataset demonstrate NeXtFusion’s superior performance in detecting small and distant objects compared to other methods. Notably, NeXtFusion achieves the highest mAP score (0.473) on the nuScenes validation set, outperforming competitors like OFT (35.1% improvement) and MonoDIS (9.5% improvement). Additionally, NeXtFusion demonstrates strong performance in other metrics like mATE (0.449) and mAOE (0.534), highlighting its overall effectiveness in 3D object detection. Furthermore, visualizations of nuScenes data processed by NeXtFusion further demonstrate its capability to handle diverse real-world scenarios. These results suggest that NeXtFusion is a promising deep fusion network for improving AV perception and safety for autonomous driving.

Edge-Triggered Three-Dimensional Object Detection Using a LiDAR Ring.

Autonomous driving recognition technology that can quickly and accurately recognize even small objects must be developed in high-speed situations. This study proposes an object point extraction method using rule-based LiDAR ring data and edge triggers to increase both speed and performance. The LiDAR's ring information is interpreted as a digital pulse to remove the ground, and object points are extracted by detecting discontinuous edges of the z value aligned with the ring ID and azimuth. A bounding box was simply created using DBSCAN and PCA to check recognition performance from the extracted object points. Verification of the results of removing the ground and extracting points through Ring Edge was conducted using SemanticKITTI and Waymo Open Dataset, and it was confirmed that both F1 scores were superior to RANSAC. In addition, extracting bounding boxes of objects also showed higher PDR index performance when verified in open datasets, virtual driving environments, and actual driving environments.

Equal Emphasis on Data and Network: A Two-Stage 3D Point Cloud Object Detection Algorithm with Feature Alignment

Three-dimensional object detection is a pivotal research topic in computer vision, aiming to identify and locate objects in three-dimensional space. It has wide applications in various fields such as geoscience, autonomous driving, and drone navigation. The rapid development of deep learning techniques has led to significant advancements in 3D object detection. However, with the increasing complexity of applications, 3D object detection faces a series of challenges such as data imbalance and the effectiveness of network models. Specifically, in an experiment, our investigation revealed a notable discrepancy in the LiDAR reflection intensity within a point cloud scene, with stronger intensities observed in proximity and weaker intensities observed at a distance. Furthermore, we have also noted a substantial disparity in the number of foreground points compared to the number of background points. Especially in 3D object detection, the foreground point is more important than the background point, but it is usually downsampled without discrimination in the subsequent processing. With the objective of tackling these challenges, we work from both data and network perspectives, designing a feature alignment filtering algorithm and a two-stage 3D object detection network. Firstly, in order to achieve feature alignment, we introduce a correction equation to decouple the relationship between distance and intensity and eliminate the attenuation effect of intensity caused by distance. Then, a background point filtering algorithm is designed by using the aligned data to alleviate the problem of data imbalance. At the same time, we take into consideration the fact that the accuracy of semantic segmentation plays a crucial role in 3D object detection. Therefore, we propose a two-stage deep learning network that integrates spatial and spectral information, in which a feature fusion branch is designed and embedded in the semantic segmentation backbone. Through a series of experiments on the KITTI dataset, it is proven that the proposed method achieves the following average precision (AP_R40) values for easy, moderate, and hard difficulties, respectively: car (Iou 0.7)—89.23%, 80.14%, and 77.89%; pedestrian (Iou 0.5)—52.32%, 45.47%, and 38.78%; and cyclist (Iou 0.5)—76.41%, 61.92%, and 56.39%. By emphasizing both data quality optimization and efficient network architecture, the performance of the proposed method is made comparable to other state-of-the-art methods.

Singular and Multimodal Techniques of 3D Object Detection: Constraints, Advancements and Research Direction

Two-dimensional object detection techniques can detect multiscale objects in images. However, they lack depth information. Three-dimensional object detection provides the location of the object in the image along with depth information. To provide depth information, 3D object detection involves the application of depth-perceiving sensors such as LiDAR, stereo cameras, RGB-D, RADAR, etc. The existing review articles on 3D object detection techniques are found to be focusing on either a singular modality (e.g., only LiDAR point cloud-based) or a singular application field (e.g., autonomous vehicle navigation). However, to the best of our knowledge, there is no review paper that discusses the applicability of 3D object detection techniques in other fields such as agriculture, robot vision or human activity detection. This study analyzes both singular and multimodal techniques of 3D object detection techniques applied in different fields. A critical analysis comprising strengths and weaknesses of the 3D object detection techniques is presented. The aim of this study is to facilitate future researchers and practitioners to provide a holistic view of 3D object detection techniques. The critical analysis of the singular and multimodal techniques is expected to help the practitioners find the appropriate techniques based on their requirement.

Addressing the Gaps of IoU Loss in 3D Object Detection with IIoU

Three-dimensional object detection involves estimating the dimensions, orientations, and locations of 3D bounding boxes. Intersection of Union (IoU) loss measures the overlap between predicted 3D box and ground truth 3D bounding boxes. The localization task uses smooth-L1 loss with IoU to estimate the object’s location, and the classification task identifies the object/class category inside each 3D bounding box. Localization suffers a performance gap in cases where the predicted and ground truth boxes overlap significantly less or do not overlap, indicating the boxes are far away, and in scenarios where the boxes are inclusive. Existing axis-aligned IoU losses suffer performance drop in cases of rotated 3D bounding boxes. This research addresses the shortcomings in bounding box regression problems of 3D object detection by introducing an Improved Intersection Over Union (IIoU) loss. The proposed loss function’s performance is experimented on LiDAR-based and Camera-LiDAR-based fusion methods using the KITTI dataset.

A LiDAR Multi-Object Detection Algorithm for Autonomous Driving

Three-dimensional object detection is the core of an autonomous driving perception system, which detects and analyzes targets around the vehicle to obtain their sizes, shapes, and categories to provide reliable operational decisions for achieving autonomous driving. To improve the detection and localization accuracy of multi-object targets such as surrounding vehicles and pedestrians in autonomous driving scenarios, based on PointPillars fast object detection network, a three-dimensional object detection algorithm based on the channel attention mechanism, ECA Modules-PointPillars, is proposed. Firstly, the improved algorithm uses point cloud columnarization features to convert a three-dimensional point cloud into a two-dimensional pseudo-image. Then, combining the 2D backbone network for feature extraction with the Efficient Channel Attention (ECA) modules to achieve the enhancement of the positional feature information in the pseudo-image and the weakening of the irrelevant feature information such as background noise. Finally, the single-shot multibox detector (SSD) algorithm was used to complete the 3D object detection task. The experimental results show that the improved algorithm improves the mAP by 3.84% and 4.04% in BEV mode and 3D mode, respectively, compared to PointPillars, which improves the mAP by 4.64% and 5.89% in BEV mode and 3D mode, respectively, compared to F-PointNet, improves the mAP by 11.78% and 14.19% in BEV mode and 3D mode, respectively, compared to VoxelNet, and improves the mAP by 9.47% and 6.55% in BEV mode and 3D mode, respectively, compared to SECOND, demonstrating the effectiveness and reliability of the improved algorithms in autonomous driving scenarios.

ConCs-Fusion: A Context Clustering-Based Radar and Camera Fusion for Three-Dimensional Object Detection

Multi-modality three-dimensional (3D) object detection is a crucial technology for the safe and effective operation of environment perception systems in autonomous driving. In this study, we propose a method called context clustering-based radar and camera fusion for 3D object detection (ConCs-Fusion) that combines radar and camera sensors at the intermediate fusion level to achieve 3D object detection. We extract features from heterogeneous sensors and input them as feature point sets into the fusion module. Within the fusion module, we utilize context cluster blocks to learn multi-scale features of radar point clouds and images, followed by upsampling and fusion of the feature maps. Then, we leverage a multi-layer perceptron to nonlinearly represent the fused features, reducing the feature dimensionality to improve model inference speed. Within the context cluster block, we aggregate feature points of the same object from different sensors into one cluster based on their similarity. All feature points within the same cluster are then fused into a radar–camera feature fusion point, which is self-adaptively reassigned to the originally extracted feature points from a simplex sensor. Compared to previous methods that only utilize radar as an auxiliary sensor to camera, or vice versa, the ConCs-Fusion method achieves a bidirectional cross-modal fusion between radar and camera. Finally, our extensive experiments on the nuScenes dataset demonstrate that ConCs-Fusion outperforms other methods in terms of 3D object detection performance.

Three-dimensional object detection with spatial-semantic features of point clouds

Three-dimensional object detection with spatial-semantic features of point clouds

CAF-RCNN: multimodal 3D object detection with cross-attention

ABSTRACT LiDAR and camera are pivotal sensors of 3D (three-dimensional) object detection. As a result of their different characteristics, increasingly multimodal-based object detection methods have been proposed. Now, popular methods are to hardly associate camera features with LiDAR features, but the features are frequently enhanced and aggregated, so there is a major challenge in how to align two features effectively. Therefore, we propose CAF-RCNN. On the basis of PointRCNN, using Feature Pyramid Network (FPN) to extract advanced semantic features at different scales, then fusing these features with the LiDAR features of the Set ion (SA) module output in PointRCNN and subsequent steps. Regarding the features fusion module, we design a module based on the cross-attention mechanism, CAFM (Cross-Attention Fusion Module). It combines two channel attention streams in a cross-over fashion to utilize rich details about significant objects in the Image Stream and Geometric Stream. We did a lot of experiments on the KITTI dataset, and the result shows that our method is 6.43% higher than PointRCNN in 3D accuracy.

PVONet: point-voxel-based semi-supervision monocular three-dimensional object detection using LiDAR camera systems

PVONet: point-voxel-based semi-supervision monocular three-dimensional object detection using LiDAR camera systems

DFA-SAT: Dynamic Feature Abstraction with Self-Attention-Based 3D Object Detection for Autonomous Driving

Autonomous vehicles (AVs) play a crucial role in enhancing urban mobility within the context of a smarter and more connected urban environment. Three-dimensional object detection in AVs is an essential task for comprehending the driving environment to contribute to their safe use in urban environments. Existing 3D LiDAR object detection systems lose many critical point features during the down-sampling process and neglect the crucial interactions between local features, providing insufficient semantic information and leading to subpar detection performance. We propose a dynamic feature abstraction with self-attention (DFA-SAT), which utilizes self-attention to learn semantic features with contextual information by incorporating neighboring data and focusing on vital geometric details. DFA-SAT comprises four modules: object-based down-sampling (OBDS), semantic and contextual feature extraction (SCFE), multi-level feature re-weighting (MLFR), and local and global features aggregation (LGFA). The OBDS module preserves the maximum number of semantic foreground points along with their spatial information. SCFE learns rich semantic and contextual information with respect to spatial dependencies, refining the point features. MLFR decodes all the point features using a channel-wise multi-layered transformer approach. LGFA combines local features with decoding weights for global features using matrix product keys and query embeddings to learn spatial information across each channel. Extensive experiments using the KITTI dataset demonstrate significant improvements over the mainstream methods SECOND and PointPillars, improving the mean average precision (AP) by 6.86% and 6.43%, respectively, on the KITTI test dataset. DFA-SAT yields better and more stable performance for medium and long distances with a limited impact on real-time performance and model parameters, ensuring a transformative shift akin to when automobiles replaced conventional transportation in cities.

CAMRL: A Joint Method of Channel Attention and Multidimensional Regression Loss for 3D Object Detection in Automated Vehicles

Fully automated vehicles collect information about their road environments to adjust their driving actions, such as braking and slowing down. The development of artificial intelligence (AI) and the Internet of Things (IoT) has improved the cognitive abilities of vehicles, allowing them to detect traffic signs, pedestrians, and obstacles for increasing the intelligence of these transportation systems. Three-dimensional (3D) object detection in front-view images taken by vehicle cameras is important for both object detection and depth estimation. In this paper, a joint channel attention and multidimensional regression loss method for 3D object detection in automated vehicles (called CAMRL) is proposed to improve the average precision of 3D object detection by focusing on the model’s ability to infer the locations and sizes of objects. First, channel attention is introduced to effectively learn the yaw angles from the road images captured by vehicle cameras. Second, a multidimensional regression loss algorithm is designed to further optimize the size and position parameters during the training process. Third, the intrinsic parameters of the camera and the depth estimate of the model are combined to reduce the object depth computation error, allowing us to calculate the distance between an object and the camera after the object’s size is confirmed. As a result, objects are detected, and their depth estimations are validated. Then, the vehicle can determine when and how to stop if an object is nearby. Finally, experiments conducted on the KITTI dataset demonstrate that our method is effective and performs better than other baseline methods, especially in terms of 3D object detection and bird’s-eye view (BEV) evaluation.

A Framework for Representing, Building and Reusing Novel State-of-the-Art Three-Dimensional Object Detection Models in Point Clouds Targeting Self-Driving Applications.

The rapid development of deep learning has brought novel methodologies for 3D object detection using LiDAR sensing technology. These improvements in precision and inference speed performances lead to notable high performance and real-time inference, which is especially important for self-driving purposes. However, the developments carried by these approaches overwhelm the research process in this area since new methods, technologies and software versions lead to different project necessities, specifications and requirements. Moreover, the improvements brought by the new methods may be due to improvements in newer versions of deep learning frameworks and not just the novelty and innovation of the model architecture. Thus, it has become crucial to create a framework with the same software versions, specifications and requirements that accommodate all these methodologies and allow for the easy introduction of new methods and models. A framework is proposed that abstracts the implementation, reusing and building of novel methods and models. The main idea is to facilitate the representation of state-of-the-art (SoA) approaches and simultaneously encourage the implementation of new approaches by reusing, improving and innovating modules in the proposed framework, which has the same software specifications to allow for a fair comparison. This makes it possible to determine if the key innovation approach outperforms the current SoA by comparing models in a framework with the same software specifications and requirements.

FANet: Improving 3D Object Detection with Position Adaptation

Three-dimensional object detection plays a crucial role in achieving accurate and reliable autonomous driving systems. However, the current state-of-the-art two-stage detectors lack flexibility and have limited feature extraction capabilities to effectively handle the disorder and irregularity of point clouds. In this paper, we propose a novel network called FANet, which combines the strengths of PV-RCNN and PAConv (position adaptive convolution). The goal of FANet is to address the irregularity and disorder present in point clouds. In our network, the convolution operation constructs convolutional kernels using a basic weight matrix, and the coefficients of these kernels are adaptively learned by LearnNet from relative points. This approach allows for the flexible modeling of complex spatial variations and geometric structures in 3D point clouds, leading to the improved extraction of point cloud features and generation of high-quality 3D proposal boxes. Compared to other methods, extensive experiments on the KITTI dataset have demonstrated that the FANet exhibits superior 3D object detection accuracy, showcasing a significant improvement in our approach.

AV PV-RCNN: Improving 3D Object Detection with Adaptive Deformation and VectorPool Aggregation

Three-dimensional object detection has attracted more and more attention from industry and academia due to its wide application in various fields such as autonomous driving and robotics. Currently, the refinement methods used by advanced two-stage detectors cannot fully adapt to different object scales, different point cloud densities, partial deformation and clutter, and excessive resource consumption. We propose a point cloud-based 3D object detection method that can adapt to different object scales and aggregate local features with less resources. The method first passes through an adaptive deformation module based on a 2D deformable convolutional network, which can adaptively collect instance-specific features from where the information content exists. Secondly, through a VectorPool aggregation module, this module can better aggregate local point features with less resource consumption. Finally, through a context fusion module, the key points can filter out relevant context information for the refinement stage. Our proposed detection method not only achieves better accuracy on the KITTI dataset, but also consumes less resources than the original detectors and has faster inference speed.

A Two-Stage Pillar Feature-Encoding Network for Pillar-Based 3D Object Detection

Three-dimensional object detection plays a vital role in the field of environment perception in autonomous driving, and its results are crucial for the subsequent processes. Pillar-based 3D object detection is a method to detect objects in 3D by dividing point cloud data into pillars and extracting features from each pillar. However, the current pillar-based 3D object-detection methods suffer from problems such as “under-segmentation” and false detections in overlapping and occluded scenes. To address these challenges, we propose an improved pillar-based 3D object-detection network with a two-stage pillar feature-encoding (Ts-PFE) module that considers both inter- and intra-relational features among and in the pillars. This novel approach enhances the model’s ability to identify the local structure and global distribution of the data, which improves the distinction between objects in occluded and overlapping scenes and ultimately reduces under-segmentation and false detection problems. Furthermore, we use the attention mechanism to improve the backbone and make it focus on important features. The proposed approach is evaluated on the KITTI dataset. The experimental results show that the detection accuracy of the proposed approach are significantly improved on the benchmarks of BEV and 3D. The improvement of AP for car, pedestrian, and cyclist 3D detection are 1.1%, 3.78%, and 2.23% over PointPillars.

MENet: Map-enhanced 3D object detection in bird’s-eye view for LiDAR point clouds

Three-dimensional (3D) object detection utilizes numerous onboard sensors to determine the position, size, and motion information of surrounding objects. Recently, some researchers have utilized HD maps in 3D object detection for LiDAR point clouds. However, existing LiDAR–map fusion detection methods simply take the HD map as an additional input and more prior information in the map is underexplored. To address this limitation, we employ the HD map in data augmentation for LiDAR 3D detection to eliminate the bias of distributions between the augmented training and test sets. Furthermore, we propose a map-enhanced 3D object detection network, called MENet, for LiDAR point clouds. Compared with the standard LiDAR-only detector in bird’s eye view (BEV), MENet extends LiDAR-based baselines by using a map encoder to encode the input rasterized HD map into a BEV feature. In addition, it uses a multimodal fuser to aggregate the map BEV feature with the LiDAR BEV feature extracted in the LiDAR stream. MENet achieved a mean average precision (mAP) of 56.90% and nuScenes detection score (NDS) of 63.43%, improving the mAP and NDS of its LiDAR-only backbone by 4.18% and 2.22%, respectively. Although the HD map is necessary at training time, the map-enhanced 3D detector can work robustly and perform better than the original baseline when there was no map input at test time. Experimental results further indicate that the proposed map plugins can improve the performance of both anchor-based and anchor-free models on different datasets. The code will be open-source at https://github.com/WHU-USI3DV/MENet.