KITTI Dataset Research Articles

MS3D: A Multi-Scale Feature Fusion 3D Object Detection Method for Autonomous Driving Applications

With advancements in autonomous driving, LiDAR has become central to 3D object detection due to its precision and interference resistance. However, challenges such as point cloud sparsity and unstructured data persist. This study introduces MS3D (Multi-Scale Feature Fusion 3D Object Detection Method), a novel approach to 3D object detection that leverages the architecture of a 2D Convolutional Neural Network (CNN) as its core framework. It integrates a Second Feature Pyramid Network to enhance multi-scale feature representation and contextual integration. The Adam optimizer is employed for efficient adaptive parameter tuning, significantly improving detection performance. On the KITTI dataset, MS3D achieves average precisions of 93.58%, 90.91%, and 88.46% in easy, moderate, and hard scenarios, respectively, surpassing state-of-the-art models like VoxelNet, SECOND, and PointPillars.

Adaptive occlusion object detection algorithm based on OL-IoU.

The continuous advancement of autonomous driving technology imposes higher demands on the accuracy of target detection in complex environments, particularly when traffic targets are occluded. Existing algorithms still face significant challenges in detection accuracy and real-time performance under such conditions. To address this issue, this paper proposes an improved YOLOX algorithm based on adaptive deformable convolution, named OCC-YOLOX. This algorithm enhances the feature extraction network's ability to focus on occluded targets by incorporating a coordinate attention mechanism. Additionally, it introduces the Overlapping IoU (OL-IoU) loss function to optimize the overlap between predicted and ground truth bounding boxes, thereby improving detection accuracy. Furthermore, the adoption of Fast Spatial Pyramid Pooling (Fast SPP) reduces computational complexity while maintaining real-time performance. Experiments on fused public datasets demonstrate that OCC-YOLOX achieves improvements in accuracy, recall, and average precision by 2.76%, 1.25%, and 1.92%, respectively. In addition to testing on the KITTI, CityPersons, and BDD100K datasets, the effectiveness of the OCC-YOLOX algorithm is further validated through comparisons with self-collected occlusion scene data. The experimental results indicate that OCC-YOLOX outperforms existing mainstream detection algorithms, particularly in handling complex occlusion scenarios, significantly enhancing the accuracy and efficiency of object detection. This study provides new insights for addressing the challenges of occluded target detection in intelligent transportation systems.

SLAM Algorithm for Mobile Robots Based on Improved LVI-SAM in Complex Environments

The foundation of robot autonomous movement is to quickly grasp the position and surroundings of the robot, which SLAM technology provides important support for. Due to the complex and dynamic environments, single-sensor SLAM methods often have the problem of degeneracy. In this paper, a multi-sensor fusion SLAM method based on the LVI-SAM framework was proposed. First of all, the state-of-the-art feature detection algorithm SuperPoint is used to extract the feature points from a visual-inertial system, enhancing the detection ability of feature points in complex scenarios. In addition, to improve the performance of loop-closure detection in complex scenarios, scan context is used to optimize the loop-closure detection. Ultimately, the experiment results show that the RMSE of the trajectory under the 05 sequence from the KITTI dataset and the Street07 sequence from the M2DGR dataset are reduced by 12% and 11%, respectively, compared to LVI-SAM. In simulated complex environments of animal farms, the error of this method at the starting and ending points of the trajectory is less than that of LVI-SAM, as well. All these experimental comparison results prove that the method proposed in this paper can achieve higher precision and robustness performance in localization and mapping within complex environments of animal farms.

PV-YOLO: A lightweight pedestrian and vehicle detection model based on improved YOLOv8

With the frequent occurrence of urban traffic accidents, fast and accurate detection of pedestrian and vehicle targets has become one of the key technologies for intelligent assisted driving systems. To meet the efficiency and lightweight requirements of smart devices, this paper proposes a lightweight pedestrian and vehicle detection model based on the YOLOv8n model, named PV-YOLO. In the proposed model, receptive-field attention convolution (RFAConv) serves as the backbone network because of its target feature extraction ability, and the neck utilizes the bidirectional feature pyramid network (BiFPN) instead of the original path aggregation network (PANet) to simplify the feature fusion process. Moreover, a lightweight detection head is introduced to reduce the computational burden and improve the overall detection accuracy. In addition, a small target detection layer is designed to improve the accuracy for small distant targets. Finally, to reduce the computational burden further, the lightweight C2f module is utilized to compress the model. The experimental results on the BDD100K and KITTI datasets demonstrate that the proposed PV-YOLO can achieve higher detection accuracy than YOLOv8n and other baseline methods with less model complexity.

Augmented and Virtual Reality Based Segmentation Algorithm for Human Pose Detection in Wearable Cameras

Pose graph optimisation is a crucial method that helps reduce cumulative errors while estimating visual trajectories for wearable cameras. However, when the posture graph's size increases with each additional camera movement, the optimization's efficiency diminishes. In terms of ongoing sensitive applications, such as extended reality and computer-generated reality, direction assessment is a major test. This research proposes an incremental pose graph segmentation technique that accounts for camera orientation variations as a solution to this challenge. The computation only improves the cameras that have seen large direction changes by breaking the posture chart during these instances. As a result, pose graph optimisation is essentially slowed down and optimised more quickly. For every camera that hasn't been optimised using a pose graph, the algorithm employs the wearable cameras at the start and end of each camera's trajectory segment. The final camera in attendance is then determined by weighted average the various postures evaluated with these wearable cameras; this eliminates the need for lengthy nonlinear enhancement computations, reduces disturbance, and achieves excellent accuracy. Experiments on the EuRoC, TUM, and KITTI datasets demonstrate that pose graph optimisation scope is reduced while maintaining camera trajectories accuracy.

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

MSSA: Multi-Representation Semantics-Augmented Set Abstraction for 3D Object Detection

Accurate recognition and localization of 3D objects is a fundamental research problem in 3D computer vision. Benefiting from transformation-free point cloud processing and flexible receptive fields, point-based methods have become accurate in 3D point cloud modeling, but still fall behind voxel-based competitors in 3D detection. We observe that the set abstraction module, commonly utilized by point-based methods for downsampling points, tends to retain excessive irrelevant background information, thus hindering the effective learning of features for object detection tasks. To address this issue, we propose MSSA, a Multi-representation Semantics-augmented Set Abstraction for 3D object detection. Specifically, we first design a backbone network to encode different representation features of point clouds, which extracts point-wise features through PointNet to preserve fine-grained geometric structure features, and adopts VoxelNet to extract voxel features and BEV features to enhance the semantic features of key points. Second, to efficiently fuse different representation features of keypoints, we propose a Point feature-guided Voxel feature and BEV feature fusion (PVB-Fusion) module to adaptively fuse multi-representation features and remove noise. At last, a novel Multi-representation Semantic-guided Farthest Point Sampling (MS-FPS) algorithm is designed to help set abstraction modules progressively downsample point clouds, thereby improving instance recall and detection performance with more important foreground points. We evaluate MSSA on the widely used KITTI dataset and the more challenging nuScenes dataset. Experimental results show that compared to PointRCNN, our method improves the AP of “moderate” level for three classes of objects by 7.02%, 6.76%, and 5.44%, respectively. Compared to the advanced point-voxel-based method PV-RCNN, our method improves the AP of “moderate” level by 1.23%, 2.84%, and 0.55% for the three classes, respectively.

PE-MCAT: Leveraging Image Sensor Fusion and Adaptive Thresholds for Semi-Supervised 3D Object Detection.

Existing 3D object detection frameworks in sensor-based applications heavily rely on large-scale annotated data to achieve optimal performance. However, obtaining such annotations from sensor data-like LiDAR or image sensors-is both time-consuming and costly. Semi-supervised learning offers an efficient solution to this challenge and holds significant potential for sensor-driven artificial intelligence (AI) applications. While it reduces the need for labeled data, semi-supervised learning still depends on a small amount of labeled samples for training. In the initial stages, relying on such limited samples can adversely affect the effective training of student-teacher networks. In this paper, we propose PE-MCAT, a semi-supervised 3D object detection method that generates high-precision pseudo-labels. First, to address the challenges of insufficient local feature capture and poor robustness in point cloud data, we introduce a point enrichment module. This module incorporates information from image sensors and combines multiple feature fusion methods of local and self-features to directly enhance the quality of point clouds and pseudo-labels, compensating for the limitations posed by using only a few labeled samples. Second, we explore the relationship between the teacher network and the pseudo-labels it generates. We propose a multi-class adaptive threshold strategy to initially filter and create a high-quality pseudo-label set. Furthermore, a joint variable threshold strategy is introduced to refine this set further, enhancing the selection of superior pseudo-labels.Extensive experiments demonstrate that PE-MCAT consistently outperforms recent state-of-the-art methods across different datasets. Specifically, on the KITTI dataset and using only 2% of labeled samples, our method improved the mean Average Precision (mAP) by 0.7% for cars, 3.7% for pedestrians, and 3.0% for cyclists.

CWGA-Net: Center-Weighted Graph Attention Network for 3D object detection from point clouds

The precision of 3D object detection from unevenly distributed outdoor point clouds is critical in autonomous driving perception systems. Current point-based detectors employ self-attention and graph convolution to establish contextual relationships between point clouds; however, they often introduce weakly correlated redundant information, leading to blurred geometric details and false detections. To address this issue, a novel Center-weighted Graph Attention Network (CWGA-Net) has been proposed to fuse geometric and semantic similarities for weighting cross-attention scores, thereby capturing precise fine-grained geometric features. CWGA-Net initially constructs and encodes local graphs between foreground points, establishing connections between point clouds from geometric and semantic dimensions. Subsequently, center-weighted cross-attention is utilized to compute the contextual relationships between vertices within the graph, and geometric and semantic similarities between vertices are fused to weight attention scores, thereby extracting strongly related geometric shape features. Finally, a cross-feature fusion Module is introduced to deeply fuse high and low-resolution features to compensate for the information loss during downsampling. Experiments conducted on the KITTI and Waymo datasets demonstrate that the network achieves superior detection capabilities, outperforming state-of-the-art point-based single-stage methods in terms of average precision metrics while maintaining good speed.

MonoCAPE: Monocular 3D object detection with coordinate-aware position embeddings

3D monocular detection remains to be a focal point of research, particularly due to its capacity to deliver available precision under conditions of low cost and simplified configurations, making it especially valuable in fields like autonomous driving. Current 3D object detection methods often overlook the spatial information missing from images, which is critical to spatial perception, and optimize bounding box attributes separately, failing to meet the requirements of autonomous driving. We introduce MonoCAPE, a novel 3D detection framework addressing these issues by encoding spatial information and co-optimizing attributes through a Coordinate-Aware Position Encoding (CAPE) Generator and a Task Co-optimization Strategy (TCS). The CAPE Generator produces sparse positional embeddings, enabling spatial awareness with low computational cost, while the TCS utilizes Gaussian modeling to prevent suboptimal outputs. In this way, our framework comprehensively takes into account what existing approaches ignore. Extensive experiments on the KITTI dataset demonstrate MonoCAPE significantly improves AP3D and APBEV metrics compared to existing advanced methods.

Semi-Supervised Online Continual Learning for 3D Object Detection in Mobile Robotics

Continual learning addresses the challenge of acquiring and retaining knowledge over time across multiple tasks and environments. Previous research primarily focuses on offline settings where models learn through increasing tasks from samples paired with ground truth annotations. In this work, we focus on an unsolved, challenging, yet practical scenario, specifically, the semi-supervised online continual learning in autonomous driving and mobile robotics. In our settings, models are tasked with learning new distributions from streaming unlabeled samples and performing 3D object detection as soon as the LiDAR point cloud arrives. Additionally, we conducted experiments on both the KITTI dataset, our newly built IUSL dataset and Canadian Adverse Driving Conditions (CADC) Dataset. The results indicate that our method achieves a balance between rapid adaptation and knowledge retention, showcasing its effectiveness in the dynamic and complex environment of autonomous driving and mobile robotics. The developed ROS packages and IUSL dataset will be publicly available at: https://github.com/npu-ius-lab/OCL3D.

Stereo and LiDAR Loosely Coupled SLAM Constrained Ground Detection.

In many robotic applications, creating a map is crucial, and 3D maps provide a method for estimating the positions of other objects or obstacles. Most of the previous research processes 3D point clouds through projection-based or voxel-based models, but both approaches have certain limitations. This paper proposes a hybrid localization and mapping method using stereo vision and LiDAR. Unlike the traditional single-sensor systems, we construct a pose optimization model by matching ground information between LiDAR maps and visual images. We use stereo vision to extract ground information and fuse it with LiDAR tensor voting data to establish coplanarity constraints. Pose optimization is achieved through a graph-based optimization algorithm and a local window optimization method. The proposed method is evaluated using the KITTI dataset and compared against the ORB-SLAM3, F-LOAM, LOAM, and LeGO-LOAM methods. Additionally, we generate 3D point cloud maps for the corresponding sequences and high-definition point cloud maps of the streets in sequence 00. The experimental results demonstrate significant improvements in trajectory accuracy and robustness, enabling the construction of clear, dense 3D maps.

A Novel Hybrid XAI Solution for Autonomous Vehicles: Real-Time Interpretability Through LIME-SHAP Integration.

The rapid advancement in self-driving and autonomous vehicles (AVs) integrated with artificial intelligence (AI) technology demands not only precision but also output transparency. In this paper, we propose a novel hybrid explainable AI (XAI) framework that combines local interpretable model-agnostic explanations (LIME) and Shapley additive explanations (SHAP). Our framework combines the precision and globality of SHAP and low computational requirements of LIME, creating a balanced approach for onboard deployment with enhanced transparency. We evaluate the proposed framework on three different state-of-the-art models: ResNet-18, ResNet-50, and SegNet-50 on the KITTI dataset. The results demonstrate that our hybrid approach consistently outperforms traditional approaches by achieving a fidelity rate of more than 85%, interpretability factor of more than 80%, and consistency of more than 70%, surpassing the conventional methods. Furthermore, the inference time of our proposed framework with ResNet-18 was 0.28 s; for ResNet-50, it was 0.571 s; and that for SegNet was 3.889 s with XAI layers. This is optimal for onboard computations and deployment. This research establishes a strong foundation for the deployment of XAI in safety-critical AV with balanced tradeoffs for real-time decision-making.

Occluded vehicle tracking by integrating full-scale features and trajectory correction

In order to improve the tracking drift and identity switching (IDs) problems caused by occlusion in vehicle tracking, a method for tracking occluded vehicles that integrates full-scale features and trajectory correction is proposed based on the Deep SORT algorithm. Firstly, a full-scale feature extraction network is introduced to extract the features of different scales of the target and realize adaptive dynamic fusion to enhance the target appearance characteristics. Then, a trajectory correction algorithm is proposed to repair the tracking trajectory during occlusion, and the Kalman filter parameters are updated to reduce the accumulated linear error during occlusion and optimize the target motion characteristics. Finally, the occluded vehicle is tracked by combining appearance features and motion features. The feasibility of the proposed method is verified by ablation experiments and visualization analysis. The experimental results on the KITTI dataset show that compared with the existing typical methods, the proposed method achieves the highest comprehensive score of 78.13 and the lowest number of IDs of 192, which effectively improves the IDs problem in occluded vehicle tracking and improves the robustness of vehicle tracking.

Three-Dimensional Outdoor Object Detection in Quadrupedal Robots for Surveillance Navigations

Quadrupedal robots are confronted with the intricate challenge of navigating dynamic environments fraught with diverse and unpredictable scenarios. Effectively identifying and responding to obstacles is paramount for ensuring safe and reliable navigation. This paper introduces a pioneering method for 3D object detection, termed viewpoint feature histograms, which leverages the established paradigm of 2D detection in projection. By translating 2D bounding boxes into 3D object proposals, this approach not only enables the reuse of existing 2D detectors but also significantly increases the performance with less computation required, allowing for real-time detection. Our method is versatile, targeting both bird’s eye view objects (e.g., cars) and frontal view objects (e.g., pedestrians), accommodating various types of 2D object detectors. We showcase the efficacy of our approach through the integration of YOLO3D, utilizing LiDAR point clouds on the KITTI dataset, to achieve real-time efficiency aligned with the demands of autonomous vehicle navigation. Our model selection process, tailored to the specific needs of quadrupedal robots, emphasizes considerations such as model complexity, inference speed, and customization flexibility, achieving an accuracy of up to 99.93%. This research represents a significant advancement in enabling quadrupedal robots to navigate complex and dynamic environments with heightened precision and safety.

Uncertainty-Aware Depth Network for Visual Inertial Odometry of Mobile Robots

Simultaneous localization and mapping, a critical technology for enabling the autonomous driving of vehicles and mobile robots, increasingly incorporates multi-sensor configurations. Inertial measurement units (IMUs), known for their ability to measure acceleration and angular velocity, are widely utilized for motion estimation due to their cost efficiency. However, the inherent noise in IMU measurements necessitates the integration of additional sensors to facilitate spatial understanding for mapping. Visual–inertial odometry (VIO) is a prominent approach that combines cameras with IMUs, offering high spatial resolution while maintaining cost-effectiveness. In this paper, we introduce our uncertainty-aware depth network (UD-Net), which is designed to estimate both depth and uncertainty maps. We propose a novel loss function for the training of UD-Net, and unreliable depth values are filtered out to improve VIO performance based on the uncertainty maps. Experiments were conducted on the KITTI dataset and our custom dataset acquired from various driving scenarios. Experimental results demonstrated that the proposed VIO algorithm based on UD-Net outperforms previous methods with a significant margin.

Adaptive sensor attack detection and defense framework for autonomous vehicles based on density

The security of autonomous vehicles heavily depends on localization systems that integrate multiple sensors, which are vulnerable to sensor attacks and increase the risk of accidents. Given the diversity of sensor attacks and the dynamic changing of driving scenarios of autonomous vehicles, an adaptive and effective attack detection and defense framework faces a considerable challenge. This paper proposes a novel real-time adaptive attack detection and defense framework based on density, which can detect and identify attacked sensors and effectively recover data. We first develop a reinforcement learning multi-armed Bandit-based Density-Based Spatial Clustering of Applications with Noise (BDBSCAN) algorithm that selects hyperparameters adaptively. The Adaptive Extended Kalman Filter (AEKF) combines with the vehicle dynamic model on the localization system and extracts data features used for the BDBSCAN algorithm to monitor potential sensor attacks. If attack detection indicates possible system compromise, AEKF is further employed on localization sensors with anomalies identified through the BDBSCAN algorithm of the attacked sensors. To ensure precision and reliability, the data recovery incorporates a redundancy mechanism to apply a decision tree to select the optimal state estimation between AEKF and Extended Kalman Filter (EKF) to replace corrupted sensor data. To evaluate the effectiveness and adaptability of the proposed framework, we conducted 15,000 experiments using the real-world KITTI and V2V4Real datasets across various driving and sensor attack scenarios. The results demonstrate that our proposed framework achieves 100% accuracy and 0% false alarm rate in various driving scenarios for attack detection within 0.15 s, with a recovery time of 0.08 s.

LapUNet: a novel approach to monocular depth estimation using dynamic laplacian residual U-shape networks

Monocular depth estimation is an important but challenging task. Although the performance has been improved by adopting various encoder-decoder architectures, the estimated depth maps lack structure details and clear edges due to simple repeated upsampling. To solve this problem, this paper presents the novel LapUNet (Laplacian U-shape networks), in which the encoder adopts ResNeXt101, and the decoder is constructed with the novel DLRU (dynamic Laplacian residual U-shape) module. The DLRU module based on the U-shape structure can supplement high-frequency features by fusing dynamic Laplacian residual into the process of upsampling, and the residual is dynamically learnable due to the addition of convolutional operation. Also, the ASPP (atrous spatial pyramid pooling) module is introduced to capture image context at multiple scales though multiple parallel atrous convolutional layers, and the depth map fusion module is used for combining high and low frequency features from depth maps with different spatial resolution. Experiments demonstrate that the proposed model with moderate model size is superior to other previous competitors on the KITTI and NYU Depth V2 datasets. Furthermore, 3D reconstruction and target ranging by utilizing the estimated depth maps prove the effectiveness of our proposed method.

Multi-Scale Candidate Fusion and Optimization-Based 3D Object Detection Algorithm

To address the issues of target omission and the inclusion of a large number of background points in keypoint sampling for point cloud-based object detection, an improved algorithm based on the PV-RCNN network is introduced. This approach employs both a regional proposal fusion network and weighted Non-Maximum Suppression (NMS) to merge proposals generated at various scales while eliminating redundancy. A segmentation network is utilized to segment foreground points from the original point cloud, and object center points are identified based on these proposals. Gaussian density functions are employed for regional density estimation, which assigns different sampling weights to solve the problem of difficult sampling in sparse areas. Experimental evaluations on the KITTI dataset indicate that the algorithm enhances the average precision at medium difficulty levels by 0.39%, 1.31%, and 0.63%for cars, pedestrians, and cyclists, respectively. Generalization experiments were also conducted on the Waymo dataset. The results suggest that the introduced algorithm achieves higher accuracy compared to most of the existing 3D object detection networks.

EFMF-pillars: 3D object detection based on enhanced features and multi-scale fusion

As unmanned vehicle technology advances rapidly, obstacle recognition and target detection are crucial links, which directly affect the driving safety and efficiency of unmanned vehicles. In response to the inaccurate localization of small targets such as pedestrians in current object detection tasks and the problem of losing local features in the PointPillars, this paper proposes a three-dimensional object detection method based on improved PointPillars. Firstly, addressing the issue of lost spatial and local information in the PointPillars, the feature encoding part of the PointPillars is improved, and a new pillar feature enhancement extraction module, CSM-Module, is proposed. Channel encoding and spatial encoding are introduced in the new pillar feature enhancement extraction module, fully considering the spatial information and local detailed geometric information of each pillar, thereby enhancing the feature representation capability of each pillar. Secondly, based on the fusion of CSPDarknet and SENet, a new backbone network CSE-Net is designed in this paper, enabling the extraction of rich contextual semantic information and multi-scale global features, thereby enhancing the feature extraction capability. Our method achieves higher detection accuracy when validated on the KITTI dataset. Compared to the original network, the improved algorithm’s average detection accuracy is increased by 3.42%, it shows that the method is reasonable and valuable.