KITTI Dataset Research Articles

KPTr: Key point transformer for LiDAR-based 3D object detection

Accurate 3D environmental perception is crucial for ensuring the safety of autonomous vehicles. However, many existing point-based 3D object detection methods rely only on independent point features to extract spatial information and ignore the dependencies between points. Meanwhile, these methods usually use max-pooling operations to aggregate neighboring point features without considering the stability of local geometric information. To address these issues, this paper proposes a novel two-stage LiDAR-based key point transformer that consists of a region proposal network module and a region-based transformer module to enhance 3D detection performance. Firstly, unprocessed raw point clouds are taken as the input of the model to preserve detailed spatial structural information. Secondly, in the region proposal network an enhanced PointNet++ is used to segment foreground points and generate high-quality first-stage proposals. Then, the representation of proposals is improved by the region-based transformer module, which constructs long-range key point cloud relationship features within the regions of interest and diffuses spatial information into the feature channel layers. Additionally experiments on the KITTI dataset show that detection performance of the proposed method surpasses most mainstream 3D object detection methods and achieves a detection speed of 14.28 Hz, striking a good balance between detection accuracy and speed. Finally, the proposed method is deployed on real vehicle hardware platforms for online detection, thereby proving its industrial value.

PV-LaP: Multi-sensor fusion for 3D Scene Understanding in intelligent transportation systems

Intelligent transportation systems are pivotal in modern urban development, aiming to enhance traffic management efficiency, safety, and sustainability. However, existing 3D Visual Scene Understanding methods often face challenges of robustness and high computational complexity in complex traffic environments. This paper proposes a Multi-Sensor Signal Fusion method based on PV-RCNN and LapDepth (PV-LaP) to improve 3D Visual Scene Understanding. By integrating camera and LiDAR data, the PV-LaP method enhances environmental perception accuracy. Evaluated on the KITTI and WHU-TLS datasets, the PV-LaP framework demonstrated superior performance. On the KITTI dataset, our method achieved an Absolute Relative Error (Abs Rel) of 0.079 and a Root Mean Squared Error (RMSE) of 3.014, outperforming state-of-the-art methods. On the WHU-TLS dataset, the method improved 3D reconstruction precision with a PSNR of 19.15 and an LPIPS of 0.299. Despite its high computational demands, PV-LaP offers significant improvements in accuracy and robustness, providing valuable insights for the future development of intelligent transportation systems.

Lightweight Object Detection Based on Autonomous Driving

Abstract Addressing the issues of low accuracy and large size for tiny objects like pedestrians, non-motor vehicles, and vehicles in traditional object detection models for autonomous driving. An advanced variant of the YOLOv7-tiny model for automatic driving object detection is proposed in this study. The VoVEGSCSP module, designed for cross-layer integration within local networks, is introduced. The SPPCSPC module was replaced with the Space Pyramid Pool Fast (SPPF). This reduces parameters and calculations while preserving the diversity of the original feature scale. CARAFE is a lightweight universal upsampling operator that replaces nearest neighbor interpolation in the upsampling module, thereby reducing feature information loss during upsampling. Results demonstrate that on the KITTI dataset, the improved YOLOv7-tiny algorithm reduces parameters by 35.7% and computation by 34.1%, with only a 0.7% decrease in mAP@0.5 accuracy.

Research on 3D laser SLAM algorithm based on graph optimization

Abstract Aiming at the problems that the positioning accuracy of laser Simultaneous Localization and Mapping (SLAM) is affected by the misclassification of close feature points and incorrect feature matching in the environment with redundant features, a 3D laser SLAM algorithm based on graph optimization was proposed. Firstly, to reduce the feature extraction error problem, the distance threshold is used to filter the close point cloud and improve the accuracy of feature classification, enhanced reliability of optoelectronic measurement data. Secondly, to improve the consistency of feature matching, based on the rough matching based on k-dimensional tree (KD-Tree), a two-stage fine matching based on geometric consistency is introduced to improved laser point alignment accuracy. The proposed method is compared with the mainstream localization algorithms based on KITTI dataset. Compared with Lidar-IMU Odometry with Sparse Adaptive Mapping and Fast LiDAR-Inertial Odometry, the positioning error of the proposed method is reduced by 21.88%and 24.43%on average, which showsthe superiority of the present algorithm in the application of optoelectronics and optical technology, and provides new ideas and methods for the development of high-precision laser navigation technology.

RS-SLAM: real time semantic slam with driverless car using LiDAR-camera-IMU sensing

Abstract Accurate and robust Simultaneous Localization and Mapping (SLAM) technology is a critical component of driverless cars, and semantic information plays a vital role in their analysis and understanding of the scene. In the actual scene, the moving object will produce the shadow phenomenon in the mapping process. The positioning accuracy and mapping effect will be affected. Therefore, we propose a semantic SLAM framework combining LiDAR, IMU, and camera, which includes a semantic fusion front-end odometry module and a closed-loop back-end optimization module based on semantic information. An improved image semantic segmentation algorithm based on Deeplabv3+ is designed to enhance the performance of the image semantic segmentation model by replacing the backbone network and introducing an attention mechanism to ensure the accuracy of point cloud segmentation. Dynamic objects are detected and eliminated by calculating the similarity score of semantic labels. A loop closure detection method based on semantic information is proposed to detect key semantic features and use threshold range detection and point cloud re-matching to establish the correct loop closure detection, and finally reduce the global cumulative error and improve the global trajectory accuracy using graph optimization to ultimately obtain the global motion trajectory and realize the construction of 3D semantic maps. We evaluated it on the KITTI dataset and collected a dataset for evaluation by ourselves, which includes four different sequences. The results show that the proposed framework has good positioning accuracy and mapping effect in large-scale urban road environments.

Towards faster yet accurate video prediction for resource-constrained platforms

Video prediction has emerged as a critical task in the field of computer vision. While conventional deep learning-based video prediction models excel in accuracy, they often impose significant computational overhead. Moreover, they tend to disregard crucial factors like inference latency and memory consumption, which may render them unsuitable for real-time video prediction applications. This issue is further exacerbated by the necessity to deploy many of these applications on resource-limited embedded devices. In this study, we introduce an innovative faster yet accurate video prediction (FAVP) model designed to mitigate inference latency and memory footprint, all the while maintaining predictive accuracy. Our model employs an encoder-convertor-decoder architecture with a novel multi-layer convertor. Each layer within the convertor uses consistent kernel sizes to precisely capture dimensional features, while varying kernel sizes between layers effectively capture both local and global features. This combination of uniformity within layers and diversity between layers enhances the model’s ability to grasp dynamic changes. Furthermore, we demonstrate that replacing conventional large-kernel convolutions with involutions significantly trims the model’s parameter count and inference latency, without compromising prediction accuracy. Through comprehensive experiments conducted on resource-adequate x86 platforms, utilizing KTH, TrafficBJ, Human3.6M, Moving MNIST, and KTH-Enhanced datasets, we showcase our model’s exceptional prediction performance, particularly in terms of inference latency. Additionally, we evaluate the model on resource-constrained NVIDIA Jetson Nano platform using KITTI & CalTech Pedestrian dataset. The results underscore our model’s superior inference speed and its competence in meeting real-time video prediction requirements, even under stringent resource limitations.

TCANet: Three-dimensional cross-attention mechanism for stereo-matching

Abstract Effective disparity estimation is a current hotspot in stereo vision research, and cost aggregation is an important part of disparity prediction. How to perform more effective cost aggregation is the core step to improve the accuracy of disparity prediction. In previous studies, 3DCNN with a stacked hourglass structure is often used for cost aggregation. In this research, we propose an effective 3D cross-attention stereo network that utilizes the attention mechanism to obtain contextual information for cost aggregation in a more efficient way. Specifically, the 3D cross-attention module in TCANet acquires the geometric information of all pixels on the 3D cross path. By repeating this operation twice, each pixel can eventually obtain global dependencies from all other pixels. Using the 3D cross-attention module in a stacked hourglass-structured 3DCNN only increases the number of parameters by a very small amount, which can effectively improve the performance of the model. Experimental results show that TCANet performs well on virtual dataset Scene Flow and realistic KITTI datasets.

When-to-Loop: Enhanced Loop Closure for LiDAR SLAM in Urban Environments Based on SCAN CONTEXT

Global Navigation Satellite Systems (GNSSs) frequently encounter challenges in providing reliable navigation and positioning within urban canyons due to signal obstruction. Micro-Electro-Mechanical System (MEMS) Inertial Measurement Units (IMUs) offers an alternative for autonomous navigation, but they are susceptible to accumulating errors. To mitigate these influences, a LiDAR-based Simultaneous Localization and Mapping (SLAM) system is often employed. However, these systems face challenges in drift and error accumulation over time. This paper presents a novel approach to loop closure detection within LiDAR-based SLAM, focusing on the identification of previously visited locations to correct time-accumulated errors. Specifically, the proposed method leverages the vehicular drivable area and IMU trajectory to identify significant environmental changes in keyframe selection. This approach differs from conventional methods that only rely on distance or time intervals. Furthermore, the proposed method extends the SCAN CONTEXT algorithm. This technique incorporates the overall distribution of point clouds within a region rather than solely relying on maximum height to establish more robust loop closure constraints. Finally, the effectiveness of the proposed method is validated through experiments conducted on the KITTI dataset with an enhanced accuracy of 6%, and the local scenarios exhibit a remarkable improvement in accuracy of 17%, demonstrating improved robustness in loop closure detection for LiDAR-based SLAM.

3D Object Detection via Residual SqueezeDet

Three-dimensional object detection is a critical task in computer vision with applications in autonomous driving, robotics, and augmented reality. This paper introduces Residual SqueezeDet, a novel network architecture that enhances the performance of 3D object detection on the KITTI dataset. Building upon the efficient SqueezeDet framework, we propose the Residual Fire module, which incorporates skip connections inspired by ResNet architectures into the original Fire module. This innovation improves gradient flow, enhances feature propagation, and allows for more effective training of deeper networks. Our method leverages point cloud and image-based features, employing a Residual SqueezeDet to effectively capture local and global context. Extensive experiments on the KITTI dataset demonstrate that Residual SqueezeDet significantly outperforms the original SqueezeDet, with particularly notable improvements in challenging scenarios. The proposed model maintains computational efficiency while achieving state-of-the-art performance, making it well-suited for real-time applications in autonomous driving. Our work contributes to the field by providing a more accurate and robust solution for 3D object detection, paving the way for improved perception systems in dynamic environments.

CTAFFNet: CNN–Transformer Adaptive Feature Fusion Object Detection Algorithm for Complex Traffic Scenarios

As the core technology of an environmental perception system, object detection has received more and more attention and has become a hot research direction for intelligent driving vehicles. The CNN–Transformer hybrid model has poor generalization ability, making it difficult to meet the detection requirements for small objects in complex scenes. We propose a novel convolutional neural network (CNN)–Transformer Adaptive Feature Fusion Network (CTAFFNet) for object detection. First, we design a Local–Global Feature Fusion unit known as the Convolutional Transformation Adaptive Fusion Kernel (CTAFFK), which is integrated into CTAFFNet. The CTAFFK kernel utilizes two branches, namely CNN and Transformer, to extract local and global features from the image, and adaptively fuses the features from both branches. Subsequently, we develop an adaptive feature fusion strategy that combines local high-frequency and global low-frequency features to obtain comprehensive feature information. Finally, CTAFFNet employs an encoder–decoder structure to facilitate the flow of fused local–global information between different stages, ensuring the model’s generalization capabilities. Results from the experiment conducted on the large and challenging KITTI dataset demonstrate the effectiveness and efficiency of the proposed network. Compared with other mainstream networks, it achieves an average precision of 91.17%, particularly excelling in the detection of small objects at longer distances with a remarkable 70.18% accuracy, while also providing real-time detection capabilities.

Global Localization using OpenStreetMap and Elevation Offsets

Localization is a critical component in autonomous vehicle navigation stacks. While GNSS-only localization cannot be fully reliable and available all the time, localization based on 3D high-definition (HD) maps have to be robust to world changes, which is still a challenging issue. Added to that, in general, HD maps are expensive and difficult to construct and maintain.In this paper, we propose a particle filter-based 2D global pose estimation method that can use the crowdsourced OpenStreetMap (OSM) API, a digital surface map, or both. The main contributions of the proposed approach are: that it is lightweight, does not require the vehicle to map the environment, does not require a GPU (can be used with low-power computing resources), is agnostic to the odometry source, and achieved relatively low position and orientation errors for this localization modality using the KITTI dataset sequences. The proposed method's implementation is open source and is available with the experimental results on our GitHub page.

High‐order multilayer attention fusion network for 3D object detection

AbstractThree‐dimensional object detection based on the fusion of 2D image data and 3D point clouds has become a research hotspot in the field of 3D scene understanding. However, different sensor data have discrepancies in spatial position, scale, and alignment, which severely impact detection performance. Inappropriate fusion methods can lead to the loss and interference of valuable information. Therefore, we propose the High‐Order Multi‐Level Attention Fusion Network (HMAF‐Net), which takes camera images and voxelized point clouds as inputs for 3D object detection. To enhance the expressive power between different modality features, we introduce a high‐order feature fusion module that performs multi‐level convolution operations on the element‐wise summed features. By incorporating filtering and non‐linear activation, we extract deep semantic information from the fused multi‐modal features. To maximize the effectiveness of the fused salient feature information, we introduce an attention mechanism that dynamically evaluates the importance of pooled features at each level, enabling adaptive weighted fusion of significant and secondary features. To validate the effectiveness of HMAF‐Net, we conduct experiments on the KITTI dataset. In the “Car,” “Pedestrian,” and “Cyclist” categories, HMAF‐Net achieves mAP performances of 81.78%, 60.09%, and 63.91%, respectively, demonstrating more stable performance compared to other multi‐modal methods. Furthermore, we further evaluate the framework's effectiveness and generalization capability through the KITTI benchmark test, and compare its performance with other published detection methods on the 3D detection benchmark and BEV detection benchmark for the “Car” category, showing excellent results. The code and model will be made available on https://github.com/baowenzhang/High‐order‐Multilayer‐Attention‐Fusion‐Network‐for‐3D‐Object‐Detection.

VIDF-Net: A Voxel-Image Dynamic Fusion method for 3D object detection

In recent years, multi-modal fusion methods have shown excellent performance in the field of 3D object detection, which select the voxel centers and globally fuse with image features across the scene. However, these approaches exist two issues. First, The distribution of voxel density is highly heterogeneous due to the discrete volumes. Additionally, there are significant differences in the features between images and point clouds. Global fusion does not take into account the correspondence between these two modalities, which leads to the insufficient fusion. In this paper, we propose a new multi-modal fusion method named Voxel-Image Dynamic Fusion (VIDF). Specifically, VIDF-Net is composed of the Voxel Centroid Mapping module (VCM) and the Deformable Attention Fusion module (DAF). The Voxel Centroid Mapping module is used to calculate the centroid of voxel features and map them onto the image plane, which can locate the position of voxel features more effectively. We then use the Deformable Attention Fusion module to dynamically calculates the offset of each voxel centroid from the image position and combine these two modalities. Furthermore, we propose Region Proposal Network with Channel-Spatial Aggregate to combine channel and spatial attention maps for improved multi-scale feature interaction. We conduct extensive experiments on the KITTI dataset to demonstrate the outstanding performance of proposed VIDF network. In particular, significant improvements have been observed in the Hard categories of Cars and Pedestrians, which shows the significant effectiveness of our approach in dealing with complex scenarios.

Improved Pedestrian Vehicle Detection for Small Objects Based on Attention Mechanism

Abstract This study aims to solve the low detection accuracy and susceptibility to false detection and omission in pedestrian and vehicle detection by proposing an improved YOLOv5s algorithm. Firstly, a small target detection module is added to better acquire and determine the information of pedestrians from long-range vehicles. Secondly, the multi-scale channel attention CBAM attention module is added, and the dual attention mechanism is not only flexible and convenient, but also improves the computational efficiency. Finally, the MPDIoU loss function based on minimum point distance is introduced to replace the original GIoU loss function, and this change not only enhances the regression accuracy of the model. At the same time, the convergence speed of the model is accelerated. KITTI data set was used for experiments, and the experimental results showed that the average accuracy of the model trained by the improved YOLOv5s algorithm on the data set reached 84.9%, which was 3.7% higher than that of the original YOLOv5s algorithm. It is verified that the model is suitable for high accuracy of pedestrian and vehicle recognition in complex environments, and has high value for promotion.

Enhancing point cloud registration with transformer: cultural heritage protection of the Terracotta Warriors

Point cloud registration technology, by precisely aligning repair components with the original artifacts, can accurately record the geometric shape of cultural heritage objects and generate three-dimensional models, thereby providing reliable data support for the digital preservation, virtual exhibition, and restoration of cultural relics. However, traditional point cloud registration methods face challenges when dealing with cultural heritage data, including complex morphological and structural variations, sparsity and irregularity, and cross-dataset generalization. To address these challenges, this paper introduces an innovative method called Enhancing Point Cloud Registration with Transformer (EPCRT). Firstly, we utilize local geometric perception for positional encoding and combine it with a dynamic adjustment mechanism based on local density information and geometric angle encoding, enhancing the flexibility and adaptability of positional encoding to better characterize the complex local morphology and structural variations of artifacts. Additionally, we introduce a convolutional-Transformer hybrid module to facilitate interactive learning of artifact point cloud features, effectively achieving local–global feature fusion and enhancing detail capture capabilities, thus effectively handling the sparsity and irregularity of artifact point cloud data. We conduct extensive evaluations on the 3DMatch, ModelNet, KITTI, and MVP-RG datasets, and validate our method on the Terracotta Warriors cultural heritage dataset. The results demonstrate that our method has significant performance advantages in handling the complexity of morphological and structural variations, sparsity and irregularity of relic data, and cross-dataset generalization.

FELC-SLAM: feature extraction and loop closure optimized lidar SLAM system

Abstract Simultaneous Localization and Mapping (SLAM) is one of the key technologies in robot navigation and autonomous driving, playing an important role in robot navigation. Due to the sparsity of LiDAR data and the singularity of point cloud features, accuracy loss of LiDAR SLAM can occur during point cloud matching and localization. In response to these issues, this paper proposes a LiDAR Measurement SLAM algorithm that integrates multi type geometric feature extraction and optimized point cloud registration algorithms. This article first adopts advanced ground segmentation methods and feature segmentation strategies, including ground features, edge features, planar features, and spherical features, to improve matching accuracy. In addition, this article improves the previous method for extracting edge and planar features, extracting clearer and more robust line and surface features to address the degradation of geometric features. Finally, by introducing a robust decoupling global registration method for loop closure detection in the backend of the system, the sparsity problem of distant point clouds and the degradation problem caused by the reduction of inner layers in point cloud registration were effectively solved. In the evaluation of the KITTI dataset, our algorithm reduced absolute trajectory error values by 60%, 29%, and 71% compared to LeGO-LOAM in multi loop and feature constrained scenarios (such as sequences 00, 01, and 02), respectively. The evaluation of the M2DGR and Botanic Garden datasets also indicates that the positioning accuracy of our algorithm is superior to other advanced LiDAR SLAM algorithms.

Learning accurate monocular 3D voxel representation via bilateral voxel transformer

Vision-based methods for 3D scene perception have been widely explored for autonomous vehicles. However, inferring complete 3D semantic scenes from monocular 2D images is still challenging owing to the 2D-to-3D transformation. Specifically, existing methods that use Inverse Perspective Mapping (IPM) to project image features to dense 3D voxels severely suffer from the ambiguous projection problem. In this research, we present Bilateral Voxel Transformer (BVT), a novel and effective Transformer-based approach for monocular 3D semantic scene completion. BVT exploits a bilateral architecture composed of two branches for preserving the high-resolution 3D voxel representation while aggregating contexts through the proposed Tri-Axial Transformer simultaneously. To alleviate the ill-posed 2D-to-3D transformation, we adopt position-aware voxel queries and dynamically update the voxels with image features through weighted geometry-aware sampling. BVT achieves 11.8 mIoU on the challenging Semantic KITTI dataset, considerably outperforming previous works for semantic scene completion with monocular images. The code and models of BVT will be available on GitHub.

SLG-SLAM: An integrated SLAM framework to improve accuracy using semantic information, laser and GNSS data

Visual Simultaneous Localization and Mapping (V-SLAM) is pivotal for precise positioning and mapping. However, visual data from crowd-sourced datasets often contains deficiencies that may lead to positioning errors. Despite existing optimization techniques, current algorithms do not adequately adapt to varied data in vehicle driving scenarios. To address this gap, this study introduces a novel SLAM framework (SLG-SLAM). This framework refines trajectories by integrating semantic information, laser point cloud, and global navigation satellite system (GNSS) data into V-SLAM. Initial trajectory estimates are made after filtering out dynamic targets and are subsequently refined with matched laser point clouds, then corrected for scale and direction using GNSS. The efficacy of this approach is assessed using four public datasets and one self-collected dataset, showing significant enhancements across all datasets. The proposed method reduces the mean absolute trajectory error by 43.50% on the KITTI dataset and 14.91% on the MVE dataset compared to the baseline. Unlike the baseline, which fails on three other datasets, the proposed method successfully performs localization and mapping. Additionally, compared to three other single-source methods (DynaSLAM, MCL, MVSLAM), the proposed method consistently outperforms, demonstrating its superior adaptability and effectiveness.

LightDepth: A resource efficient depth estimation approach for dealing with ground truth sparsity via curriculum learning

Accurate depth estimation from monocular images is critical for various applications such as robotics, augmented reality, and autonomous navigation. However, achieving high accuracy while maintaining computational efficiency is a major challenge, particularly for resource-constrained devices. In this paper, we present LightDepth, an approach that leverages curriculum learning to estimate depth efficiently while taking into account resource constraints. It modifies the ground truth sparse depth maps from the KITTI dataset by resizing them to 31 extents during training to reduce sparsity and control complexity. The resulting model achieves comparable accuracy to state-of-the-art large models while outperforming them in response time by 71%. Our approach outperforms resource-efficient models regarding depth accuracy (measured by RMSE), achieving a 56% improvement. LightDepth is designed to be fast and resource-efficient, making it suitable for deployment in resource-constrained devices. It also balances the trade-off between accuracy and resource efficiency. All codes are available online at https://github.com/fatemehkarimii/lightdepth.

Incremental Learning for LiDAR Attack Recognition Framework in Intelligent Driving Using Gaussian Processes

The perception system plays a crucial role by integrating LiDAR and various sensors to perform localization and object detection, which ensures the security of intelligent driving. However, existing research indicates that LiDAR is vulnerable to sensor attacks, which lead to inappropriate driving strategies and need effective attack recognition methods. Previous LiDAR attack recognition methods rely on fixed anomaly thresholds obtained from depth map data distributions in specific scenarios as static anomaly boundaries, which lead to reduced accuracy, increased false alarm rates, and a lack of performance stability. To address these problems, we propose an adaptive LiDAR attack recognition framework capable of adjusting to different driving scenarios. This framework initially models the perception system by integrating the vehicle dynamics model and object tracking algorithms to extract data features, subsequently employing Gaussian Processes for the probabilistic modeling of these features. Finally, the framework employs sparsification computing techniques and a sliding window strategy to continuously update the Gaussian Process model with window data, which achieves incremental learning that generates uncertainty estimates as dynamic anomaly boundaries to recognize attacks. The performance of the proposed framework has been evaluated extensively using the real-world KITTI dataset covering four driving scenarios. Compared to previous methods, our framework achieves a 100% accuracy rate and a 0% false positive rate in the localization system, and an average increase of 3.43% in detection accuracy in the detection system across the four scenarios, which demonstrates superior adaptive capabilities.