KITTI Data Research Articles

Exploring augmentation strategies in mixed reality for autonomous driving with depth cameras

One significant challenge in improving autonomous driving algorithms is the lack of diverse real-world data. Moreover, transferring models from simulation to reality faces the reality gap problem. This study addresses this issue by developing an augmentation technique for mixed-reality environments, aimed at improving the testing and training of autonomous vehicles. Tested offline, it lays the groundwork for future online applications. The methodology focuses on creating virtual depth images using a virtual camera and applying an augmentation strategy to the KITTI data set. This creates a mixed-reality representation by combining virtual and real depth maps, leveraging depth information in the fusion process. The outcomes of this process are notably effective, achieving a balance between virtual and real-world aspects. This fusion method adeptly combines elements from both environments, maintaining the quality of the images. The novel contributions of this work include a detailed augmentation strategy that seamlessly integrates virtual objects into real depth maps, accounting for occlusions and ensuring realistic depth representations. In addition, this work demonstrates the feasibility of generating a large data set using the proposed method, significantly expanding the available data for training autonomous driving models. The use of metrics such as SSIM, peak signal-to-noise ratio (PSNR), and MAE, alongside object detection models such as Faster RCNN, provides a complete evaluation of both quantitative and qualitative aspects. The results demonstrate the quality of the augmented images, setting a foundation for potential online applications. The proposed strategy enables the generation of larger data set and facilitates safe, effective training in scenarios considered too risky or challenging to simulate accurately.

Fast Motion State Estimation Based on Point Cloud by Combing Deep Learning and Spatio-Temporal Constraints

Moving objects in the environment have a higher priority and more challenges in growing domains like unmanned vehicles and intelligent robotics. Estimating the motion state of objects based on point clouds in outdoor scenarios is currently a challenging area of research. This is due to factors such as limited temporal information, large volumes of data, extended network processing times, and the ego-motion. The number of points in a point cloud frame is typically 60,000–120,000 points, but most current motion state estimation methods for point clouds only downsample to a few thousand points for fast processing. The downsampling step will lead to the loss of scene information, which means these methods are far from being used in practical applications. Thus, this paper proposes a motion state estimation method that combines spatio-temporal constraints and deep learning. It starts by estimating and compensating the ego-motion of multi-frame point cloud data and mapping multi-frame data to a unified coordinate system; then the point cloud motion segmentation model on the multi-frame point cloud is proposed for motion object segmentation. Finally, spatio-temporal constraints are utilized to correlate the moving object at different moments and estimate the motion vectors. Experiments on KITTI, nuScenes, and real captured data show that the proposed method has good results, with an average vector deviation of only 0.036 m and 0.043 m in KITTI and nuScenes under a processing time of about 80 ms. The EPE3D error under the KITTI data is only 0.076 m, which proves the effectiveness of the method.

Improved Pedestrian Vehicle Detection for Small Objects Based on Attention Mechanism

Abstract This study aims to solve the low detection accuracy and susceptibility to false detection and omission in pedestrian and vehicle detection by proposing an improved YOLOv5s algorithm. Firstly, a small target detection module is added to better acquire and determine the information of pedestrians from long-range vehicles. Secondly, the multi-scale channel attention CBAM attention module is added, and the dual attention mechanism is not only flexible and convenient, but also improves the computational efficiency. Finally, the MPDIoU loss function based on minimum point distance is introduced to replace the original GIoU loss function, and this change not only enhances the regression accuracy of the model. At the same time, the convergence speed of the model is accelerated. KITTI data set was used for experiments, and the experimental results showed that the average accuracy of the model trained by the improved YOLOv5s algorithm on the data set reached 84.9%, which was 3.7% higher than that of the original YOLOv5s algorithm. It is verified that the model is suitable for high accuracy of pedestrian and vehicle recognition in complex environments, and has high value for promotion.

Detection of pedestrians and vehicles on foggy roads based on improved YOLOv7

Abstract Nowadays, object detection technology plays an indispensable role in the direction of automatic vehicle driving. However, inclement weather, such as fog and haze, poses severe challenges to the ability to understand visual perception and scene understanding of automatic driving technology, making it extremely difficult. Therefore, this paper proposes an improved YOLOv7 algorithm. Firstly, Gaussian blur and Gaussian noise are used to process the original data set, reducing the sensitivity of our model to image information, improving the ability to adapt to changing environments and tasks, and enhancing the generalization performance. Then, using the SIOU loss function, we optimize the algorithm to accelerate the convergence speed while considering the vector angle problem. Add Coordinate Attention integrates accurate position information into the coding network, optimizes the recognition accuracy of the model for a specific position, significantly improves its overall feature extraction ability, and captures key features more accurately, thereby improving the overall performance, capturing key information, and reducing the possibility of missed detection. The introduction of CoordConv into ELAN in the backbone allows for a more sensitive perception of the position of objects during image processing, thus improving the accuracy of its recognition and positioning. The RepGhostNet structure is introduced into the neck to realize the implicit reuse of features. According to the experiments in this paper, our improved YOLOv7 algorithm achieves 87.3% map in the RTTS data set and partial synthetic fog KITTI data set, which improves by 3.9% based on the original model.

A Vehicle Monocular Ranging Method Based on Camera Attitude Estimation and Distance Estimation Networks

A monocular ranging method for forward vehicles in intelligent driving is proposed. This method measures vehicle distance more accurately under the condition of a single camera and can estimate camera attitude in real-time. For the estimation of camera pitch and yaw angles, it is achieved using road vanishing points. The images collected by the camera are sequentially processed through the Roberts operator amplitude calculation, feature point extraction, feature line segment generation, road vanishing point voting, and estimation of camera attitude to obtain pitch and yaw angles. A distance estimation network was designed, which is divided into multiple levels based on image size and incorporates image feature, integrating vehicle grounding point, and vehicle width information, effectively improving ranging accuracy. Finally, validation was conducted on KITTI data, with a relative error (AbsRel) of 8.3%. Additionally, the TuSimple dataset and continuous driving scenarios were also validated, resulting in improved performance compared to previous algorithms.

A study on 3D LiDAR-based point cloud object detection using an enhanced PointPillars network

Abstract The PointPillar target detection algorithm is a mainstream 3D lidar point cloud target detection algorithm that has a fast response speed but low detection accuracy. Addressing the problem of the low detection accuracy of the PointPillar target detection network, we propose an improved PointPillar target detection algorithm that integrates an attention mechanism. The algorithm first introduces the attention mechanism and strengthens the feature extraction module based on PointPillar to realize the amplification of the local information in the three scale feature maps and to better extract the more important feature information. Then, our algorithm adds an anchor free type detector head to further optimize the detector head module. The experimental results show that the optimized PointPillar target detection algorithm has achieved good test results in the KITTI data set. Under medium difficulty, the AOS mode mAP reaches 79.76%, the 3D mode mAP reaches 82.03%, and the BEV mode mAP reaches 82.30%. Compared with that of other point cloud target detection algorithms, the detection accuracy of our algorithm is improved by approximately 10%.

A Robust Monocular and Binocular Visual Ranging Fusion Method Based on an Adaptive UKF.

Visual ranging technology holds great promise in various fields such as unmanned driving and robot navigation. However, complex dynamic environments pose significant challenges to its accuracy and robustness. Existing monocular visual ranging methods are susceptible to scale uncertainty, while binocular visual ranging is sensitive to changes in lighting and texture. To overcome the limitations of single visual ranging, this paper proposes a fusion method for monocular and binocular visual ranging based on an adaptive Unscented Kalman Filter (AUKF). The proposed method first utilizes a monocular camera to estimate the initial distance based on the pixel size, and then employs the triangulation principle with a binocular camera to obtain accurate depth. Building upon this foundation, a probabilistic fusion framework is constructed to dynamically fuse monocular and binocular ranging using the AUKF. The AUKF employs nonlinear recursive filtering to estimate the optimal distance and its uncertainty, and introduces an adaptive noise-adjustment mechanism to dynamically update the observation noise based on fusion residuals, thus suppressing outlier interference. Additionally, an adaptive fusion strategy based on depth hypothesis propagation is designed to autonomously adjust the noise prior of the AUKF by combining current environmental features and historical measurement information, further enhancing the algorithm's adaptability to complex scenes. To validate the effectiveness of the proposed method, comprehensive evaluations were conducted on large-scale public datasets such as KITTI and complex scene data collected in real-world scenarios. The quantitative results demonstrate that the fusion method significantly improves the overall accuracy and stability of visual ranging, reducing the average relative error within an 8 m range by 43.1% and 40.9% compared to monocular and binocular ranging, respectively. Compared to traditional methods, the proposed method significantly enhances ranging accuracy and exhibits stronger robustness against factors such as lighting changes and dynamic targets. The sensitivity analysis further confirmed the effectiveness of the AUKF framework and adaptive noise strategy. In summary, the proposed fusion method effectively combines the advantages of monocular and binocular vision, significantly expanding the application range of visual ranging technology in intelligent driving, robotics, and other fields while ensuring accuracy, robustness, and real-time performance.

Lightweight environment sensing algorithm for intelligent driving based on improved YOLOv7

AbstractAccurately and quickly detecting obstacles ahead is a prerequisite for intelligent driving. The combined detection scheme of light detection and ranging (LiDAR) and the camera is far more capable of coping with complex road conditions than a single sensor. However, immediately afterward, ensuring the real‐time performance of the sensing algorithms through a significantly increased amount of computation has become a new challenge. For this purpose, the paper introduces an improved dynamic obstacle detection algorithm based on YOLOv7 (You Only Look Once version 7) to overcome the drawbacks of slow and unstable detection of traditional methods. Concretely, Mobilenetv3 supplants the backbone network utilized in the original YOLOv7 architecture, thereby achieving a reduction in computational overhead. It integrates a specialized layer for the detection of small‐scale targets and incorporates a convolutional block attention module to enhance detection efficacy for diminutive obstacles. Furthermore, the framework adopts the Efficient Intersection over Union Loss function, which is specifically designed to mitigate the issue of mutual occlusion among detected objects. On a dataset consisting of 27,362 labelled KITTI data samples, the improved YOLOv7 algorithm achieves 92.6% mean average precision and 82 frames per second, which reduces the Model_size by 85.9% and loses only 1.5% accuracy compared with the traditional YOLOv7 algorithm. In addition, this paper builds a virtual scene to test the improved algorithm and fuses LiDAR and camera data. Experimental results conducted on a test vehicle equipped with a camera and LiDAR sensor demonstrate the effectiveness and significant performance of the method. The improved obstacle detection algorithm proposed in this research can significantly reduce the computational cost of the environment perception task, meet the requirements of real‐world applications, and is crucial for achieving safer and smarter driving.

RMS-FlowNet++: Efficient and Robust Multi-scale Scene Flow Estimation for Large-Scale Point Clouds

The proposed RMS-FlowNet++ is a novel end-to-end learning-based architecture for accurate and efficient scene flow estimation that can operate on high-density point clouds. For hierarchical scene flow estimation, existing methods rely on expensive Farthest-Point-Sampling (FPS) to sample the scenes, must find large correspondence sets across the consecutive frames and/or must search for correspondences at a full input resolution. While this can improve the accuracy, it reduces the overall efficiency of these methods and limits their ability to handle large numbers of points due to memory requirements. In contrast to these methods, our architecture is based on an efficient design for hierarchical prediction of multi-scale scene flow. To this end, we develop a special flow embedding block that has two advantages over the current methods: First, a smaller correspondence set is used, and second, the use of Random-Sampling (RS) is possible. In addition, our architecture does not need to search for correspondences at a full input resolution. Exhibiting high accuracy, our RMS-FlowNet++ provides a faster prediction than state-of-the-art methods, avoids high memory requirements and enables efficient scene flow on dense point clouds of more than 250K points at once. Our comprehensive experiments verify the accuracy of RMS-FlowNet++ on the established FlyingThings3D data set with different point cloud densities and validate our design choices. Furthermore, we demonstrate that our model has a competitive ability to generalize to the real-world scenes of the KITTI data set without fine-tuning.

Learning‐based monocular visual‐inertial odometry with SE2(3) $S{E}_{2}(3)$‐EKF

AbstractLearning‐based visual odometry (VO) becomes popular as it achieves a remarkable performance without manually crafted image processing and burdensome calibration. Meanwhile, the inertial navigation can provide a localization solution to assist VO when the VO produces poor state estimation under challenging visual conditions. Therefore, the combination of learning‐based technique and classical state estimation method can further improve the performance of pose estimation. In this paper, we propose a learning‐based visual‐inertial odometry (VIO) algorithm, which consists of an end‐to‐end VO network and an ‐Extended Kalman Filter (EKF). The VO network mainly combines a convolutional neural network with a recurrent neural network, taking advantage of two consecutive monocular images to produce relative pose estimation with associated uncertainties. The ‐EKF, which is proved to overcome the inconsistency issues of VIO, propagates inertial measurement unit kinematics‐based states, and fuses relative measurements and uncertainties from the VO network in its update step. The extensive experimental results on the KITTI data set and the EuRoC data set demonstrate the superior performance of the proposed method compared to other related methods.

SiLK-SLAM: accurate, robust and versatile visual SLAM with simple learned keypoints

PurposeWeak repeatability is observed in handcrafted keypoints, leading to tracking failures in visual simultaneous localization and mapping (SLAM) systems under challenging scenarios such as illumination change, rapid rotation and large angle of view variation. In contrast, learning-based keypoints exhibit higher repetition but entail considerable computational costs. This paper proposes an innovative algorithm for keypoint extraction, aiming to strike an equilibrium between precision and efficiency. This paper aims to attain accurate, robust and versatile visual localization in scenes of formidable complexity.Design/methodology/approachSiLK-SLAM initially refines the cutting-edge learning-based extractor, SiLK, and introduces an innovative postprocessing algorithm for keypoint homogenization and operational efficiency. Furthermore, SiLK-SLAM devises a reliable relocalization strategy called PCPnP, leveraging progressive and consistent sampling, thereby bolstering its robustness.FindingsEmpirical evaluations conducted on TUM, KITTI and EuRoC data sets substantiate SiLK-SLAM’s superior localization accuracy compared to ORB-SLAM3 and other methods. Compared to ORB-SLAM3, SiLK-SLAM demonstrates an enhancement in localization accuracy even by 70.99%, 87.20% and 85.27% across the three data sets. The relocalization experiments demonstrate SiLK-SLAM’s capability in producing precise and repeatable keypoints, showcasing its robustness in challenging environments.Originality/valueThe SiLK-SLAM achieves exceedingly elevated localization accuracy and resilience in formidable scenarios, holding paramount importance in enhancing the autonomy of robots navigating intricate environments. Code is available at https://github.com/Pepper-FlavoredChewingGum/SiLK-SLAM.

Transformer based 3D semantic segmentation of urban bicycle infrastructure

ABSTRACT This work investigates Point Cloud Semantic Segmentation (PCSS) to detect bicycle paths from 3D point cloud data of bicycle-mounted LiDARs. For the task of PCSS, an existing Convolutional Neural Network architecture (CNN) is first pre-trained on the Semantic KITTI data set, an important data set for PCSS in LiDAR scans of driving situations. Furthermore, a new, semantically-labelled 3D LiDAR data set, the Salzburg Bicycle LiDAR Data Set (SBLD), is presented, which consists of 16,008 point clouds recorded by a ROS2-enabled sensor bicycle with five 3D LiDARs covering every direction. After fine-tuning the CNN on the SBLD train set, the segmentation performance is evaluated on the SBLD test set. The CNN shows promising results in recognising bike paths, vegetation, terrain and buildings in the SBLD. To further push the segmentation performance, the existing CNN is enhanced with self-attention blocks. The evaluation of this enhanced architecture shows a performance gain of 2.79% points compared to the original architecture. The proposed segmentation approach shows practical potential for detecting bike paths. The SBLD is provided to the scientific community to further drive research in the field of sensor bicycles.

Research and Implementation of 3D Object Detection Based on Autonomous Driving Scenarios

Perception function, as an important part of autonomous driving, ensures the safety and intelligence of driving. We collect the surrounding environment of driving through hardware sensors, providing a basis for subsequent decision-making of autonomous driving. In recent years, deep learning has made breakthrough progress in object detection. Based on this, this paper uses lidar point cloud data, combined with deep learning theory and method, to carry out three-dimensional target detection tasks around lidar point cloud data, and carries out theoretical analysis, method verification, and result analysis. A 3D object detection method based on LiDAR point cloud data is proposed. The backbone network of the network model corresponding to the method is VoxelNet’s backbone network. After outputting the feature matrix, sparse point clouds are supplemented with point clouds, and then the feature matrix is decoded to generate candidate boxes. The model used in this paper on the KITTI data set can effectively solve the problem of 3D target detection in the process of autonomous driving and has good performance.

Monocular 3D Object Detection Using Depth Fusion

It is an important task to estimate a 3D bounding box from monocular images for autonomous driving. However, the monocular pictures do not have distance information, so it is difficult to acquire accurate results. For the sake of solving the trouble of low accuracy of the monocular image in 3D target detection because of lacking distance information, an improved monocular three-dimensional target detection algorithm based on GUPNet and neural network was proposed to promote the precision of target detection. First, based on the geometric method proposed by GUPNet, the depth, and uncertainty are obtained by direct regression using a neural network. According to the difference in the accuracy of the two methods, a parameter α was introduced, and their depth scores are obtained from the uncertainty. According to the depth score and parameter α, the depth obtained by the two methods is fused to get the final depth. Test results prove that the proposed algorithm promotes average detection precision of KITTI data set in simple, medium, and difficult cases.

A LiDAR-intensity SLAM and loop closure detection method using an intensity cylindrical-projection shape context descriptor

LiDAR Simultaneous Localization and Mapping (SLAM) is able to map unknown scene while online estimating the LiDAR’s pose. Traditional LiDAR SLAM frameworks mainly rely on geometric features in the environments, but ignore LiDAR intensity information, leading to low accuracy in scenes with sparse environmental features. This paper proposes a novel intensity-based LiDAR-SLAM framework. In the front-end, the geometric and intensity features are extracted to match the two consecutive scans. To keep time efficiency, a self-adaptive feature selection strategy is proposed to select geometric and intensity features and the 6 DOF transformation from the corresponding features are weighted fused for odometry estimation. To improve the effectiveness of loop closure detection (LCD), we propose a novel intensity cylindrical-projection shape context (ICPSC) descriptor and a row-column similarity estimation based on ICPSC. To guarantee the accuracy of LCD, a double-value loop candidate verification strategy is employed. We have conducted comprehensive experimental verification of our LiDAR SLAM framework, including both the public KITTI datasets and data from our platform that involve multiple scenes. The results display that our framework achieves an averaged improvement of about 3 m in scenes with sparse environmental features in terms of on-line localization relative to LeGO-LOAM, while only increases maximum 9 ms in terms of time cost.

Fast Context-Awareness Encoder for LiDAR Point Semantic Segmentation

A LiDAR sensor is a valuable tool for environmental perception as it can generate 3D point cloud data with reflectivity and position information by reflecting laser beams. However, it cannot provide the meaning of each point cloud cluster, so many studies focus on identifying semantic information about point clouds. This paper explores point cloud segmentation and presents a lightweight convolutional network called Fast Context-Awareness Encoder (FCAE), which can obtain semantic information about the point cloud cluster at different levels. The surrounding features of points are extracted as local features through the local context awareness network, then combined with global features, which are highly abstracted from the local features, to obtain more accurate semantic segmentation of the discrete points in space. The proposed algorithm has been compared and verified against other semantic KITTI data algorithms and has achieved state-of-the-art performance. Due to its ability to note fine-grained features on the z-axis in space, the algorithm shows higher prediction accuracy for certain types of objects. Moreover, the training and validation time is short, and the algorithm can meet high real-time requirements for 3D perception tasks.

An Object Detection Method Based on Feature Uncertainty Domain Adaptation for Autonomous Driving

The environment perception algorithm in autonomous driving is trained in the source domain, leading to domain drift and reduced detection accuracy in the target domain due to shifts in background feature distribution. To address this issue, a domain adaptive object detection algorithm based on feature uncertainty is proposed, which can improve the detection performance of object detection algorithms in unlabeled data. Firstly, a local alignment module based on channel information is proposed, which can obtain the model’s uncertainty about different domain data based on the feature channels obtained through the feature extraction network, achieving adaptive dynamic local alignment. Secondly, an instance-level alignment module guided by local feature uncertainty is proposed, which can obtain the corresponding instance-level uncertainty through ROI mapping. To improve the domain invariance of bounding box regression, a multi-class, multi-regression instance-level uncertainty alignment module is proposed, which can achieve spatial decoupling of classification and regression tasks, further improving the model’s domain adaptive ability. Finally, the effectiveness of the proposed algorithm is validated on Cityscapes, KITTI, and real vehicle data.

Comparative Analysis of SLAM Algorithms for Mechanical LiDAR and Solid-State LiDAR

With the advancement of light detection and ranging (LiDAR) technology in recent years, various new types of LiDAR have emerged, and the price of LiDAR equipment has gradually decreased. At present, low-cost solid-state LiDARs are gradually occupying the market. To evaluate the performance of two LIDARs for simultaneous localization and mapping. This study investigated the application of solid-state LiDAR and mechanical LiDAR in localization and mapping systems and comparatively analyzed their advantages and disadvantages. We selected some classic open-source algorithms [such as LiDAR odometry and mapping (A-LOAM)] to evaluate the performance of mechanical LiDAR and solid-state LiDAR in localization. The experimental data are adopted from some representative open-source data (such as KITTI data) and real data collected by Shenzhen University. The results showed that the localization accuracy of solid-state LiDAR was lower than that of mechanical LiDAR when the mobile robot moved to the corner and faced square to the wall at close range. Moreover, the localization accuracy of solid-state LiDAR was the same as or even higher than that of mechanical LiDAR when the mobile robots had small changes in the field of view (FOV) and the mobile robot moved along straight lines or other tracks.

Semantic stereo visual SLAM toward outdoor dynamic environments based on ORB-SLAM2

PurposeThe prerequisite for most traditional visual simultaneous localization and mapping (V-SLAM) algorithms is that most objects in the environment should be static or in low-speed locomotion. These algorithms rely on geometric information of the environment and restrict the application scenarios with dynamic objects. Semantic segmentation can be used to extract deep features from images to identify dynamic objects in the real world. Therefore, V-SLAM fused with semantic information can reduce the influence from dynamic objects and achieve higher accuracy. This paper aims to present a new semantic stereo V-SLAM method toward outdoor dynamic environments for more accurate pose estimation.Design/methodology/approachFirst, the Deeplabv3+ semantic segmentation model is adopted to recognize semantic information about dynamic objects in the outdoor scenes. Second, an approach that combines prior knowledge to determine the dynamic hierarchy of moveable objects is proposed, which depends on the pixel movement between frames. Finally, a semantic stereo V-SLAM based on ORB-SLAM2 to calculate accurate trajectory in dynamic environments is presented, which selects corresponding feature points on static regions and eliminates useless feature points on dynamic regions.FindingsThe proposed method is successfully verified on the public data set KITTI and ZED2 self-collected data set in the real world. The proposed V-SLAM system can extract the semantic information and track feature points steadily in dynamic environments. Absolute pose error and relative pose error are used to evaluate the feasibility of the proposed method. Experimental results show significant improvements in root mean square error and standard deviation error on both the KITTI data set and an unmanned aerial vehicle. That indicates this method can be effectively applied to outdoor environments.Originality/valueThe main contribution of this study is that a new semantic stereo V-SLAM method is proposed with greater robustness and stability, which reduces the impact of moving objects in dynamic scenes.

An optimised extreme learning machine (OELM) for simultaneous localisation and mapping in autonomous vehicles

Autonomous robots can navigate unexpected situations without human involvement. Multiple sensors for object recognition and mapping are used in autonomous driving, factory automation, and security service robots. Advances in motion planning allow autonomous cars to map their position and orientation. Vehicle position estimates may be off. SLAM algorithms map an unfamiliar environment and estimate a vehicle's position. We present a grey wolf optimised extreme learning machine (OELM) approach to leverage deep learning networks. It can build an environment map independently and help the system navigate autonomously. First, a CNN-GRU hybrid model for training Stereo and IMU sensor data is provided. ELM is then optimised for SLAM. The proposed method minimises posture estimate error for autonomous vehicles. Our method captures autonomous driving scenarios using KITTI data. IMU and stereo images collect data. Comparing the computed path to ground truth poses enhances accuracy and decreases error. Improved ELM has lower RMSE than prior SLAM. Data show that OELM-SLAM is more accurate than ELM.