Karlsruhe Institute Of Technology And Toyota Technological Institute Research Articles

Z-YOLOv8s-based approach for road object recognition in complex traffic scenarios

Object detection in road scenarios is crucial for intelligent transport systems and autonomous driving, but complex traffic conditions pose significant challenges. This paper introduces Z-You Only Look Once version 8 small (Z-YOLOv8s), designed to improve both accuracy and real-time efficiency under real-world uncertainties. By incorporating Revisiting Perspective Vision Transformer (RepViT) and C2f into the YOLOv8s framework, and integrating the Large Selective Kernel Network (LSKNet), the model enhances spatial feature extraction. Additionally, the YOLOv8s backbone is optimized with Space-to-Depth Convolution (SPD-Conv) for better small object detection. The Softpool-Spatial Pyramid Pooling Fast (SoftPool-SPPF) module ensures precise characteristic information preservation. Z-YOLOv8s improves mean average precision (mAP)@0.5 on the Berkeley Deep Drive 100 K (BDD100K) and Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) datasets by 7.3 % and 3.8 %, respectively. It also achieves accuracy increases of 5.7 % and 6.5 % in Average Precision (AP)-Small, and a real-time detection speed of 78.41 frames per second (FPS) on the BDD100K. Z-YOLOv8s balances detection precision and processing speed more effectively than other detectors, as demonstrated by experimental results and comparisons.

Monocular Depth Estimation Based on Dilated Convolutions and Feature Fusion

Depth estimation represents a prevalent research focus within the realm of computer vision. Existing depth estimation methodologies utilizing LiDAR (Light Detection and Ranging) technology typically obtain sparse depth data and are associated with elevated hardware expenses. Multi-view image-matching techniques necessitate prior knowledge of camera intrinsic parameters and frequently encounter challenges such as depth inconsistency, loss of details, and the blurring of edges. To tackle these challenges, the present study introduces a monocular depth estimation approach based on an end-to-end convolutional neural network. Specifically, a DNET backbone has been developed, incorporating dilated convolution and feature fusion mechanisms within the network architecture. By integrating semantic information from various receptive fields and levels, the model’s capacity for feature extraction is augmented, thereby enhancing its sensitivity to nuanced depth variations within the image. Furthermore, we introduce a loss function optimization algorithm specifically designed to address class imbalance, thereby enhancing the overall predictive accuracy of the model. Training and validation conducted on the NYU Depth-v2 (New York University Depth Dataset Version 2) and KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) datasets demonstrate that our approach outperforms other algorithms in terms of various evaluation metrics.

Curvature Scale Space LiDAR Odometry And Mapping (LOAM)

The LiDAR Odometry and Mapping (LOAM) algorithm ranks in second place in the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI), Visual Odometry/SLAM Evaluations. It utilizes a feature extraction algorithm based on the evaluation of the curvature of points under test, to produce estimated smooth and non-smooth regions within typically laser based Point Cloud Data (PCD). This feature extractor (FE) however, does not take into account PCD spatial or detection uncertainty, which can result in the divergence of the LOAM algorithm. Therefore, this article proposes the use of the Curvature Scale Space (CSS) algorithm as a replacement for LOAM’s current feature extractor. It justifies the substitution, based on the CSS algorithm’s similar computational complexity but improved feature detection repeatability. LOAM’s current feature extractor and the proposed CSS feature extractor are tested and compared with simulated and real data, including the KITTI odometry-laser data set. Additionally, a recent deep learning based LiDAR Odometry (LO) algorithm, the Convolutional Auto-Encoder (CAE)-LO algorithm, will also be compared, using this data set, in terms of its computational speed and performance. Performance comparisons are made based on the Absolute Trajectory Error (ATE) and Cardinalized Optimal Linear Assignment (COLA) metrics. Based on these metrics, the comparisons show significant improvements of the LOAM algorithm with the CSS feature extractor compared with the benchmark versions.

3D VAE Video Prediction Model with Kullback Leibler Loss Enhancement

The Video Prediction (VP) models adopted many techniques to build suitable structures to extract the spatiotemporal features and predict the future frame. The VP techniques extracted the spatial and temporal features in separated models and then fused both features to generate the future frame. However, these architectures suffered from the design complexity and time for prediction required. So, many efforts introduced VP based on decreasing design complexity and producing good results. This study produces the VP model based on a Three-Dimensional Variational Auto Encoder (3D VAE). The proposed model builds all layers depending on 3D convolutional layers. This leads to better extraction of spatiotemporal information and decreases the design complexity. Second, the Kullback Leibler Loss (KL Loss) is enhanced by a 3D sampling stage which allows to calculation of the 3D latent loss. This helps to extract the better and proper spatiotemporal latent variable from the 3D Encoder. The 3D sampling represents a good regularizer in the model. The proposed model outperforms in terms of SNR=34.8673, SSIM= 0.9616 which applied to Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) and Caltech pedestrian datasets and records 5.2 M parameters.

Distance Transform Pooling Neural Network for LiDAR Depth Completion.

Recovering dense depth maps from sparse depth sensors, such as LiDAR, is a recently proposed task with many computer vision and robotics applications. Previous works have identified input sparsity as the key challenge of this task. To solve the sparsity challenge, we propose a recurrent distance transform pooling (DTP) module that aggregates multi-level nearby information prior to the backbone neural network. The intuition of this module is originated from the observation that most pixels within the receptive field of the network are zero. This indicates a deep and heavy network structure has to be used to enlarge the receptive field aiming at capturing enough useful information as most processed signals are uninformative zeros. Our recurrent DTP module can fill in empty pixels with the nearest value in a local patch and recurrently transform distance to reach farther nearest points. The output of the proposed DTP module is a collection of multi-level semi-dense depth maps from original sparse to almost full. Processing this collection of semi-dense depth maps alleviates the network from the input sparsity, which helps a lightweight simplified ResNet-18 with 1M parameters achieve state-of-the-art performance on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) depth completion benchmark with LiDAR only. Besides the sparsity, the input LiDAR map also contains some incorrect values due to the sensor error. Thus, we further enhance the DTP with an error correction (EC) module to avoid the spreading of the incorrect input values. At last, we discuss the benefit of only using LiDAR for nighttime driving and the potential extension of the proposed method for sensor fusion and the indoor scenario. The code has been released online at https://github.com/placeforyiming/DistanceTransform-DepthCompletion.

INS/LIDAR/Stereo SLAM Integration for Precision Navigation in GNSS-Denied Environments.

Traditionally, navigation systems have relied solely on global navigation satellite system (GNSS)/inertial navigation system (INS) integration. When a temporal loss of GNSS signal lock is encountered, these systems would rely on INS, which can sustain short bursts of outages, albeit drift significantly in prolonged outages. In this study, an extended Kalman filter (EKF) is proposed to develop an integrated INS/LiDAR/Stereo simultaneous localization and mapping (SLAM) navigation system. The first update stage of the filter integrates the INS with the LiDAR, after which the resultant navigation solution is integrated with the stereo SLAM solution, which yields the final integrated navigation solution. The system was tested for different driving scenarios in urban and rural environments using the raw Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset in the complete absence of the GNSS signal. In addition, the selected KITTI drives covered low and high driving speeds in feature-rich and feature-poor environments. It is shown that the proposed INS/LiDAR/Stereo SLAM navigation system yielded better position estimations in comparison to using the INS without any assistance from onboard sensors. The accuracy improvement was expressed as a reduction of the root-mean-square error (RMSE) by 83% and 82% in the horizontal and up directions, respectively. In addition, the proposed system outperformed the positioning accuracy of some of the state-of-the-art algorithms.

Monocular Distance Estimation-based Approach using Deep Artificial Neural Network

Those in authority are evaluating the test evaluation for threat assessments currently in place. Since people often depend on their feelings and moods, this may create inequality. Therefore, this study suggested applying deep learning for Autonomous Emergency Steering (AES) and Autonomous Emergency Braking (AEB) assessments in the safety rating protocol. The suggested method for the test in situation-based threat assessments is a monocular distance estimation-based approach. The camera's objective is to make it simple to conduct assessments using only an onboard dash camera. This study proposes a method based on a monocular distance estimation-based approach for test methodology in the situational-based threat assessments using deep learning for the AES system to complement the AEB system for active safety features. Then, the accuracy of the distance estimation models has validated with the ground truth distances from the KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) dataset. Thus, the output of this study can contribute to the methodological base for further understanding of drivers the following behaviour with a long-term goal of reducing rear-end collisions.

GAF-RCNN: Grid attention fusion 3D object detection from point cloud

Background: Due to the refinement of region of the interests (RoIs), two-stage 3D detection algorithms can usually obtain better performance compared with most single-stage detectors. However, most two-stage methods adopt feature connection, to aggregate the grid point features using multi-scale RoI pooling in the second stage. This connection mode does not consider the correlation between multi-scale grid features. Methods: In the first stage, we employ 3D sparse convolution and 2D convolution to fully extract rich semantic features. Then, a small number of coarse RoIs are predicted based region proposal network (RPN) on generated bird’s eye view (BEV) map. After that, we adopt voxel RoI-pooling strategy to aggregate the neighborhood nonempty voxel features of each grid point in RoI in the last two layers of 3D sparse convolution. In this way, we obtain two aggregated features from 3D sparse voxel space for each grid point. Next, we design an attention feature fusion module. This module includes a local and a global attention layer, which can fully integrate the grid point features from different voxel layers. Results: We carry out relevant experiments on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset. The average precisions of our proposed method are 88.21%, 81.51%, 77.07% on three difficulty levels (easy, moderate, and hard, respectively) for 3D detection, and 92.30%, 90.19%, 86.00% on three difficulty levels (easy, moderate, and hard, respectively) for BEV detection. Conclusions: In this paper, we propose a novel two-stage 3D detection algorithm named Grid Attention Fusion Region-based Convolutional Neural Network (GAF-RCNN) from point cloud. Because we integrate multi-scale RoI grid features with attention mechanism in the refinement stage, different multi-scale features can be better correlated, achieving a competitive level compared with other well tested detection algorithms. This 3D object detection has important implications for robot and cobot technology.

MFF-Net: Multimodal Feature Fusion Network for 3D Object Detection

In complex traffic environment scenarios, it is very important for autonomous vehicles to accurately perceive the dynamic information of other vehicles around the vehicle in advance. The accuracy of 3D object detection will be affected by problems such as illumination changes, object occlusion, and object detection distance. To this purpose, we face these challenges by proposing a multimodal feature fusion network for 3D object detection (MFF-Net). In this research, this paper first uses the spatial transformation projection algorithm to map the image features into the feature space, so that the image features are in the same spatial dimension when fused with the point cloud features. Then, feature channel weighting is performed using an adaptive expression augmentation fusion network to enhance important network features, suppress useless features, and increase the directionality of the network to features. Finally, this paper increases the probability of false detection and missed detection in the non-maximum suppression algorithm by increasing the one-dimensional threshold. So far, this paper has constructed a complete 3D target detection network based on multimodal feature fusion. The experimental results show that the proposed achieves an average accuracy of 82.60% on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset, outperforming previous state-of-the-art multimodal fusion networks. In Easy, Moderate, and hard evaluation indicators, the accuracy rate of this paper reaches 90.96%, 81.46%, and 75.39%. This shows that the MFF-Net network has good performance in 3D object detection.

Vehicle Detection in Challenging Scenes Using CenterNet Based Approach

Contemporarily numerous analysts labored in the field of Vehicle detection which improves Intelligent Transport System (ITS) and reduces road accidents. The major obstacles in automatic detection of tiny vehicles are due to occlusion, environmental conditions, illumination, view angles and variation in size of objects. This research centers on tiny and partially occluded vehicle detection and identification in challenging scene specifically in crowed area. In this paper we present comprehensive methodology of tiny vehicle detection using Deep Neural Networks (DNN) namely CenterNet. Substantially DNN disregards objects that are small in size 5 pixels and more false positives likely to happen in crowded area. Primarily there are two categories of deep learning models single-step and two-step. A single forward pass model is the one in which detection is performed directly to possible location over dense sampling, wherein two-step models incorporated by Region proposals followed by object detection. We in this research scrutinize one-step State of the art (SOTA) model CenteNet as proposed recently with three different feature extractor ResNet-50, HourGlass-104 and ResNet-101 one by one. We train our model on challenging KITTI dataset which outperforms in comparison with SOTA single-step technique MSSD300* which depicts performance improvement by 20.2% mAP and SMOKE by with 13.2% mAP respectively. Effectiveness of CenterNet can be justified through the huge improved performance. The performance of our model is evaluated on KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) benchmark dataset with different backbones such as ResNet-50 gives 62.3% mAP ResNet-101 82.5% mAP, last but not the least HourGlass-104 outperforms with 98.2% mAP CenterNet-HourGlass-104 achieved high mAP among above mentioned feature extractors. We also compare our model with other SOTA techniques.

A Deep Learning Framework for Robust and Real-Time Taillight Detection Under Various Road Conditions

In this paper, we present a deep learning model for high-accuracy, high-speed detection of vehicle taillights in traffic. The model consists of three major modules: the lane detector, the car detector, and the taillight detector. Unlike most previously proposed algorithms where hand-coded schemes are used, we have adopted a data-driven approach. This data-driven scheme was implemented in both the car and taillight detection modules. First, we used an intricately designed lane detection module, then we adopted the Recurrent Rolling Convolution (RRC) architecture and tracking mechanism for detecting car boundaries. Subsequently, we used the same RRC architecture to extract the taillight regions of the detected cars. The lane detection and car detection modules improve both the speed and detection rate of the final taillight detection. The robustness of our model was verified using datasets from Sungkyunkwan University (SKKU) as well as the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI). Our model works well even in hostile conditions. It achieves detection rates as high as 99% in testing with the SKKU dataset. When using the KITTI 2D Object dataset, the model achieves a taillight detection rate of 86%. The model achieves 100% taillight detection rate on a certain, small subset of the KITTI Tracking dataset.

Self-Supervised Monocular Depth Estimation Using Hybrid Transformer Encoder

Depth estimation using monocular camera sensors is an important technique in computer vision. Supervised monocular depth estimation requires a lot of data acquired from depth sensors. However, acquiring depth data is an expensive task. We sometimes cannot acquire data due to the limitations of the sensor. View synthesis-based depth estimation research is a self-supervised learning method that does not require depth data supervision. Previous studies mainly use the convolutional neural network (CNN)-based networks in encoders. The CNN is suitable for extracting local features through convolution operation. Recent vision transformers (ViTs) are suitable for global feature extraction based on multiself-attention modules. In this article, we propose a hybrid network combining the CNN and ViT networks in self-supervised learning-based monocular depth estimation. We design an encoder–decoder structure that uses CNNs in the earlier stage of extracting local features and a ViT in the later stages of extracting global features. We evaluate the proposed network through various experiments based on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) and Cityscapes datasets. The results showed higher performance than previous studies and reduced parameters and computations. Codes and trained models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/fogfog2/manydepthformer</uri> .

Multiple Object Tracking in Robotic Applications: Trends and Challenges

The recent advancement in autonomous robotics is directed toward designing a reliable system that can detect and track multiple objects in the surrounding environment for navigation and guidance purposes. This paper aims to survey the recent development in this area and present the latest trends that tackle the challenges of multiple object tracking, such as heavy occlusion, dynamic background, and illumination changes. Our research includes Multiple Object Tracking (MOT) methods incorporating the multiple inputs that can be perceived from sensors such as cameras and Light Detection and Ranging (LIDAR). In addition, a summary of the tracking techniques, such as data association and occlusion handling, is detailed to define the general framework that the literature employs. We also provide an overview of the metrics and the most common benchmark datasets, including Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI), MOTChallenges, and University at Albany DEtection and TRACking (UA-DETRAC), that are used to train and evaluate the performance of MOT. At the end of this paper, we discuss the results gathered from the articles that introduced the methods. Based on our analysis, deep learning has introduced significant value to the MOT techniques in recent research, resulting in high accuracy while maintaining real-time processing.

Anomaly recognition method of perception system for autonomous vehicles based on distance metric

Environmental perception system is the premise of the safety and stability of the autonomous vehicle system. However, studies have shown that the on-board sensors included in the perception system are extremely vulnerable to external attacks and interference, leading to incorrect driving strategies and bringing great security threats. Aiming at the problem, this paper divides the vehicle-mounted sensors into a positioning group and an identification group according to their role in the perception system. Then, based on the information correlation between sensors in the same group and the information correlation of a single sensor on adjacent time series, the distance metric model between sensors in a group and the distance metric model for each sensor of this group on time series is established. And the normal distance intervals corresponding to the confidence interval are calculated respectively. According to the distance metric model between sensors, the authors can detect anomalies in the perception system in real time. Further, according to the distance metric model for each sensor on adjacent time series, the authors can identify anomaly sensors. The authors’ experimental results quantitatively show that the method achieves real-time anomaly recognition, and demonstrate the effectiveness and robustness of the method on the open-source Karlsruhe institute of technology and toyota technological institute (KITTI) dataset.

Monocular three-dimensional object detection using data augmentation and self-supervised learning in autonomous driving

Monocular three-dimensional (3D) object detection (OD) is an essential and challenging task in the domain of autonomous driving. Modern convolution neural network-based architectures for OD heavily rely on data augmentation (DA) and self-supervised learning (SSL). However, they have been relatively less explored for monocular 3D OD, especially in the field of autonomous driving. DAs for two-dimensional OD techniques do not directly extend to the 3D objects. Literature shows that this requires adaptation of the 3D geometry of the input scene and synthesis of new viewpoints. This requires accurate depth information of the scene which may not be available always. We propose augmentations for monocular 3D OD without creating view synthesis. The proposed method uses DA with SSL approach via multiobject labeling as the pretext task. We evaluate the proposed DA-SSL approach on RTM3D detection model (baseline), with and without the application of DA. The results demonstrate improvements between 2% and 3% in mAP 3D and 0.9% to 1.5% BEV scores using SSL over the baseline scores. We propose an inverse class frequency weighted (ICFW) mAP score that highlights improvements in detection for low-frequency classes in a class imbalanced datasets with long tails. We observe improvements in both ICFW mAP 3D and Bird’s Eye View (BEV) scores to take into account the class imbalance in the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) validation dataset. We achieve 4% to 5% increase in ICFW metrics with the pretext task.

3D object detection combining semantic and geometric features from point clouds

Background: 3D object detection based on point clouds in road scenes has attracted much attention recently. The voxel-based methods voxelize the scene to regular grids, which can be processed with the advanced feature learning frameworks based on convolutional layers for semantic feature learning. The point-based methods can extract the geometric feature of the point due to the coordinate reservations. The combination of the two is effective for 3D object detection. However, the current methods use a voxel-based detection head with anchors for classification and localization. Although the preset anchors cover the entire scene, it is not suitable for detection tasks with larger scenes and multiple categories of objects, due to the limitation of the voxel size. Additionally, the misalignment between the predicted confidence and proposals in the Regions of the Interest (ROI) selection bring obstacles to 3D object detection. Methods: We investigate the combination of voxel-based methods and point-based methods for 3D object detection. Additionally, a voxel-to-point module that captures semantic and geometric features is proposed in the paper. The voxel-to-point module is conducive to the detection of small-size objects and avoids the presets of anchors in the inference stage. Moreover, a confidence adjustment module with the center-boundary-aware confidence attention is proposed to solve the misalignment between the predicted confidence and proposals in the regions of the interest selection. Results: The proposed method has achieved state-of-the-art results for 3D object detection in the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) object detection dataset. Actually, as of September 19, 2021, our method ranked 1st in the 3D and Bird Eyes View (BEV) detection of cyclists tagged with difficulty level ‘easy’, and ranked 2nd in the 3D detection of cyclists tagged with ‘moderate’. Conclusions: We propose an end-to-end two-stage 3D object detector with voxel-to-point module and confidence adjustment module.

Residual 3-D Scene Flow Learning With Context-Aware Feature Extraction

Scene flow estimation is the task to predict the point-wise or pixel-wise 3-D displacement vector between two consecutive frames of point clouds or images, which has important application in fields such as service robots and autonomous driving. Although many previous works have explored greatly on scene flow estimation based on point clouds, there are two problems that have not been noticed or well solved before: 1) points of adjacent frames in repetitive patterns tend to be wrongly associated due to similar spatial structure in their neighborhoods and 2) scene flow between adjacent frames of point clouds with long-distance movement tends to be inaccurately estimated. To solve the first problem, a novel context-aware set convolution layer is proposed in this article to exploit contextual structure information of Euclidean space and learn soft aggregation weights for local point features. This design is inspired by human perception of contextual structure information during scene understanding with repetitive patterns. For the second problem, an explicit residual flow learning structure is proposed in the residual flow refinement layer to cope with long-distance movement. The experiments and ablation study on FlyingThings3D and Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) scene flow datasets demonstrate the effectiveness of each proposed component. The qualitative results show that the problems of ambiguous interframe association and long-distance movement estimation are well handled. Quantitative results on both FlyingThings3D and KITTI scene flow datasets show that the proposed method achieves state-of-the-art performance, surpassing all previous works to the best of our knowledge by at least 26.22% on FlyingThings3D dataset and 25.42% on KITTI scene flow dataset for 3-D endpoint error (EPE3D) metric. The source codes are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/IRMVLab/CARFlow</uri> .

Cooperative Visual Augmentation Algorithm of Intelligent Vehicle Based on Inter-Vehicle Image Fusion

In a connected vehicle environment based on vehicle-to-vehicle (V2V) technology, images from front and ego vehicles are fused to augment a driver’s or autonomous system’s visual field, which is helpful in avoiding road accidents by eliminating the blind point (the objects occluded by vehicles), especially tailgating in urban areas. Realizing multi-view image fusion is a tough problem without knowing the relative location of two sensors and the fusing object is occluded in some views. Therefore, we propose an image geometric projection model and a new fusion method between neighbor vehicles in a cooperative way. Based on a 3D inter-vehicle projection model, selected feature matching points are adopted to estimate the geometric transformation parameters. By adding deep information, our method also designs a new deep-affine transformation to realize fusing of inter-vehicle images. Experimental results on KIITI (Karlsruhe Institute of Technology and Toyota Technological Institute) datasets are shown to validate our algorithm. Compared with previous work, our method improves the IoU index by 2~3 times. This algorithm can effectively enhance the visual perception ability of intelligent vehicles, and it will help to promote the further development and improvement of computer vision technology in the field of cooperative perception.

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Scene flow tracks the 3D motion of each point in adjacent point clouds. It provides fundamental 3D motion perception for autonomous driving and server robot. Although red green blue depth (RGBD) camera or light detection and ranging (LiDAR) capture discrete 3D points in space, the objects and motions usually are continuous in the macroworld. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the generative adversarial networks (GAN) is utilized to self‐learn 3D scene flow without ground truth. The fake point cloud is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset show that our method realizes promising results. Just as human, the proposed method can identify the similar local structures of two adjacent frames even without knowing the ground truth scene flow. Then, the local correspondence can be correctly estimated, and further the scene flow is correctly estimated. An interactive preprint version of the article can be found here:https://www.authorea.com/doi/full/10.22541/au.163335790.03073492.

Lane Lines Detection under Complex Environment by Fusion of Detection and Prediction Models

The lane lines’ length, width, and direction are very regular, serialized, and structurally associated, which are not easily affected by the environment. To enhance lane detection in a complicated environment, an approach combines visual information with the spatial distribution. Firstly, the grid density of the target detection algorithm YOLOv3 (you only look once V3) is improved from S×S to S×2S, aiming at the particular points in the bird’s-eye view where the lane lines had different densities in the horizontal and vertical directions. The obtained YOLOv3 (S×2S) is more suitable for detecting objects with small and large aspect ratios. It also identifies image features along with balances the detection speed and accuracy. Secondly, based on a bi-directional gated recurrent unit (BGRU), a new lane line prediction model BGRU-Lane (BGRU-L) based on the distribution of lane lines is proposed using the characteristic of lane line serialization and structural correlation. Finally, Dempster-Shafer (D-S) algorithm based on confidence was used to integrate the results of YOLOv3 (S×2S) and BGRU-L to improve the lane line detection ability under complex environments. The experiment was carried out on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset, while Euro Truck Simulator 2 (ETS2) is used as a supplement dataset. After fusing YOLOv3 (S×2S) and BGRU-L models in the D-S model, the detection results have high accuracy in a complex environment by 90.28 mAP. The detection speed is 40.20fps, which enables real-time detection.