Lane Detection System Research Articles

Implementation of an Autonomous Overtaking System Based on Time to Lane Crossing Estimation and Model Predictive Control

According to statistics, the majority of accidents are attributed to driver negligence, especially when a driver intends to lane change or to overtake another vehicle, which is most likely to cause accidents. In addition, overtaking is one of the most difficult and complex functions for the development of autonomous driving technologies because of the dynamic and complicated task involved in the control strategy and electronic control systems, such as steering, throttle, and brake control. This paper proposes a safe overtaking maneuver procedure for an autonomous vehicle based on time to lane crossing (TLC) estimation and the model predictive control scheme. As overtaking is one of the most complex maneuvers that require both lane keeping and lane changing, a vision-based lane-detection system is used to estimate TLC to make a timely and accurate decision about whether to overtake or remain within the lane. Next, to maintain the minimal safe distance and to choose the best timing to overtake, the successive linearization-based model predictive control is employed to derive an optimal vehicle controller, such as throttle, brake, and steering angle control. Simultaneously, it can make certain that the longitudinal acceleration and steering velocity are maintained under constraints to maintain driving safety. Finally, the proposed system is validated by real-world experiments performed on a prototype electric golf cart and executed in real-time on the automotive embedded hardware with limited computational power. In addition, communication between the sensors and actuators as well as the vehicle control unit (VCU) are based on the controller area network (CAN) bus to realize vehicle control and data collection. The experiments demonstrate the ability of the proposed overtaking decision and control strategy to handle a variety of driving scenarios, including a lane-following function when a relative yaw angle exists and an overtaking function when the approaching vehicle has a different lateral velocity.

SUPER: A Novel Lane Detection System

AI-based lane detection algorithms were actively studied over the last few years. Many have demonstrated superior performance compared with traditional feature-based methods. However, most methods remain riddled with assumptions and limitations, still not good enough for safe and reliable driving in the real world. In this paper, we propose a novel lane detection system, called Scene Understanding Physics-Enhanced Real-time (SUPER) algorithm. The proposed method consists of two main modules: 1) a hierarchical semantic segmentation network as the scene feature extractor and 2) a physics enhanced multi-lane parameter optimization module for lane inference. We train the proposed system using heterogeneous data from Cityscapes, Vistas and Apollo, and evaluate the performance on four completely separate datasets (that were never seen before), including Tusimple, Caltech, URBAN KITTI-ROAD, and Mcity-3000. The proposed approach performs the same or better than lane detection models already trained on the same dataset, and performs well even on datasets that was never trained on. Real-world vehicle tests were also conducted. Preliminary test results show promising real-time lane-detection performance compared with the Mobileye.

Lane Detection and Lane Departure Warning System

Lane detection is important for autonomous vehicles. For this reason, many approaches use lane boundary information to locate the vehicle inside the street, or to integrate GPS-based localization. Advanced driverassistance systems are developed to assist drivers in the driving process reducing road accidents. In this work, we present an end-to-end system for lane identification, clustering and classification, based on two cascaded neural networks, that runs in real-time. The first step is camera calibration which is used to remove the effect of lens distortion. Then a canny edge detection algorithm finds the edges of the images. Then the region of interest (ROI) is selected. The ROI is actually based on the rectangular shape appearing at the bottom of the image. ROI removes the unwanted region in the image. The potential lane markers are then determined using the Hough transform to analyze lane boundaries. Once the lane pixels are found, these pixels are continuously scanned to obtain the best linear regression analysis.It is qualified to be applied on highways and urban roadways. It also has been successfully verified in sunny, and rainy conditions for both day and night.

Autonomous Lane Keeping System: Lane Detection, Tracking and Control on Embedded System

Autonomous lane keeping system is the key technique to autonomous driving. It includes lane detection, lane tracking and control. It has been developed enormously, but it is still a challenge work due to different factors such as illumination, general hyper-parameters setting for different road condition and lane boundary correction. In addition, due to imbalance on accuracy and processing time, it is hard to conduct in embedding system. In this study, an autonomous lane keeping system is developed based on deep learning. First, a lane detection and tracking system is designed, which is robust to lane boundary correction. Especially for lane detection, a light-weight network named as LaneFCNet is proposed, which can balance accuracy and processing time. Then, lane tracking was followed by detector to improve the detection performance and create autonomous driving trajectory. Finally, to brief lane fitting problem, it was treated as ridge regression problem, which can enhance the effectiveness to the whole system. Experimental results show that our integrated lane detection and tracking system can trade off accuracy and processing time and the whole line keeping system is robust enough to autonomous driving.

Innovative lane detection method to increase the accuracy of lane departure warning system

Lane departure warning is one important feature in Advanced Driver Assistance Systems (ADAS), which aims to improve overall safety on the road. However, challenges such as inconsistent shadows and fading lane markings often plague the road surface and cause the lane detection system to produce false warnings. Users are aggravated by the warning and tend to disable this safety feature. This paper proposes an efficient Gabor filtering-based lane detection method to overcome the aforementioned conditions and improves the accuracy of lane departure warning system. Furthermore, it serves as a cost-effective solution to a lane departure warning problem, allowing it to be widely deployed. It is heuristically found that lane marking has a general directional property, which can be further enhanced by Gabor filter while suppressing inconsistent road shadows and road markers. Enhanced lane markings are then subjected to adaptive canny edge detection to extract distinct edge markings. Lastly, Hough transformation is applied to label the correct lane candidates on the road surface. Furthermore, we generate a dataset of Malaysia road with various driving conditions. As a proof of concept, a lane departure warning system is built based on the proposed lane detection method, which is able to achieve an accuracy of 93.67% for lane detection and 95.24% for lane departure warning tested on our challenging dataset. The codes are implemented on Raspberry pi 3B and installed in a vehicle for real-time application. The codes are multithreaded and found to achieve a desirable frame speed of 20 fps at 75% CPU utilization.

Perception, Planning and Control for Self-Driving System Based on On-board Sensors

The autonomous vehicle can recognize and understand environment, self-control, and achieve the driving level of the human driver. To create this kind of system, the following work was carried out: (a) a real time lane detection system is proposed, based on vision system functions, using webcam camera; (b) In order to detect curve lanes, deep learning is applied for lane detection, based on fully Convolutional Neural Network (CNN); (c) The cubic spline interpolation method is used for path generation, based on Global Positioning System (GPS) data, where distance between two adjacent path generation points is same. Compared with the connection method of the cubic polynomial fitting algorithm, the curve fitted path by the cubic spline interpolation method is smoother and more satisfied with the vehicle motion pattern; (d) Based on frenet coordinate frame, optimize trajectory planning method is used to plan safe, easy and comfort trajectory. Compared to the Cartesian coordinate frame, frenet coordinate frame simplifies the solution of road curve fitting problems, especially in the case of complex road environment; (e) Fuzzy sliding mode control method is proposed for vehicle steering control. Finally, the simulation and experiment results are used to prove the effectiveness of the proposed methods.

Development of an embedded road boundary detection system based on deep learning

The ability to sense the surrounding environment is an important developing technology in the field of automated vehicles. Lane line detection could determine a vehicle's travelable area. An embedded road boundary detection system based on deep learning was developed in this study. The system can detect structured and unstructured roads in a variety of situations. To obtain an image with clear lane markings, a convolution auto-encoder with the characteristics of noise reduction and reconstruction was used to remove all objects in the images except lane markings. Then, the feature points of the lane line were extracted, and the lane line was fitted with a hyperbolic model. Finally, a particle filter was used for lane tracking. The road boundary detection system was implemented on the NVIDIA Jetson TX2 platform. Three different situations, day, night, and rainy day were selected to demonstrate the performance of the proposed algorithm. Additionally, to deal with structured roads, some special scenes, such as shadows, tunnels, degenerate lane markings, and blocked lane markings, were considered. According to the experimental results, the accuracy of the proposed lane detection system for structured and unstructured roads was 90.02%.

Advanced Driver Security Application

The introduction of modern and more advanced vehicles has stretched their performance boundaries dramatically in terms of pace and maneuverability. It has also significantly enhanced the likelihood of people losing control of their vehicle, contributing to accidents. Within the past, several strategies have been suggested which resolve this issue by restricting the car's travel to just one specific path. To various road situations, this is achieved by applying lane identification utilizing algorithms such as canny edge identification, Hough transformations, vanishing point estimates, principal component analysis etc. Practical deployment of these programs, however, requires extremely powerful hardware such as TDA3x which can process in real time. This paper aims to introduce the usage of the Lane Detection and Alert System on a Texas Instruments Driver Assist 3x (TDA3x) board with a frame resolution of (1920 * 1080p) at 2 GHz relative to the current implementation, which has a resolution of only (480 * 270p) at 100 MHz [16]. This system too makes use of canny edge detection and hough transforms to identify the lane points, and tracks the vehicle movement by extracting the corresponding polar co-ordinates.

Multi-Dimensional Failure Modeling for Shared Data in Cooperative Systems

Abstract Autonomous systems will share data to enrich their environmental model and provide cooperative functionality. However, as shared data might be imprecise or inaccurate, its failure characteristics have to be analyzed by the receiving system before using the data. A corresponding failure model for describing failure characteristics was proposed by Jager et al. (2018), but is limited to one-dimensional sensory data. In this work, we extend the failure model to support multi-dimensional feature data as well. We exemplary evaluate the approach by modeling the failure characteristics of a lane detection system of a simulated car. By comparing it to state-of-the-art failure modeling techniques, we can show that the model accurately predicts failure amplitudes of previously unseen tracks even when trained on limited data.

A fuzzy logic-based autonomous car control system for the JavaScript Racer game

In this study, we propose a fuzzy logic-based autonomous car control system and its deployment into the JavaScript Racer game. The main design goal of the self-driving car control system is to end up with an intelligent vehicle control system that is capable to keep the car finish race in the shortest time as possible while keeping the car on the road course regardless the road conditions. To accomplish such a design goal, the proposed intelligent driving system takes up and merges the frameworks of control theory, fuzzy logic and image processing. The proposed structure is composed of vision-based lane detection system, fuzzy logic-based position reference (FPR) generator, fuzzy logic-based velocity reference (FVR) generator, low level position and velocity controllers. The vision-based lane detection system solves the road curve estimation problem in real time and provides this estimation feedback for the FPR and FVR to generate appropriate position and velocity reference signals. The design of the intelligent FPR and FVR generators is accomplished by converting expert knowledge into fuzzy rules. The generated reference signals from the FPR and FVR are then tracked by low level position and velocity controllers. The proposed structure is capable to percept the environment (i.e. road tracking), make decisions (reference signal generation) and control car dynamics (i.e. speed and position) as expected from an autonomous system. The paper presents the structure and design of the intelligent driving system by providing all the design steps in detail. The efficiency of the fuzzy logic-based car control system is shown with results collected from the game environment. It is revealed that the developed intelligent system can successfully operate in various racetracks and different operating points.

Learning full-reference quality-guided discriminative gradient cues for lane detection based on neural networks

Learning an intelligent lane detection system is significant to autonomous vehicles, which is a crucial module to smart cars. Although conventional approaches have achieved impressive performance, they suffer from the following limitations: (1) lane perception are confronted with different weather conditions and varied illumination. Existing methods lack a unified framework for characterizing different sceneries and (2) the inefficiency of utilizing images due to the potential label noise. To solve these limitations, we propose a lane detection framework towards autonomous vehicles by learning a full-reference quality-aware discriminative gradient deep model, where two types of deep networks are proposed. More specifically, we first design a gradient-guided deep convolutional network to detect the presence of lane, since the gradient value of lane edge is larger than that of other regions. We leverage full-reference image quality assessment (FR-IQA) method to discover more discriminative gradient cues, and geometric attributes are exploited simultaneously. Subsequently, a recurrent neural layer is designed to represent the spatial distribution of detected lanes whose visual cues are difficult to explicitly define. Noticeably, we only utilize a small proportion of the labeled images, whereas the noisy features are abandoned using sparsity penalty. Extensive experiments have demonstrated the effectiveness of our proposed method.

Straight and Curve Lane Detection System for Car Safety using Hough Transform and Sobel Operator

Straight and Curve Lane Detection System for Car Safety using Hough Transform and Sobel Operator

Design and Development of In-Vehicle Lane Departure Warning System using Standard Global Positioning System Receiver

A lane departure warning system is a critical element among advanced driver-assistance systems functions, which has significant potential to reduce crashes. Generally, lane departure warning systems use image processing or optical scanning techniques to detect a lane departure. These systems have some limitations, however, such as harsh weather or irregular lane markings having a negative influence on their performance. Integrating global positioning system (GPS) and digital maps of lane-level resolution with an image processing based lane detection system can improve its efficiency but make the overall system more complex and expensive. In this paper, a lane detection method is proposed which uses a standard GPS receiver without any lane-level resolution maps. The proposed algorithm determines the lateral shift of a vehicle by comparing the vehicle’s trajectory acquired by standard GPS receiver to the reference road direction. The reference road direction is extracted from a standard digital mapping database commonly available in any navigational device containing maps with only road-level information. Extensive field tests were performed to evaluate the efficiency of the proposed system. The field test results show that the proposed system can detect a true lane departure with an accuracy of almost 100%. Although no true lane departure was left undetected, occasional false lane departures were detected when the vehicle did not actually depart its lane. Furthermore, a modification in the proposed algorithm was also tested which has significant potential to reduce the frequency of false alarms.

Robust and real‐time lane detection filter based on adaptive neuro‐fuzzy inference system

Lane departure warning system used in vehicles has recently become very popular and is about to become a vital component in advanced driver assistance systems. The performance of this system is directly related to lane detection accuracy. In this study, a fuzzy inference system-based filter for robust lane detection is proposed. The proposed filter has three input parameters which are as follows: the difference between a pixel and its left and right neighbours at a certain distance along the horizontal direction and standard deviation of the pixels between the left and right neighbours. The parameters of the proposed fuzzy filter are determined in a learning phase by taking challenging scenarios such as varying lighting conditions, shadows, and road cracks. Experimental results reveal that the proposed method outperforms existing lane detection filters when integrated into a lane detection system. Since the proposed approach is computationally lightweight, it is suitable for real-time devices and applications.

Robust lane‐detection algorithm based on improved symmetrical local threshold for feature extraction and inverse perspective mapping

Here, an extended version of the symmetrical local threshold (SLT) algorithm is introduced for lane feature extraction and used in a novel lane-detection system. The introduced feature map extractor utilises parallel lane border features as well as the dark-light-dark (DLD) pattern of the lane marking used in SLT. Hence, compared to the SLT, the true positive to positive rate of the calculated feature maps is increased from 69% to 86% on the ROMA dataset. In addition, the proposed algorithm supplies orientation information for the estimated feature points, which can be useful for many optimisation algorithms. Consequently, based on the estimated lane feature orientations, a global lane orientation is calculated and used for both enhancing the feature map and estimating a one-dimensional (1D) lateral offset likelihood function. Then, the estimated 1D functions are filtered temporally and up to two linear lane candidates are detected. For increased flexibility, robust fitting is applied to the feature points in the region of interest (ROI). Finally, based on the detection of the previous frame, a mask is created and applied to the next frame. When tested on 2301 road images, mean error in lateral offset is calculated as 4.1 pixel on the IPM images.

Extraction of road using soft computing techniques

One of the most challenging research topics is extraction of road network automatically from remotely sensed imagery. The roads are considered as backbone and important modes of transportation. For efficient automatic road network extraction, ground points and building points extracted were integrated and it was used with multistage road network extraction model. For road network extraction, a lane detection system is considered as a more important element. From the lane marking in the roads, the roads are easily detected from the remotely sensed images. In order to detect the lanes more effectively, B-snake with fuzzy C-means (FCM) clustering is proposed in this paper. The input image is divided into equal halves by using FCM clustering algorithm. Then, the lower part is binarized by analyzing RGB channel and it is analyzed by using B-snake model because it contains more information about road lane than upper part of the image. Then, the lane detection problem is formulated by finding the set of lane model control points. A good initial position to the B-snake lane model is provided by introducing Canny/Hough estimation of vanishing points. Moreover, the external force field for lane detection is constructed by using gradient vector flow. The correspondence between B-snake and the real edge image is determined by using minimum mean square error which is presented to find out the parameters of road model iteratively. The detected lane points are fused with the road and ground points by using constraint satisfaction neural network-complementary information integration. Moreover, the road and non-road regions are separated by using region part segmentation. Finally, the medial-axis-transform-based hypothesis verification is used to automatically complete the road network extraction framework. The performance of the existing road network extraction, ground point-based road extraction model, building and ground point-based road extraction model and proposed building, ground and lane point-based road extraction model is analyzed in terms of correctness and completeness.

Ultra-Low Complexity Block-Based Lane Detection and Departure Warning System

This paper proposes an ultra-low complexity block-based lane detection and departure warning system. Based on the distribution of the lane markings in the region close to the vehicle, a parameterized region of interest (ROI) is determined. The lane markings in the ROI are enhanced by increasing the pixel intensity for detection in various environmental conditions. To reduce the computational burden, the ROI is partitioned into non-overlapping blocks and two simplified masks are proposed to obtain the block gradients and block angles. The driving conditions are classified into four classes to simplify the lane detection process and the proposed lane departure warning system is based on the lane detection results. The experimental results reveal that the average lane detection rate and the departure warning rate are 96.12% and 98.60%, respectively. With a $1920\times 1080$ resolution, the average processing time is 4.28 ms per frame.

An All-Weather Lane Detection System Based on Simulation Interaction Platform

Computer vision is an important part of the autonomous vehicles to gain the perception of the surrounding environment. In order to satisfy the adaptability of lane detection system under different illumination and different road conditions, an effective lane detection method based on image classification and hybrid isomeric operators is proposed. Camera correction matrix and distortion coefficient are obtained by using checkerboard grid images, and they will be cached to be shared with subsequent image streams to improve the speed of real-time lane detection. In the process of preprocessing, the images under different illumination conditions are classified and processed, and the inverse perspective transformation is used to deal with the 2-D image flow so that the image flow has a certain depth of data. According to the intensity of the whole gray mean, the edge detection is carried out by using heterogeneous operators and combining with a variety of threshold filtering methods to extract the lane pixels. The lane detection module transmits the vehicle steering angle and the deviate distance from the road center line to the decision layer, effectively implementing the interactive simulation. Experiments show that the algorithm has good real-time performance, stability, and robustness under different illumination and road conditions.

Dynamic integration and online evaluation of vision‐based lane detection algorithms

Lane detection techniques have been widely studied in the last two decades and applied in many advance driver assistance systems. However, the development of a robust lane detection system, which can deal with various road conditions and efficiently evaluate its detection results in real time, is still of great challenge. In this study, a vision-based lane detection system with dynamic integration and online evaluation is proposed. To increase the robustness of the lane detection system, the integration system dynamically processes two lane detection modules. First, a primary lane detection module is designed based on the steerable filter and Hough transform algorithm. Then, a secondary algorithm, which combines the Gaussian mixture model for image segmentation and random sample consensus for lane model fitting, will be activated when the primary algorithm encounters a low detection confidence. To detect the colour and line style of the ego lanes and evaluate the lane detection system in real time, a lane sampling and voting technique is proposed. By combining the sampling and voting system system with prior lane geometry knowledge, the evaluation system can efficiently recognise the false detections. The system works robustly in various complex situations (e.g. shadows, night, and lane missing scenarios) with a monocular camera.

Lane Detection and Classification for Forward Collision Warning System Based on Stereo Vision

This paper presents a lightweight stereo vision-based driving lane detection and classification system to achieve the ego-car's lateral positioning and forward collision warning to aid advanced driver assistance systems (ADAS). For lane detection, we design a self-adaptive traffic lanes model in Hough Space with a maximum likelihood angle and dynamic pole detection region of interests (ROIs), which is robust to road bumpiness, lane structure changing while the ego-car's driving and interferential markings on the ground. What's more, this model can be improved with geographic information system or electronic map to achieve more accurate results. Besides, the 3-D information acquired by stereo matching is used to generate an obstacle mask to reduce irrelevant objects' interfere and detect forward collision distance. For lane classification, a convolutional neural network is trained by using manually labeled ROI from KITTI data set to classify the left/right-side line of host lane so that we can provide significant information for lane changing strategy making in ADAS. Quantitative experimental evaluation shows good true positive rate on lane detection and classification with a real-time (15Hz) working speed. Experimental results also demonstrate a certain level of system robustness on variation of the environment.