Traffic Scene Understanding Research Articles

Nighttime scene understanding with label transfer scene parser

Semantic segmentation plays a crucial role in traffic scene understanding, especially in nighttime conditions. This paper tackles the task of semantic segmentation in nighttime scenes. The largest challenge of this task is the lack of annotated nighttime images to train a deep learning-based scene parser. The existing annotated datasets are abundant in daytime conditions but scarce in nighttime due to the high cost. Thus, we propose a novel Label Transfer Scene Parser (LTSP) framework for nighttime scene semantic segmentation by leveraging daytime annotation transfer. Our framework performs segmentation in the dark without training on real nighttime annotated data. In particular, we propose translating daytime images to nighttime conditions to obtain more data with annotation in an efficient way. In addition, we utilize the pseudo-labels inferred from unlabeled nighttime scenes to further train the scene parser. The novelty of our work is the ability to perform nighttime segmentation via daytime annotated labels and nighttime synthetic versions of the same set of images. The extensive experiments demonstrate the improvement and efficiency of our scene parser over the state-of-the-art methods with a similar semi-supervised approach on the benchmark of Nighttime Driving Test dataset. Notably, our proposed method utilizes only one-tenth of the amount of labeled and unlabeled data in comparison with the previous methods. Code is available at https://github.com/danhntd/Label_Transfer_Scene_Parser.git.

Cross-Domain Traffic Scene Understanding by Integrating Deep Learning and Topic Model.

Understanding cross-domain traffic scenarios from multicamera surveillance network is important for environmental perception. Most of existing methods select the source domain which is most similar to the target domain by comparing entire domains for cross-domain similarity and then transferring the motion model learned in the source domain to the target domain. The cross-domain similarity between overall different scenarios with similar local layouts is usually not utilized to improve any automatic surveillance tasks. However, these local commonalities, which may be shared across multiple traffic scenarios, can be transferred across scenarios as prior knowledge. To address these issues, we present a novel framework for cross-domain traffic scene understanding by integrating deep learning and topic model. This framework leverages the labeled samples with activity attribute labels from the source domain to annotate the target domain, where each label represents the local activity of some objects in the scene. When labeling the activity attributes of the target domain, there is no need to select the source domain, which avoids the phenomenon of performance degradation or even negative transfer due to wrong source domain selection. The effectiveness of the proposed framework is verified by extensive experiments carried out using public road traffic data.

ABSSNet: Attention-Based Spatial Segmentation Network for Traffic Scene Understanding.

The location information of road and lane lines is the supremely important thing for the automatic drive and auxiliary drive. The detection accuracy of these two elements dramatically affects the reliability and practicality of the whole system. In real applications, the traffic scene can be very complicated, which makes it particularly challenging to obtain the precise location of road and lane lines. Commonly used deep learning-based object detection models perform pretty well on the lane line and road detection tasks, but they still encounter false detection and missing detection frequently. Besides, existing convolution neural network (CNN) structures only pay attention to the information flow between layers, while it cannot fully utilize the spatial information inside the layers. To address those problems, we propose an attention-based spatial segmentation network for traffic scene understanding. We use the convolutional attention module to improve the network's understanding capacity of spatial location distribution. Spatial CNN (SCNN) obtains through the information flow within one single convolutional layer and improves the spatial relationship modeling ability of the network. The experimental results demonstrate that this method effectively improves the neural network's application ability of the spatial information, thereby improving the effect of traffic scene understanding. Furthermore, a pixel-level road segmentation dataset called NWPU Road Dataset is built to help improve the process of traffic scene understanding.

The Traffic Scene Understanding and Prediction Based on Image Captioning

The traffic scene understanding is the core technology in Intelligent Transportation Systems (ITS) and Advanced Driver Assistance System (ADAS), and it is becoming increasingly important for smart or autonomous vehicles. The recent methods for traffic scene understanding, such as Traffic Sign Recognition (TSR), Pedestrian Detection, and Vehicle Detection, have three major shortcomings. First, most models are customized for recognizing a specific category of traffic target instead of general traffic targets. Second, as for these recognition modules, the task of traffic scene understanding is to recognize objects rather than make driving suggestions or strategies. Third, numerous independent recognition modules disadvantage to fusing multi-modal information to make a comprehensive decision for driving operation in accordance with complicated traffic scenes. In this paper, we first introduce the image captioning model to alleviate the aforementioned shortcomings. Different from existing methods, our primary idea is to accurately identify all categories of traffic objects and understand traffic scenes by making full use of all information, and making the suggestions or strategy for driving operation in natural language by using Long Short Term Memory network (LSTM) rather than keywords. The proposed solution naturally solves the problems of feature fusion, general object recognition, and low-level semantic understanding. We tested the solution on our created traffic scene image dataset for evaluation of image captioning. Extensive experiments including quantitative and qualitative comparisons demonstrate that the proposed solution can identify more objects and produce higher-level semantic information than the state-of-the-arts.

Image guidance based 3D vehicle detection in traffic scene

3D vehicle detection is a significant step for traffic scene understanding. Several recent works have achieved state-of-the-art performance with only LiDAR point clouds. However, there are still existing challenges due to the intrinsic limitations of lidar data. In this paper, we propose a novel 3D vehicle detection method, in which visual information is introduced to remedy the deficiency of sparse point clouds. Our method is composed of two stages. In first stage, a novel proposal generator with the guidance of visual information is proposed. In the proposal generator, 2D detected bounding boxes are registered with 3D candidates from LiDAR by calibrating and then weighted non-maximum suppression (WNMS) is applied to increase the confidence of proposals to remove redundant proposals. In second stage, to construct a box predictor, instead of sampling in a sphere space, a novel sampling with cylinder space for pointnet++ is leveraged to learn local features of the point clouds. Furthermore, to achieve a better balance between confidence and localization accuracy of boxes, an Intersection-over-Union (IoU) prediction branch is modified and attached to the network. We conduct multiple experiments on the KITTI 3D object detection dataset and compare our method with state-of-the-art methods. The comparison results show that our algorithm is effective and can improve the performance of 3D vehicle detection.

Lane Marking Regression From Confidence Area Detection to Field Inference

Lane marking detection is a fundamental yet challenging task for traffic scene understanding. Previous works generally predict traffic line segmentation and obtain lane marking coordinates with a post-processing step, which are vulnerable to various challenging environments including occlusion, illumination variation, shadow, background clutter. In this paper, we propose a novel method, named as Lane marking Regression Network (LRN), which can simultaneously consider confidence area detection and field inference for producing more precise lane markings in an unified encoder-decoder framework. For obtaining confidence detection of lane area, we introduce the graph diffusion mechanism to aggregate the contextual information for better detecting hard areas. The field inference considers the geometric relationship between lane area and lane markings by adopting a novel constraint regression function. Comprehensive evaluations on CULane and GD datasets well demonstrate the significant superiority of our proposed LRN over other state-of-the-arts for lane marking detection.

VR‐based dataset for autonomous‐driving system

At present, visual recognition systems have acquired wide employment in the autonomous-driving area. The lack of fully featured benchmarks that mimic scenarios faced by autonomous-driving system is the core factor limiting the visual understanding of complex urban traffic scenes. However, to establish a dataset adequately captures the complexity of real-world urban traffics consuming time and effort. In order to solve these difficulties, authors involve virtual reality to develop a large-scale dataset, which trains and tests approaches for autonomous-driving vehicles. Using the label of the object in virtual scenes, the coordinate transformation of a 3D object to a 2D plane is calculated, which makes the label of the pixel block corresponding to the object in the 2D plane accessible. Their recording platform is equipped with video camera models, LiDAR model and positioning system. By using the pilot-in-the-loop method with driving simulator hardware and VR devices, the authors acquire and establish a large, diverse dataset comprising stereo video sequences recorded in streets and mountain roads from several different environments. Their pioneering method of using VR technology significantly mitigates the costs of acquisition of training data. Crucially, their effort exceeds previous attempts in terms of dataset size, scene variability and complexity.

RAPNet: Residual Atrous Pyramid Network for Importance-Aware Street Scene Parsing.

Street Scene Parsing (SSP) is a fundamental and important step for autonomous driving and traffic scene understanding. Recently, Fully Convolutional Network (FCN) based methods have delivered expressive performances with the help of large-scale dense-labeling datasets. However, in urban traffic environments, not all the labels contribute equally for making the control decision. Certain labels such as pedestrian, car, bicyclist, road lane or sidewalk would be more important in comparison with labels for vegetation, sky or building. Based on this fact, in this paper we propose a novel deep learning framework, named Residual Atrous Pyramid Network (RAPNet), for importance-aware SSP. More specifically, to incorporate the importance of various object classes, we propose an Importance-Aware Feature Selection (IAFS) mechanism which automatically selects the important features for label predictions. The IAFS can operate in each convolutional block, and the semantic features with different importance are captured in different channels so that they are automatically assigned with corresponding weights. To enhance the labeling coherence, we also propose a Residual Atrous Spatial Pyramid (RASP) module to sequentially aggregate global-to-local context information in a residual refinement manner. Extensive experiments on two public benchmarks have shown that our approach achieves new state-of-the-art performances, and can consistently obtain more accurate results on the semantic classes with high importance levels.

SADANet: Integrating Scale-Aware and Domain Adaptive for Traffic Sign Detection

It holds great implications for large-scale traffic scene understanding, to enable traffic sign detection in real environment. The large intra-category variance in features caused by instances of different spatial scales and appearances are the main problem. In this paper, a traffic sign detection framework using scale-aware and domain adaptive network (SADANet) was proposed, which seamlessly combines a multiscale prediction network (MSPN) with a domain adaptive network (DAN) in a tightly-coupled manner in order to tackle the challenge. Specifically, MSPN is dedicated to extracting the multiscale feature. It utilizes fully, the low-level location and high-level semantic information, and considers the combination of the context information and the instance specific content awareness in the scale transformation. DAN is dedicated to making features domain invariant without enough labeled test data. It aligns the domain distributions from the different scales effectively by leveraging the mapping relationship between the image representation and the multiscale features. Experimental results show that the SADANet is effective in traffic sign detection task and is also competitive when compared with the state-of-the-art methods.

Causal Reasoning in Multi-Object Interaction on the Traffic Scene: Occlusion-Aware Prediction of Visibility Fluent

Occlusion caused by multi-object interaction makes the traffic scene understanding intractable. In this paper, we focus on predicting the visibility status of vehicle in the framework of causality perception. The visibility fluent is employed to present the varying state of an object, involving visible and occluded. We introduce a probabilistic grammar model, named Hierarchical And-Or Graph (H-AOG), to construct the causal relations between fluents and actions. It consists of a Causal And-Or Graph (C-AOG) module and an Action And-Or Graph (A-AOG) module. An influence field is constructed by the polar coordinate transformation to model interactions in the A-AOG module. This method interprets the occurrence of occlusion due to multi-vehicle interaction. We evaluate our approach on both synthetic data and real data from the KITTI dataset. Compared to the state-of-the-art models like LSTM/GRU, it proves to achieve a promising accuracy in prediction of objects' visibility states and have better generalization on real data.

Learning Probabilistic Awareness Models for Detecting Abnormalities in Vehicle Motions

This paper proposes a method to detect abnormal motions in real vehicle situations based on trajectory data. Our approach uses a Gaussian process (GP) regression that facilitates to approximate expected vehicle’s movements over a whole environment based on sparse observed data. The main contribution of this paper consists in decomposing the GP regression into spatial zones, where quasi-constant velocity models are valid. Such obtained models are employed to build a set of Kalman filters that encode observed vehicle’s dynamics. This paper shows how proposed filters enable the online identification of abnormal motions. Detected abnormalities can be modeled and learned incrementally, automatically by intelligent systems. The proposed methodology is tested on real data produced by a vehicle that interacts with pedestrians in a closed environment. Automatic detection of abnormal motions benefits the traffic scene understanding and facilitates to close the gap between human driving and autonomous vehicle awareness.

Intelligent Driving Data Recorder in Smartphone Using Deep Neural Network-Based Speedometer and Scene Understanding

This paper proposes a smartphone-based Driving Data Recorder (DDR). The proposed DDR has the functions of accurate speed estimation and intelligent traffic scene understanding. DDRs are used to store the relevant driving data to provide feedback on driver behavior for accident analysis, insurance issue, and so on. The conventional DDRs are standalone devices with multiple sensors. The current DDR products record many useless data or lose important information. On the other hand, the widely used smartphones already have the hardware conditions to replace the conventional DDR products. This paper proposes to develop the intelligent DDR in the smartphones. Considering the requirements of the DDRs, two functions are developed in this paper: motion sensor-based speedometer and vision sensor-based scene understanding. The proposed speedometer function adopts double-layered Long Short-Term Memory (LSTM) network as the model, which can estimate the vehicle speed directly from gyroscope and accelerometer of a smartphone. The scene understanding function can detect road facilities such as traffic lights, crosswalks, and stop lines. The driving data recorded in those areas are very important for analyzing driver behaviors. In the development of the scene understanding function, maintaining high detection accuracy with reduced computation cost is significant due to the limitation of smartphones’ processing resources. This paper uses a lightweight architecture deep learning network to achieve the goal. The proposed system has been evaluated using the real traffic data. Speed estimation function only has 1.8 km/h of speed mean error. In addition, there is no accumulated error even for a long time driving. The evaluation of the scene understanding function indicates that the proposed method can provide a high-accuracy detection at 2 FPS, which is faster than the state-of-the-art method.

Cross-Domain Traffic Scene Understanding: A Dense Correspondence-Based Transfer Learning Approach

Understanding traffic scene images taken from vehicle mounted cameras is important for high-level tasks, such as advanced driver assistance systems and autonomous driving. It is a challenging problem due to large variations under different weather or illumination conditions. In this paper, we tackle the problem of traffic scene understanding from a cross-domain perspective. We attempt to understand the traffic scene from images taken from the same location but under different weather or illumination conditions (e.g., understanding the same traffic scene from images on a rainy night with the help of images taken on a sunny day). To this end, we propose a dense correspondence-based transfer learning (DCTL) approach, which consists of three main steps: 1) extracting deep representations of traffic scene images via a fine-tuned convolutional neural network; 2) constructing compact and effective representations via cross-domain metric learning and subspace alignment for cross-domain retrieval; and 3) transferring the annotations from the retrieved best matching image to the test image based on cross-domain dense correspondences and a probabilistic Markov random field. To verify the effectiveness of our DCTL approach, we conduct extensive experiments on a challenging data set, which contains 1828 images from six weather or illumination conditions.

Early and Accurate Recognition of Highway Traffic Maneuvers Considering Real World Application: A Novel Framework Using Bayesian Networks

This paper presents a novel application of artificial cognitive systems to traffic scene understanding and early recognition of highway maneuvers. This is achieved by use of Bayesian networks for knowledge representation, to mimic the human reasoning on situation analysis and to manage inherited uncertainties in the automotive domain, that requires efficient and effective analysis of high volume and frequency data streams. The maneuver recognition uses features, analyzing the observed vehicles behavior and available free space on the target lane.

Integrating 3D structure into traffic scene understanding with RGB-D data

RGB Video now is one of the major data sources of traffic surveillance applications. In order to detect the possible traffic events in the video, traffic-related objects, such as vehicles and pedestrians, should be first detected and recognized. However, due to the 2D nature of the RGB videos, there are technical difficulties in efficiently detecting and recognizing traffic-related objects from them. For instance, the traffic-related objects cannot be efficiently detected in separation while parts of them overlap, and complex background will influence the accuracy of the object detection. In this paper, we propose a robust RGB-D data based traffic scene understanding algorithm. By integrating depth information, we can calculate more discriminative object features and spatial information can be used to separate the objects in the scene efficiently. Experimental results show that integrating depth data can improve the accuracy of object detection and recognition. We also show that the analyzed object information plus depth data facilitate two important traffic event detection applications: overtaking warning and collision avoidance.

Vehicles overtaking detection using RGB-D data

Outstanding breakthrough in traffic video surveillance has been made with the development of computer vision techniques. However, there are still some problems to be solved due to the limitations of two-dimensional (2D) video data. With the popularity of three-dimensional (3D) cameras, high-quality color and depth data can be obtained simultaneously in real time through the video stream. In this paper, we propose a vehicles overtaking detection method using RGB-D data captured by the Kinect device in simulated traffic scenes. Vehicles are detected and tracked with a robust traffic scene understanding on RGB-D data. The depth data is utilized to recognize vehicles overtaking by analyzing the posture change of vehicles in different scenes. The principle of vehicles overtaking detection is to fit the line of vehicle side in the coordinate system and calculate the angle between the vehicle side and the road orientation. As a consequence, the posture change of the vehicle can be recognized. The proposed method is evaluated using simulation experiments which show that our work has good performance in vehicles overtaking detection and tracking.

3D Traffic Scene Understanding From Movable Platforms.

In this paper, we present a novel probabilistic generative model for multi-object traffic scene understanding from movable platforms which reasons jointly about the 3D scene layout as well as the location and orientation of objects in the scene. In particular, the scene topology, geometry, and traffic activities are inferred from short video sequences. Inspired by the impressive driving capabilities of humans, our model does not rely on GPS, lidar, or map knowledge. Instead, it takes advantage of a diverse set of visual cues in the form of vehicle tracklets, vanishing points, semantic scene labels, scene flow, and occupancy grids. For each of these cues, we propose likelihood functions that are integrated into a probabilistic generative model. We learn all model parameters from training data using contrastive divergence. Experiments conducted on videos of 113 representative intersections show that our approach successfully infers the correct layout in a variety of very challenging scenarios. To evaluate the importance of each feature cue, experiments using different feature combinations are conducted. Furthermore, we show how by employing context derived from the proposed method we are able to improve over the state-of-the-art in terms of object detection and object orientation estimation in challenging and cluttered urban environments.

Object-Oriented Bayesian Networks for Detection of Lane Change Maneuvers

This article introduces a novel approach towards the recognition of typical driving maneuvers in structured highway scenarios and shows some key benefits of traffic scene modeling with object-oriented Bayesian networks (OOBNs). The approach exploits the advantages of an introduced lane-related coordinate system together with individual occupancy schedule grids for all modeled vehicles. This combination allows an efficient classification of the existing vehicle-lane and vehicle- vehicle relations in traffic scenes and thus substantially improves the understanding of complex traffic scenes. Probabilities and variances within the network are propagated systematically which results in probabilistic sets of the modeled driving maneuvers. Using this generic approach, the network is able to classify a total of 27 driving maneuvers including merging and object following.