Perception Of Driving Environment Research Articles

Spatial awareness enhancement based single-stage anchor-free 3D object detection for autonomous driving

The real-time and accurate detection of three-dimensional (3D) objects based on LiDAR is a focal problem in the field of autonomous driving environment perception. Compared to two-stage and anchor-based 3D object detection methods that suffer from inference latency challenges, single-stage anchor-free 3D object detection approaches are more suitable for deployment in autonomous driving vehicles with the strict real-time requirement. However, they face the issue of insufficient spatial awareness, which can result in detection errors such as false positives and false negatives, thereby increasing the potential risks of autonomous driving. In response to this, we focus on enhancing the spatial awareness of CenterPoint, a widely used single-stage anchor-free 3D object detector in the industry. Considering the limited allocation of computational resources and the performance bottleneck caused by pillar encoder, we propose an efficient SSDCM backbone to strengthen feature representation and extraction. Furthermore, a simple BGC neck is devised to weight and exchange contextual information in order to deeply fuse multi-scale features. Combining improved backbone and neck networks, we construct a single-stage anchor-free 3D object detection model with spatial awareness enhancement, named CenterPoint-Spatial Awareness Enhancement (CenterPoint-SAE). We evaluate CenterPoint-SAE on two large-scale and challenging autonomous driving datasets, nuScenes and Waymo. It achieves 53.3% mAP and 62.5% NDS on nuScenes detection benchmark, and runs inference at a speed of 11.1 FPS. Compared to the baseline, the upgraded networks deliver a performance improvement of 1.6% mAP and 1.2% NDS at minor cost. Notably, on Waymo dataset, our method achieves competitive detection performance compared to two-stage and point-based methods.

A multi-level privacy-preserving scheme for extracting traffic images

Traffic images are constantly used as a stick to assess traffic conditions. Traffic flow statistical analysis, road condition safety monitoring, vehicle violation detection, accident surveillance, and the driving environment perception of autonomous vehicles are all functionalities that depend on the processing of traffic images. However, traffic images contain privacy-sensitive information related to users, such as license plate numbers, drivers and passengers. Extracting, analyzing and sharing such privacy-sensitive information without any security measures may raise concerns among users about potential privacy violations. This paper proposes a multi-level privacy protection scheme for traffic image extraction based on compressive sensing, which has the advantages of data undersampling, privacy protection and data access control. Detailed compression rate analysis demonstrates that the proposed solution can effectively reduce the transmission load of image information. Security analysis demonstrates that the proposed approach achieves multi-level reconstruction qualities and high-security strength for users with different authorities. Thus, it has a good application prospect in an image-based intelligent traffic management system.

Vehicle Trajectory Prediction Considering Multi-feature Independent Encoding Based on Graph Neural Network

Background: Today, self-driving cars are already on the roads. However, driving safety remains a huge challenge. Trajectory prediction of traffic targets is one of the important tasks of an autonomous driving environment perception system, and its output trajectory can provide necessary information for decision control and path planning. Although there are many patents and articles related to trajectory prediction, the accuracy of trajectory prediction still needs to be improved. Objective: This paper aimed to propose a novel scheme that considers multi-feature independent encoding trajectory prediction (MFIE). Methods: MFIE is an independently coded trajectory prediction algorithm that consists of a spacetime interaction module and trajectory prediction module, and considers speed characteristics and road characteristics. In the spatiotemporal interaction module, an undirected and weightless static traffic graph is used to represent the interaction between vehicles, and multiple graph convolution blocks are used to perform data mining on the historical information of target vehicles, capture temporal features, and process spatial interaction features. In the trajectory prediction module, three long short-term memory (LSTM) encoders are used to encode the trajectory feature, motion feature, and road constraint feature independently. The three hidden features are spliced into a tensor, and the LSTM decoder is used to predict the future trajectory. Results: On datasets, such as Apollo and NGSIM, the proposed method has shown lower prediction error than traditional model-driven and data-driven methods, and predicted more target vehicles at the same time. It can provide a basis for vehicle path planning on highways and urban roads, and it is of great significance to the safety of autonomous driving. Conclusion: This paper has proposed a multi-feature independent encoders’ trajectory prediction data-driven algorithm, and the effectiveness of the algorithm is verified with a public dataset. The trajectory prediction algorithm considering multi-feature independent encoders provides some reference value for decision planning.

Real-Time Sketching of Harshly Lit Driving Environment Perception by Neuromorphic Sensing

Real-Time Sketching of Harshly Lit Driving Environment Perception by Neuromorphic Sensing

Drivable Area Detection in Unstructured Environments based on Lightweight Convolutional Neural Network for Autonomous Driving Car

Road detection technology is an important part of the automatic driving environment perception system. With the development of technology, the situations that automatic driving needs to consider will become broader and more complex. This paper contributes a lightweight convolutional neural network model, incorporating novel convolution and parallel pooling modules, an improved network activation function, and comprehensive training and verification with multiple datasets. The proposed model achieves high accuracy in detecting drivable areas in complex autonomous driving situations while significantly improving real-time performance. In addition, we collect data in the field and create small datasets as reference datasets for testing algorithms. This paper designs relevant experimental scenarios based on the datasets and experimental platforms and conducts simulations and real-world vehicle experiments to verify the effectiveness and stability of the algorithm models and technical solutions. The method achieves an MIoU of 90.19 and a single batch time of 340 ms with a batch size of 8, which substantially reduces the runtime relative to a typical deep network structure like ResNet50.

Feature selection based on the self-calibration of binocular camera extrinsic parameters

The accuracy of feature-based vision algorithms, including the self-calibration of binocular camera extrinsic parameters used in autonomous driving environment perception techniques relies heavily on the quality of the features extracted from the images. This study investigates the influence of the depth distance between objects and the camera, the feature points in different object regions, and the feature points in dynamic object regions on the self-calibration of binocular camera extrinsic parameters. To achieve this, the study first filters out different types of objects in the image through semantic segmentation. Then, it identifies the areas of dynamic objects and extracts the feature points in the static object region for the self-calibration of binocular camera extrinsic parameters. By calculating the baseline error of the binocular camera and the row alignment error of the matching feature points, this study evaluates the influence of feature points in dynamic object regions, feature points in different object regions, and feature points at different distances on the self-calibration algorithm. The experimental results demonstrate that feature points at static objects close to the camera are beneficial for the self-calibration of extrinsic parameters of binocular camera.

Perception Enhancement and Improving Driving Context Recognition of an Autonomous Vehicle Using UAVs

The safety of various road users and vehicle passengers is very important in our increasingly populated roads and highways. To this end, the correct perception of driving conditions is imperative for a driver to react accordingly to a given driving situation. Various sensors are currently being used in recognizing driving context. To further enhance such driving environment perception, this paper proposes the use of UAVs (unmanned aerial vehicles, also known as drones). In this work, drones are equipped with sensors (radar, lidar, camera, etc.), capable of detecting obstacles, accidents, and the like. Due to their small size and capability to move places, drones can be used collect perception data and transmit them to the vehicle using a secure method, such as an RF, VLC, or hybrid communication protocol. These data obtained from different sources are then combined and processed using a knowledge base and some set of logical rules. The knowledge base is represented by ontology; it contains various logical rules related to the weather, the appropriateness of sensors with respect to the weather, and the activation mechanism of UAVs containing these sensors. Logical rules about which communication protocols to use also exist. Finally, various driving context cognition rules are provided. The result is a more reliable environment perception for the vehicle. When necessary, users are provided with driving assistance information, leading to safe driving and fewer road accidents. As a proof of concept, various use cases were tested in a driving simulator in the laboratory. Experimental results show that the system is an effective tool in improving driving context recognition and in preventing road accidents.

Development of an Energy Efficient and Cost Effective Autonomous Vehicle Research Platform.

Commercialization of autonomous vehicle technology is a major goal of the automotive industry, thus research in this space is rapidly expanding across the world. However, despite this high level of research activity, literature detailing a straightforward and cost-effective approach to the development of an AV research platform is sparse. To address this need, we present the methodology and results regarding the AV instrumentation and controls of a 2019 Kia Niro which was developed for a local AV pilot program. This platform includes a drive-by-wire actuation kit, Aptiv electronically scanning radar, stereo camera, MobilEye computer vision system, LiDAR, inertial measurement unit, two global positioning system receivers to provide heading information, and an in-vehicle computer for driving environment perception and path planning. Robotic Operating System software is used as the system middleware between the instruments and the autonomous application algorithms. After selection, installation, and integration of these components, our results show successful utilization of all sensors, drive-by-wire functionality, a total additional power* consumption of 242.8 Watts (*Typical), and an overall cost of $118,189 USD, which is a significant saving compared to other commercially available systems with similar functionality. This vehicle continues to serve as our primary AV research and development platform.

MVP-Net: Multiple View Pointwise Semantic Segmentation of Large-Scale Point Clouds

Semantic segmentation of 3D point cloud is an essential task for autonomous driving environment perception. The pipeline of most pointwise point cloud semantic segmentation methods includes points sampling, neighbor searching, feature aggregation, and classification. Neighbor searching method like K-nearest neighbors algorithm, KNN, has been widely applied. However, the complexity of KNN is always a bottleneck of efficiency. In this paper, we propose an end-to-end neural architecture, Multiple View Pointwise Net, MVP-Net, to efficiently and directly infer large-scale outdoor point cloud without KNN or any complex pre/postprocessing. Instead, assumption-based space filling curves and multi-rotation of point cloud methods are introduced to point feature aggregation and receptive field expanding. Numerical experiments show that the proposed MVP-Net is 11 times faster than the most efficient pointwise semantic segmentation method RandLA-Net [Qin20a] and achieves the same accuracy on the large-scale benchmark SemanticKITTI dataset.

Multitarget Vehicle Tracking and Motion State Estimation Using a Novel Driving Environment Perception System of Intelligent Vehicles

The multitarget vehicle tracking and motion state estimation are crucial for controlling the host vehicle accurately and preventing collisions. However, current multitarget tracking methods are inconvenient to deal with multivehicle issues due to the dynamically complex driving environment. Driving environment perception systems, as an indispensable component of intelligent vehicles, have the potential to solve this problem from the perspective of image processing. Thus, this study proposes a novel driving environment perception system of intelligent vehicles by using deep learning methods to track multitarget vehicles and estimate their motion states. Firstly, a panoramic segmentation neural network that supports end-to-end training is designed and implemented, which is composed of semantic segmentation and instance segmentation. A depth calculation model of the driving environment is established by adding a depth estimation branch to the feature extraction and fusion module of the panoramic segmentation network. These deep neural networks are trained and tested in the Mapillary Vistas Dataset and the Cityscapes Dataset, and the results showed that these methods performed well with high recognition accuracy. Then, Kalman filtering and Hungarian algorithm are used for the multitarget vehicle tracking and motion state estimation. The effectiveness of this method is tested by a simulation experiment, and results showed that the relative relation (i.e., relative speed and distance) between multiple vehicles can be estimated accurately. The findings of this study can contribute to the development of intelligent vehicles to alert drivers to possible danger, assist drivers’ decision-making, and improve traffic safety.

Driving Environment Perception Based on the Fusion of Vehicular Wireless Communications and Automotive Remote Sensors.

Driving environment perception for automated vehicles is typically achieved by the use of automotive remote sensors such as radars and cameras. A vehicular wireless communication system can be viewed as a new type of remote sensor that plays a central role in connected and automated vehicles (CAVs), which are capable of sharing information with each other and also with the surrounding infrastructure. In this paper, we present the design and implementation of driving environment perception based on the fusion of vehicular wireless communications and automotive remote sensors. A track-to-track fusion of high-level sensor data and vehicular wireless communication data was performed to accurately and reliably locate the remote target in the vehicle surroundings and predict the future trajectory. The proposed approach was implemented and evaluated in vehicle tests conducted at a proving ground. The experimental results demonstrate that using vehicular wireless communications in conjunction with the on-board sensors enables improved perception of the surrounding vehicle located at varying longitudinal and lateral distances. The results also indicate that vehicle future trajectory and potential crash involvement can be reliably predicted with the proposed system in different cut-in driving scenarios.

Vehicle Detection and Ranging Using Two Different Focal Length Cameras

Vehicle detection is a crucial task for autonomous driving and demands high accuracy and real-time speed. Considering that the current deep learning object detection model size is too large to be deployed on the vehicle, this paper introduces the lightweight network to modify the feature extraction layer of YOLOv3 and improve the remaining convolution structure, and the improved Lightweight YOLO network reduces the number of network parameters to a quarter. Then, the license plate is detected to calculate the actual vehicle width and the distance between the vehicles is estimated by the width. This paper proposes a detection and ranging fusion method based on two different focal length cameras to solve the problem of difficult detection and low accuracy caused by a small license plate when the distance is far away. The experimental results show that the average precision and recall of the Lightweight YOLO trained on the self-built dataset is 4.43% and 3.54% lower than YOLOv3, respectively, but the computing speed of the network decreases 49 ms per frame. The road experiments in different scenes also show that the long and short focal length camera fusion ranging method dramatically improves the accuracy and stability of ranging. The mean error of ranging results is less than 4%, and the range of stable ranging can reach 100 m. The proposed method can realize real-time vehicle detection and ranging on the on-board embedded platform Jetson Xavier, which satisfies the requirements of automatic driving environment perception.

Vehicle Trajectory Prediction and Collision Warning via Fusion of Multisensors and Wireless Vehicular Communications.

Driver inattention is one of the leading causes of traffic crashes worldwide. Providing the driver with an early warning prior to a potential collision can significantly reduce the fatalities and level of injuries associated with vehicle collisions. In order to monitor the vehicle surroundings and predict collisions, on-board sensors such as radar, lidar, and cameras are often used. However, the driving environment perception based on these sensors can be adversely affected by a number of factors such as weather and solar irradiance. In addition, potential dangers cannot be detected if the target is located outside the limited field-of-view of the sensors, or if the line of sight to the target is occluded. In this paper, we propose an approach for designing a vehicle collision warning system based on fusion of multisensors and wireless vehicular communications. A high-level fusion of radar, lidar, camera, and wireless vehicular communication data was performed to predict the trajectories of remote targets and generate an appropriate warning to the driver prior to a possible collision. We implemented and evaluated the proposed vehicle collision system in virtual driving environments, which consisted of a vehicle–vehicle collision scenario and a vehicle–pedestrian collision scenario.

Technology of intelligent driving radar perception based on driving brain

Radar is an important sensor to realise intelligent driving environment perception, enabling the detection of static obstacles and dynamic obstacles, and the tracking of a dynamic obstacle. The models, quantities, and installing location of the platform radar sensors as well as the information processing modules differ from each other on different intelligent driving testing platforms, resulting in different quantities and interfaces on the intelligent driving system. Here, the authors build the software architecture of intelligent driving vehicle based on driving brain which is used to adapt to different types of radar sensors and use the variable granularity road ownership radar for radar information fusion. Under the condition of complete driving information, increasing or reducing the number of radar sensors and changing the radar sensor model or installing location will not affect the intelligent driving decision directly. Therefore, the authors meet the demands of multi-radar sensor adapting to different intelligent driving hardware testing platforms.

A cooperative vehicle-infrastructure based urban driving environment perception method using a D-S theory-based credibility map

Driving environment perception technology is a key issue for driving assistance systems and autonomous vehicles. Driving environment information collection and information fusion and representation are two important contents of this technology. On-board sensors such as camera, radar, lidar, etc., are popularly used to collect relevant information. However, when using these on-board sensors, uncertainty arises from ignorance and errors, where ignorance is usually caused by vehicle occlusion, and errors is caused by imprecise pose estimation and noisy measurements. Increasing sensor number would decrease such uncertainty, but complexity and cost is also increased in this way. In this paper, we present a V2I (vehicle-infrastructure) based urban driving environment collection method achieved only by two sensors, i.e., a Road Side Unit camera (RSU camera) and vehicle GPS. RSU camera is used to extract lane marking information, including position and type, and vehicle position information. Vehicle GPS provide position information of vehicle. A credibility map is proposed as the compact representation of urban environment, and accept those information from RSU camera and vehicle GPS. In order to hand uncertainty caused by sensors, Dempster-Shafer theory is employed to estimate the credibility of each cell, fuse the credibility root in the information provided by RSU camera and vehicle GPS, and map onto the credibility map. The performance of this method has been validated with real-world urban traffic data.