Environment Perception System Research Articles

CTAFFNet: CNN–Transformer Adaptive Feature Fusion Object Detection Algorithm for Complex Traffic Scenarios

As the core technology of an environmental perception system, object detection has received more and more attention and has become a hot research direction for intelligent driving vehicles. The CNN–Transformer hybrid model has poor generalization ability, making it difficult to meet the detection requirements for small objects in complex scenes. We propose a novel convolutional neural network (CNN)–Transformer Adaptive Feature Fusion Network (CTAFFNet) for object detection. First, we design a Local–Global Feature Fusion unit known as the Convolutional Transformation Adaptive Fusion Kernel (CTAFFK), which is integrated into CTAFFNet. The CTAFFK kernel utilizes two branches, namely CNN and Transformer, to extract local and global features from the image, and adaptively fuses the features from both branches. Subsequently, we develop an adaptive feature fusion strategy that combines local high-frequency and global low-frequency features to obtain comprehensive feature information. Finally, CTAFFNet employs an encoder–decoder structure to facilitate the flow of fused local–global information between different stages, ensuring the model’s generalization capabilities. Results from the experiment conducted on the large and challenging KITTI dataset demonstrate the effectiveness and efficiency of the proposed network. Compared with other mainstream networks, it achieves an average precision of 91.17%, particularly excelling in the detection of small objects at longer distances with a remarkable 70.18% accuracy, while also providing real-time detection capabilities.

Land-air coupled environmental perception system during the construction phase of pumped storage power stations

Land-air coupled environmental perception system during the construction phase of pumped storage power stations

Sensor-Fused Nighttime System for Enhanced Pedestrian Detection in ADAS and Autonomous Vehicles.

Ensuring a safe nighttime environmental perception system relies on the early detection of vulnerable road users with minimal delay and high precision. This paper presents a sensor-fused nighttime environmental perception system by integrating data from thermal and RGB cameras. A new alignment algorithm is proposed to fuse the data from the two camera sensors. The proposed alignment procedure is crucial for effective sensor fusion. To develop a robust Deep Neural Network (DNN) system, nighttime thermal and RGB images were collected under various scenarios, creating a labeled dataset of 32,000 image pairs. Three fusion techniques were explored using transfer learning, alongside two single-sensor models using only RGB or thermal data. Five DNN models were developed and evaluated, with experimental results showing superior performance of fused models over non-fusion counterparts. The late-fusion system was selected for its optimal balance of accuracy and response time. For real-time inferencing, the best model was further optimized, achieving 33 fps on the embedded edge computing device, an 83.33% improvement in inference speed over the system without optimization. These findings are valuable for advancing Advanced Driver Assistance Systems (ADASs) and autonomous vehicle technologies, enhancing pedestrian detection during nighttime to improve road safety and reduce accidents.

Validation Scores to Evaluate the Detection Capability of Sensor Systems Used for Autonomous Machines in Outdoor Environments

The characterization of the detection capability assumes significance when the reliable monitoring of the region of interest by a non-contact sensor is a safety-relevant function. This paper introduces new validation scores that evaluate the detection capability of non-contact sensors intended to be applied to outdoor machines. The scores quantify, in terms of safety, the suitability of the sensor for the intended implementation in an environmental perception system of (highly) automated machines. This was achieved by developing an extension to the new Real Environment Detection Area (REDA) method and linking the methodology with the sensor standard IEC/TS 62998-1. The extension includes point-by-point and statistic-based error evaluation which leads to the Usability-Score, Availability-Score, and Reliability-Score. By applying the principle in the agricultural sector using ISO 18497 and linking this with data from a real outdoor test stand, it was possible to show that the validation scores offer a generic approach to quantify the detection capability and express this in a machine manufacturer-oriented manner. The findings of this study have significant implications for the advancement of safety-related sensor systems integrated into machines operating in complex environments. In order to achieve full implementation, it is necessary to define in the standards which score is required for each Performance Level (PL).

Vehicle Trajectory Prediction Considering Multi-feature Independent Encoding Based on Graph Neural Network

Background: Today, self-driving cars are already on the roads. However, driving safety remains a huge challenge. Trajectory prediction of traffic targets is one of the important tasks of an autonomous driving environment perception system, and its output trajectory can provide necessary information for decision control and path planning. Although there are many patents and articles related to trajectory prediction, the accuracy of trajectory prediction still needs to be improved. Objective: This paper aimed to propose a novel scheme that considers multi-feature independent encoding trajectory prediction (MFIE). Methods: MFIE is an independently coded trajectory prediction algorithm that consists of a spacetime interaction module and trajectory prediction module, and considers speed characteristics and road characteristics. In the spatiotemporal interaction module, an undirected and weightless static traffic graph is used to represent the interaction between vehicles, and multiple graph convolution blocks are used to perform data mining on the historical information of target vehicles, capture temporal features, and process spatial interaction features. In the trajectory prediction module, three long short-term memory (LSTM) encoders are used to encode the trajectory feature, motion feature, and road constraint feature independently. The three hidden features are spliced into a tensor, and the LSTM decoder is used to predict the future trajectory. Results: On datasets, such as Apollo and NGSIM, the proposed method has shown lower prediction error than traditional model-driven and data-driven methods, and predicted more target vehicles at the same time. It can provide a basis for vehicle path planning on highways and urban roads, and it is of great significance to the safety of autonomous driving. Conclusion: This paper has proposed a multi-feature independent encoders’ trajectory prediction data-driven algorithm, and the effectiveness of the algorithm is verified with a public dataset. The trajectory prediction algorithm considering multi-feature independent encoders provides some reference value for decision planning.

Spatio-Temporal-Attention-Based Vehicle Trajectory Prediction Considering Multi-Vehicle Interaction in Mixed Traffic Flow

As a link connecting the environmental perception system and the decision-making system, accurate obstacle trajectory prediction provides a reliable guarantee of correct decision-making by autonomous vehicles. Oriented toward a mixed human-driven and machine-driven traffic environment, a vehicle trajectory prediction algorithm based on an encoding–decoding framework composed of a multiple-attention mechanism is proposed. Firstly, a directed graph is used to describe vehicle–vehicle motion dependencies. Then, by calculating the repulsive force between vehicles using a priori edge information based on the artificial potential field theory, vehicle–vehicle interaction coefficients are extracted via a graph attention mechanism (GAT). Subsequently, after concatenating the vehicle–vehicle interaction feature with the encoded vehicle trajectory vectors, a spatio-temporal attention mechanism is applied to determine the coupling relationship of hidden vectors. Finally, the predicted trajectory is generated by a gated recurrent unit (GRU) decoder. The training and evaluation of the proposed model were conducted on the NGSIM public dataset. The test results demonstrated that compared with existing baseline models, our approach has fewer prediction errors and better robustness. In addition, introducing artificial potential fields into the attention mechanism causes the model to have better interpretability.

The Data Fusion and Semantic Segmentation Method Based on Multi-Sensor System

The design and implementation of a prototype environmental perception system are introduced, which consists of the multi-camera, LiDAR, and IMU sensor. The processing steps of the environmental data collected by the multi-sensor platform include joint calibration, data fusion, and semantic segmentation. First, the optimization method based on the epipolar constraint is proposed for the joint calibration of the multi-camera and the LiDAR system. In the data fusion section, an improved data association method for the point cloud is proposed where the foreground segmentation method is used to reduce scale estimation error due to the scale-sudden-change. Finally, the semantic segmentation method based on the deep-learning is proposed. The convolutional neural network based on the Squeeze-and-Excitation Net is designed to identify and classify the point cloud data accurately. The physical simulation result shows that the system collects and identifies the environmental information accurately.

ConCs-Fusion: A Context Clustering-Based Radar and Camera Fusion for Three-Dimensional Object Detection

Multi-modality three-dimensional (3D) object detection is a crucial technology for the safe and effective operation of environment perception systems in autonomous driving. In this study, we propose a method called context clustering-based radar and camera fusion for 3D object detection (ConCs-Fusion) that combines radar and camera sensors at the intermediate fusion level to achieve 3D object detection. We extract features from heterogeneous sensors and input them as feature point sets into the fusion module. Within the fusion module, we utilize context cluster blocks to learn multi-scale features of radar point clouds and images, followed by upsampling and fusion of the feature maps. Then, we leverage a multi-layer perceptron to nonlinearly represent the fused features, reducing the feature dimensionality to improve model inference speed. Within the context cluster block, we aggregate feature points of the same object from different sensors into one cluster based on their similarity. All feature points within the same cluster are then fused into a radar–camera feature fusion point, which is self-adaptively reassigned to the originally extracted feature points from a simplex sensor. Compared to previous methods that only utilize radar as an auxiliary sensor to camera, or vice versa, the ConCs-Fusion method achieves a bidirectional cross-modal fusion between radar and camera. Finally, our extensive experiments on the nuScenes dataset demonstrate that ConCs-Fusion outperforms other methods in terms of 3D object detection performance.

Vehicle-mounted imaging lidar with nonuniform distribution of instantaneous field of view

Imaging lidar is an important sensor for use in vehicle environment perception systems, and its field of view (FOV) and instantaneous field of view (IFOV) determine the system’s spatial coverage and the spatial resolution of detection, respectively. A smaller IFOV indicates that lidar can recognize a target over a greater distance. The FOV and the IFOV of conventional array-type lidar systems are constrained by the detector and cannot increase their target recognition distance while maintaining the same spatial coverage. To solve this problem, an imaging vehicle-mounted lidar system with a nonuniform IFOV distribution is proposed. By reducing the central IFOV while maintaining the same detection space, the system effectively doubles the available range for target discovery and recognition. Simultaneously, the central FOV has a larger entrance pupil, which matches the characteristic required to allow the central FOV to detect targets at greater distances. The proposed lidar system can recognize vehicle information at a longer range than conventional systems, and can play an important role in the 3D environmental perception field for intelligent vehicles.

Vision-Based Autonomous Driving: A Hierarchical Reinforcement Learning Approach

Human drivers have excellent perception and reaction abilities in complex environments such as dangerous highways, busy intersections, and harsh weather conditions. To achieve human-level driving performance, autonomous driving systems require powerful environmental perception systems and the ability to make accurate decisions in difficult situations, enabling vehicles to maintain smooth driving. However, making decisions based on visual perception is still a daunting challenge for autonomous vehicles, also current end-to-end networks and modular frameworks have limitations in effectively addressing perception, decision-making, and control, such as a lack of interpretability and weak generalization ability in complex environments. This paper proposes an elaborate modular pipeline for autonomous driving that effectively integrates semantic perception information, multi-level decision tasks, and control modules. The decision-making module comprehensively considers high-level maneuver selection and low-level motion control in both horizontal and vertical directions. The proposed MP framework is trained end-to-end by a novel hierarchical reinforcement learning method with a new action sampling mechanism. To adapt to the experimental scenarios, we use the CARLA simulation platform to collect data to evaluate the proposed autonomous driving framework and its new training method, considering various environmental factors such as sunny, dusk, and rainy weather. The results show that the framework exhibits smooth and effective driving strategies in different environments and can converge quickly and stably. The method improves learning efficiency and reduces unnecessary coupling and error propagation. Overall, the proposed framework and training method provide new ideas for improving existing autonomous driving systems.

Drivable Area Detection in Unstructured Environments based on Lightweight Convolutional Neural Network for Autonomous Driving Car

Road detection technology is an important part of the automatic driving environment perception system. With the development of technology, the situations that automatic driving needs to consider will become broader and more complex. This paper contributes a lightweight convolutional neural network model, incorporating novel convolution and parallel pooling modules, an improved network activation function, and comprehensive training and verification with multiple datasets. The proposed model achieves high accuracy in detecting drivable areas in complex autonomous driving situations while significantly improving real-time performance. In addition, we collect data in the field and create small datasets as reference datasets for testing algorithms. This paper designs relevant experimental scenarios based on the datasets and experimental platforms and conducts simulations and real-world vehicle experiments to verify the effectiveness and stability of the algorithm models and technical solutions. The method achieves an MIoU of 90.19 and a single batch time of 340 ms with a batch size of 8, which substantially reduces the runtime relative to a typical deep network structure like ResNet50.

Energy-Efficient Spiking Segmenter for Frame and Event-Based Images.

Semantic segmentation predicts dense pixel-wise semantic labels, which is crucial for autonomous environment perception systems. For applications on mobile devices, current research focuses on energy-efficient segmenters for both frame and event-based cameras. However, there is currently no artificial neural network (ANN) that can perform efficient segmentation on both types of images. This paper introduces spiking neural network (SNN, a bionic model that is energy-efficient when implemented on neuromorphic hardware) and develops a Spiking Context Guided Network (Spiking CGNet) with substantially lower energy consumption and comparable performance for both frame and event-based images. First, this paper proposes a spiking context guided block that can extract local features and context information with spike computations. On this basis, the directly-trained SCGNet-S and SCGNet-L are established for both frame and event-based images. Our method is verified on the frame-based dataset Cityscapes and the event-based dataset DDD17. On the Cityscapes dataset, SCGNet-S achieves comparable results to ANN CGNet with 4.85 × energy efficiency. On the DDD17 dataset, Spiking CGNet outperforms other spiking segmenters by a large margin.

Environment Modeling Method for Intelligent Robots Based on Binocular Vision

With the development of technology, the demand for intelligent mobile robots is growing steadily. Environment perception and modeling, are the key elements to achieve robot intelligence. Therefore, binocular stereo vision sensor is used as the core device to perform operations such as binocular camera calibration, obtaining binocular camera internal/external parameters, and distortion correction. On this basis, a multi-dimensional and all-round environment perception system is constructed, and the dynamic target detection results and 3D scene map reconstruction results are displayed dynamically in real time. The experimental results of binocular camera calibration show that the re-projection error lies in the sub-pixel interval, and the image distortion caused by the camera is basically eliminated to meet the experimental expectation of distortion correction; the experimental results of environment modeling show that the mean value of deviation between the object representation value of 3D scene map and the actual object real value is -0.041m, among which the minimum deviation value is 0.001m and the maximum deviation value is 0.532m, in The experimental results are excellent in the tasks of 3D scene map reconstruction and feature point matching. In addition, the storage space of 3D scene map is significantly reduced, and the real-time performance of the system is significantly better than the traditional method.

Development and Key Technologies of the Cognition system of Autonomous Driving

With the rapid development of automotive electronics technology and artificial intelligence, autonomous driving has become an important solution for future transportation challenges. As an integral component of autonomous driving systems, the intelligent cognitive system has emerged as a prominent area of technological research. This paper focuses on the development and key technologies of autonomous driving cognitive systems, based on published literature. The architecture of cognitive systems for autonomous driving is typically divided into two subsystems: the environmental perception system and the assessment and prediction system. Our study provides a detailed description of the methods used in both subsystems. We then conduct a comparative analysis and discussion to evaluate their strengths and limitations. Overall, we hope that this study will provide timely and valuable information to a rapidly evolving field concerning cognitive technology for autonomous vehicles.

ViF-GTAD: A new automotive dataset with ground truth for ADAS/AD development, testing, and validation

A new dataset for automated driving, which is the subject matter of this paper, identifies and addresses a gap in existing similar perception datasets. While most state-of-the-art perception datasets primarily focus on the provision of various onboard sensor measurements along with the semantic information under various driving conditions, the provided information is often insufficient since the object list and position data provided include unknown and time-varying errors. The current paper and the associated dataset describe the first publicly available perception measurement data that include not only the onboard sensor information from the camera, Lidar, and radar with semantically classified objects but also the high-precision ground-truth position measurements enabled by the accurate RTK-assisted GPS localization systems available on both the ego vehicle and the dynamic target objects. This paper provides insight on the capturing of the data, explicitly explaining the metadata structure and the content, as well as the potential application examples where it has been, and can potentially be, applied and implemented in relation to automated driving and environmental perception systems development, testing, and validation.

Evaluating impact of remote-access cyber-attack on lane changes for connected automated vehicles

Connected automated vehicles (CAVs) rely heavily on intelligent algorithms and remote sensors. If the control center or on-board sensors are under cyber-attack due to the security vulnerability of wireless communication, it can cause significant damage to CAVs or passengers. The primary objective of this study is to model cyber-attacked traffic flow and evaluate the impacts of cyber-attack on the traffic system filled with CAVs in a connected environment. Based on the analysis on environmental perception system and possible cyber-attacks on sensors, a novel lane-changing model for CAVs is proposed and multiple traffic scenarios for cyber-attacks are designed. The impact of the proportion of cyber-attacked vehicles and the severity of the cyber-attack on the lane-changing process is then quantitatively analyzed. The evaluation indexes include spatio-temporal evolution of average speed, spatial distribution of selected lane-changing gaps, lane-changing rate distribution, lane-changing preparation search time, efficiency and safety. Finally, the numerical simulation results show that the freeway traffic near an off-ramp is more sensitive to the proportion of cyber-attacked vehicles than to the severity of the cyber-attack. Also, when the traffic system is under cyber-attack, more unsafe back gaps are chosen for lane-changing, especially in the center lane. Therefore, more lane-changing maneuvers are concentrated on approaching the off-ramp, causing severe congestions and potential rear-end collisions. In addition, as the number of cyber-attacked vehicles and the severity of cyber-attacks increase, the road capacity and safety level will rapidly decrease. The results of this study can provide a theoretical basis for accident avoidance and efficiency improvement for the design of CAVs and management of automated highway systems.

Research on Environment Perception System of Quadruped Robots Based on LiDAR and Vision

Due to the high stability and adaptability, quadruped robots are currently highly discussed in the robotics field. To overcome the complicated environment indoor or outdoor, the quadruped robots should be configured with an environment perception system, which mostly contain LiDAR or a vision sensor, and SLAM (Simultaneous Localization and Mapping) is deployed. In this paper, the comparative experimental platforms, including a quadruped robot and a vehicle, with LiDAR and a vision sensor are established firstly. Secondly, a single sensor SLAM, including LiDAR SLAM and Visual SLAM, are investigated separately to highlight their advantages and disadvantages. Then, multi-sensor SLAM based on LiDAR and vision are addressed to improve the environmental perception performance. Thirdly, the improved YOLOv5 (You Only Look Once) by adding ASFF (adaptive spatial feature fusion) is employed to do the image processing of gesture recognition and achieve the human–machine interaction. Finally, the challenge of environment perception system for mobile robot based on comparison between wheeled and legged robots is discussed. This research provides an insight for the environment perception of legged robots.

IVP-YOLOv5: an intelligent vehicle-pedestrian detection method based on YOLOv5s

Computer vision is now vital in intelligent vehicle environment perception systems. However, real-time small-scale pedestrian detection in intelligent vehicle environment perception systems is still needs to be improved. This paper proposes an intelligent vehicle-pedestrian detection method based on YOLOv5s, named IVP-YOLOv5, to use in vehicle environment perception systems. Based on the network structure of YOLOv5s, we replaced BottleNeck CSP with Ghost-Bottleneck to reduce the complexity of processing feature maps while maintaining good detection performance. To reduce the error between the ground truth box and the predicted box, we apply Alpha-IoU as the bounding box loss function, improving pedestrian detection accuracy and robustness. We introduce the slicing-aided hyper inference (SAHI) strategy, which enables the lightweight backbone network to capture more detailed features of pedestrians by enlarging image pixels. Experiments on the BDD100 K dataset show that the proposed IVP-YOLOv5 achieves 67.1% AP and 18.5% APs of pedestrian detection, and the GFLOPs and the number of parameters are only 10.5 and 4.9M.

Framework for environment perception: Ensemble method for vision-based scene understanding algorithms in agriculture.

The safe and reliable operation of autonomous agricultural vehicles requires an advanced environment perception system. An important component of perception systems is vision-based algorithms for detecting objects and other structures in the fields. This paper presents an ensemble method for combining outputs of three scene understanding tasks: semantic segmentation, object detection and anomaly detection in the agricultural context. The proposed framework uses an object detector to detect seven agriculture-specific classes. The anomaly detector detects all other objects that do not belong to these classes. In addition, the segmentation map of the field is utilized to provide additional information if the objects are located inside or outside the field area. The detections of different algorithms are combined at inference time, and the proposed ensemble method is independent of underlying algorithms. The results show that combining object detection with anomaly detection can increase the number of detected objects in agricultural scene images.

MCS-YOLO: A Multiscale Object Detection Method for Autonomous Driving Road Environment Recognition

Object detection and recognition of road scenes are crucial tasks of the autonomous driving environmental perception system. The low inference speed and accuracy in object detection models hinder the development of autonomous driving technology. Searching for improvement of detection accuracy and speed is still a challenging task. For solving these problems, we proposed an MCS-YOLO algorithm. Firstly, a coordinate attention module is inserted into the backbone to aggregate the feature map’s spatial coordinate and cross-channel information. Then, we designed a multiscale small object detection structure to improve the recognition sensitivity of dense small object. Finally, we applied the Swin Transformer structure to the CNN to enable the network to focus on contextual spatial information. Conducting ablation study on the autonomous driving dataset BDD100K, MCS-YOLO algorithm achieves a mean average precision of 53.6% and a recall rate of 48.3%, which are 4.3% and 3.9% better than the YOLOv5s algorithm respectively. In addition, it can achieve real-time detection speed of 55 frames per second in a real scene. The results show that the MCS-YOLO algorithm is effective and superior in the task of automatic driving object detection.