Autonomous Driving Decision-making Based on the Combination of Deep Reinforcement Learning and Rule-based Controller

Abstract

Listen

Translate article

As autonomous vehicles begin to drive on the road, rational decision making is essential for driving safety and efficiency. The decision-making of autonomous vehicles is a difficult problem since it depends on the surrounding dynamic environment constraints and its own motion constraints. As the result of the combination of deep learning (DL) and reinforcement learning (RL), deep reinforcement learning (DRL) integrates DL's strong understanding of perception problems such as visual and semantic text, as well as the decision-making ability of RL. Hence, DRL can be used to solve complex problems in real scenarios. However, as an end-to-end method, DRL is inefficient and the final result tend to be poorly robust. Considering the usefulness of existing domain knowledge for autonomous vehicle decision-making, this paper uses domain knowledge to establish behavioral rules and combine rule-based behavior strategies with DRL methods, so that we can achieve efficient training of autonomous vehicle decision-making models and ensure the vehicle to chooses safe actions under unknown circumstances. First, the continuous decision-making problem of autonomous vehicles is modeled as a Markov decision process (MDP). Taking into account the influence of unknown intentions of other road vehicles on self-driving decisions, a recognition model of the behavioral intentions of other vehicles was established. Then, the linear dynamic model of the conventional vehicle is used to establish the relationship between the vehicle decision-making behavior and the motion trajectory. Finally, by designing the reward function of the MDP, we use a combination of RL and behavior rules-based controller, the expected driving behavior of the autonomous vehicle is obtained. In this paper, the simulation environment of scenes of intersections in urban roads and highways is established, and each situation is formalized as an RL problem. Meanwhile, a large number of numerical simulations were carried out, and the comparison of our method and the end-to-end form of DRL technology were discussed. "Due to its robust operation and high performance during bad weather conditions and overnight as well as the ability of using the Doppler Effect to measure directly the velocity of objects, the radar sensor is used in many application fields. Especially in automotive many radar sensors are used for the perception of the environment to increase the safety of the traffic. To increase the security level especially for vulnerable road users (VRU’s) like pedestrians or cyclists, radar sensors are used in driver assistance systems. Radar sensors are also used in the infrastructure, e.g. a commercial application is the detection of cars and pedestrians to manage traffic lights. Furthermore, radar sensors installed in the infrastructure are used in research projects for safeguarding future autonomous traffic. The object recognition and accuracy of radar-based sensing in the infrastructure can be increased by cooperating radar systems, which consist out of several sensors. This paper focus on the data fusion method of two radar sensors to increase the performance of detection and localization. For data fusion the high level cluster data of the two radar sensors are used as input data in a neuronal net (NN) structure. The results are compared to the localization obtained by using only a single radar sensor operating with an ordinary tracking algorithm. First, different models for chosen region of interests (ROI) and operating mode of cooperative sensors are developed and the data structure is discussed. In addition, the data are preprocessed with a coordinate transformation and time synchronization for both sensors, as well as the noise filtering to reduce the amount of clusters for the algorithm. Furthermore, three NN structures (CNN, DNN and LSTM) for static + dynamic objects and only dynamic objects are created, trained and discussed. Also, based on the results further improvements for the NN performance will be discussed."