Deep Inverse Reinforcement Learning Research Articles

Joint path planning and power allocation of a cellular-connected UAV using apprenticeship learning via deep inverse reinforcement learning

This paper investigates an interference-aware joint path planning and power allocation mechanism for a cellular-connected unmanned aerial vehicle (UAV) in a sparse suburban environment. The UAV’s goal is to fly from an initial point and reach a destination point by moving along the cells to guarantee the required quality of service (QoS). In particular, the UAV aims to maximize its uplink throughput and minimize interference to the ground user equipment (UEs) connected to neighboring cellular base stations (BSs), considering both the shortest path and limitations on flight resources. Expert knowledge is used to experience the scenario and define the desired behavior for the sake of the agent (i.e., UAV) training. To solve the problem, an apprenticeship learning method is utilized via inverse reinforcement learning (IRL) based on both Q-learning and deep reinforcement learning (DRL). The performance of this method is compared to learning from a demonstration technique called behavioral cloning (BC) using a supervised learning approach. Simulation and numerical results show that the proposed approach can achieve expert-level performance. We also demonstrate that, unlike the BC technique, the performance of our proposed approach does not degrade in unseen situations.

Map matching on low sampling rate trajectories through deep inverse reinforcement learning and multi-intention modeling

Analyzing freight vehicle movements using GPS trajectory data presents challenges due to environmental conditions and hardware limitations impacting data accuracy. Map matching, the process of aligning GPS signals with road networks, facilitates accurate route reconstruction. However, existing methods have limitations, particularly with low and ultra-low sampling rates. They often assume the shortest path between points, overlook historical data insights and neglect diverse driving behaviors, which may not align with real-world scenarios where shortest paths are not always optimal and different drivers exhibit varied behaviors. These limitations affect existing methods’ reliability, especially when we face low sampling rate trajectories. In this study, we propose multi-intention deep inverse reinforcement learning for map matching (MIDIRL) to address these challenges. MIDIRL integrates deep neural networks and multi-intention capturing mechanisms with inverse reinforcement learning to model complex driving preferences from historical trajectories, improving map matching accuracy, especially in ultra-low-frequency trajectories. Our experiments on real-world datasets demonstrate MIDIRL’s improved accuracy and efficiency of map matching compared to previous methods, even with limited training data.

Predicting Pedestrian Crossing Behavior at Unsignalized Mid-Block Crosswalks Using Maximum Entropy Deep Inverse Reinforcement Learning

Predicting Pedestrian Crossing Behavior at Unsignalized Mid-Block Crosswalks Using Maximum Entropy Deep Inverse Reinforcement Learning

Uncertainty-aware human-like driving policy learning with deep Bayesian inverse reinforcement learning

The application of deep reinforcement learning in driving policy learning for automated vehicles is limited by the difficulty of designing reward functions. Most existing inverse reinforcement learning (IRL) methods make a structure assumption on the reward function and omit the uncertainty of neural networks. In view of this, this paper proposes a novel deep Bayesian IRL method that addresses both the reward function designing and the uncertainty-measuring issues by learning an approximate posterior distribution over the reward function. Furthermore, we propose to train uncertainty-aware human-like driving policies by maximising the predicted reward and penalising its uncertainty. Finally, the proposed methods were validated in simulated highway driving scenarios. The results show that AVRL models uncertainty and learns reward functions significantly outperforming the existing IRL method applied in automated driving. It was also found that penalising the uncertainty of the reward function during policy training improves the success rate and human likeness of the learned policy.

Car-Following Behavior Modeling With Maximum Entropy Deep Inverse Reinforcement Learning

Car-Following Behavior Modeling With Maximum Entropy Deep Inverse Reinforcement Learning

GeoPM-DMEIRL: A deep inverse reinforcement learning security trajectory generation framework with serverless computing

Vehicle trajectory data is essential for traffic management and location-based services. However, the release of trajectories raises serious privacy concerns because they contain sensitive information, such as home addresses and workplaces, etc, making it indispensable to consider privacy protection when releasing trajectory data. It is urgent to reconcile data utility with privacy in trajectory generation calls for advanced methods that adhere to real-world traffic rules. Leveraging serverless computing for deep learning provides an efficient solution, bypassing infrastructure complexities to handle the scale trajectory data. In this paper, a Geo-Piecewise Mechanism Deep Maximum Entropy Inverse Reinforcement Learning (GeoPM-DMEIRL) is proposed for secure trajectory generation, which consists of three key components, namely GeoPM, A2C reinforcement learning and DMEIRL. Firstly, GeoPM is a novel local differential privacy mechanism that incorporates Geo-aware gridding techniques. The Piecewise Mechanism is used to perturb the user trajectory while ensuring that the perturbed trajectory conforms to real-world traffic rules. Secondly, an A2C reinforcement learning network is refined to train the optimal trajectory generation strategy. Thirdly, Deep Maximum Entropy Inverse Reinforcement Learning is improved to train the weights of the A2C reward function. Finally, real-world data are collected in the experiments and the serverless computing platform AWS Lambda is used for training the reinforcement learning models. Experimental results show that our proposed GeoPM-DMEIRL framework can effectively resist user re-identification attacks, which can improve the utility by an average of 54.605% and enhance the privacy by an average of 30.678%. Meanwhile, GeoPM-DMEIRL is able to maintain the utility of the data while protecting the privacy of the user’s trajectories.

Anomalous ride-hailing driver detection with deep transfer inverse reinforcement learning

The rapid expansion in group size of online ride-hailing drivers has made anomalous driver detection become a critical issue, which substantially affects the safety and operation aspects of ride-hailing services Existing studies mainly focus on the identification of abnormal trajectories, while none of them investigate anomalous driver detection. The former evaluates a specific trajectory, while the latter comprehensively evaluates a driver. The driving preferences of anomalous ride-hailing drivers differ from those of most ride-hailing drivers. Inverse reinforcement learning (IRL) mines the hidden preference of an agent from its observed behavior Although existing IRL models can mine the driver’s route preference from trajectories, they depend on massive trajectories. To deal with insufficient trajectories, we propose deep transfer inverse reinforcement learning, which reuses the partial pre-trained common reward network and transfers it to be a part of the personalized reward network. Then, we use an autoencoder network to reduce the dimension of personalized route preference and combine it with K-means clustering to distinguish all the drivers into two categories (i.e., normal and abnormal). Numerical experiments using ride-hailing trajectories in Chengdu, China are conducted to verify the effectiveness of our proposed approach. The results demonstrate that precision and recall of our proposed approach are 91.7% and 97.1%, respectively.

Advancements in Deep Reinforcement Learning and Inverse Reinforcement Learning for Robotic Manipulation: Toward Trustworthy, Interpretable, and Explainable Artificial Intelligence

Advancements in Deep Reinforcement Learning and Inverse Reinforcement Learning for Robotic Manipulation: Toward Trustworthy, Interpretable, and Explainable Artificial Intelligence

A Long-Term Actor Network for Human-Like Car-Following Trajectory Planning Guided by Offline Sample-Based Deep Inverse Reinforcement Learning

A Long-Term Actor Network for Human-Like Car-Following Trajectory Planning Guided by Offline Sample-Based Deep Inverse Reinforcement Learning

Danger-zone and maximum entropy deep inverse reinforcement learning for human-robot navigation

As a result of the development of artificial intelligence in recent years, scientists have gone further in the field of human-robot interaction (HRI), and one of the remaining problems is how to achieve a safe and human comfort-satisfying navigation when it comes to crowd-robot interaction (CRI). This article explores two existing deep learning reinforcement (DRL) methods, Danger-zone (DZ) and Maximum Entropy Deep Inverse Reinforcement Learning (MEDIRL). The former method proposes a Danger-Zone to predict the trajectory of pedestrians, using the DRL network to achieve obstacle avoidance and combining advanced algorithms with it to summarize the most effective combination. The latter method collects expert demonstrations in an uncontrolled environment. It uses DNN networks to predict human behavior and compares the results with the actual trajectories to demonstrate their validity. This article summarizes and evaluates these two methods. Moreover, this article also gives various outlooks on the direction of human-computer interaction.

AdaBoost-Bagging deep inverse reinforcement learning for autonomous taxi cruising route and speed planning

Taxi cruising route planning has attracted considerable attention, and relevant studies can be broadly categorized into three main streams: recommending one or multiple areas, providing a detailed cruising route, and deriving the optimal routing policy. However, these studies depend on accurate pick-up/drop-off information, and seldom pay attention to cruising speed planning. In view of the rapid development of autonomous taxis, this study proposes AdaBoost-Bagging maximum entropy deep inverse reinforcement learning to learn cruising policy from experienced taxi drivers’ trajectories. Moreover, we develop a trajectory-based self-attention bidirectional LSTM model to adjust cruising speeds on different roads. Numerical experiments using real taxi trajectories in Chengdu, China demonstrate the effectiveness of our approach in learning taxi drivers’ policies and improving taxis’ operational efficiency.

Towards more efficient and robust evaluation of sepsis treatment with deep reinforcement learning

BackgroundIn recent years, several studies have applied advanced AI methods, i.e., deep reinforcement learning, in discovering more efficient treatment policies for sepsis. However, due to a paucity of understanding of sepsis itself, the existing approaches still face a severe evaluation challenge, that is, how to properly evaluate the goodness of treatments during the learning process and the effectiveness of the final learned treatment policies.MethodsWe propose a deep inverse reinforcement learning with mini-tree model that integrates different aspects of factors into the reward formulation, including the critical factors in causing mortality and the key indicators in the existing sepsis treatment guidelines, in order to provide a more comprehensive evaluation of treatments during learning. A new off-policy evaluation method is then proposed to enable more robust evaluation of the learned policies by considering the weighted averaged value functions estimated until the current step.ResultsResults in the MIMIC-III dataset show that the proposed methods can achieve more efficient treatment policies with higher reliability compared to those used by the clinicians.ConclusionsA more sound and comprehensive evaluation of treatments of sepsis should consider the most critical factors in infulencing the mortality during treatment as well as those key indicators in the existing sepsis diagnosis guidelines.

A deep inverse reinforcement learning approach to route choice modeling with context-dependent rewards

Route choice modeling is a fundamental task in transportation planning and demand forecasting. Classical methods generally adopt the discrete choice model (DCM) framework with linear utility functions and high-level route characteristics. While several recent studies have started to explore the applicability of deep learning for route choice modeling, they are all path-based with relatively simple model architectures and require choice set generation. Existing link-based models can capture the dynamic nature of link choices within the trip without the need for choice set generation, but still assume linear relationships and link-additive features. To address these issues, this study proposes a general deep inverse reinforcement learning (IRL) framework for link-based route choice modeling, which is capable of incorporating diverse features (of the state, action and trip context) and capturing complex relationships. Specifically, we adapt an adversarial IRL model to the route choice problem for efficient estimation of context-dependent reward functions without value iteration. Experiment results based on taxi GPS data from Shanghai, China validate the superior prediction performance of the proposed model over conventional DCMs and other imitation learning baselines, even for destinations unseen in the training data. Further analysis shows that the model exhibits competitive computational efficiency and reasonable interpretability. The proposed methodology provides a new direction for future development of route choice models. It is general and can be adaptable to other route choice problems across different modes and networks.

Driver Intent-Based Intersection Autonomous Driving Collision Avoidance Reinforcement Learning Algorithm

With the rapid development of artificial intelligent technology, the deep learning method is widely applied to predict human driving intentions due to its relative accuracy of prediction, which is one of critical links for security guarantee in the distributed, mixed driving scenario. In order to sense the intention of human-driven vehicles and reduce the self-driving collision avoidance rate, an improved intention prediction method for human-driving vehicles based on unsupervised, deep inverse reinforcement learning is proposed. Firstly, a contrast discriminator module was proposed to extract richer features. Then, the residual module was created to overcome the drawbacks of gradient disappearance and network degradation with the increase in network layers. Furthermore, the dropout layer was generated to prevent the over-fitting phenomenon in the whole training process of the GRU network, so as to improve the generalization ability of the network model. Finally, abundant experiments were conducted on datasets to evaluate our proposed method. The pass rate of self-driving vehicles with conservative driver probabilities of p = 0.25, p = 0.4, and p = 0.6 improved by a maximum of 8%, 10%, and 3%, compared with the classical method LSTM and VAE + RNN. It indicates that the prediction results of our proposed method fit more with the basic structure of the given traffic scenario in a long-term prediction range, which verifies the effectiveness of our proposed method.

An Improved Multimodal Trajectory Prediction Method Based on Deep Inverse Reinforcement Learning

With the rapid development of artificial intelligence technology, the deep learning method has been introduced for vehicle trajectory prediction in the internet of vehicles, since it provides relative accurate prediction results, which is one of the critical links to guarantee security in the distributed mixed-driving scenario. In order to further enhance prediction accuracy by making full utilization of complex traffic scenes, an improved multimodal trajectory prediction method based on deep inverse reinforcement learning is proposed. Firstly, a fused dilated convolution module for better extracting raster features is introduced into the existing multimodal trajectory prediction network backbone. Then, a reward update policy with inferred goals is improved by learning the state rewards of goals and paths separately instead of original complex rewards, which can reduce the requirement for predefined goal states. Furthermore, a correction factor is introduced in the existing trajectory generator module, which can better generate diverse trajectories by penalizing trajectories with little difference. Abundant experiments on the current popular public dataset indicate that the prediction results of our proposed method are a better fit with the basic structure of the given traffic scenario in a long-term prediction range, which verifies the effectiveness of our proposed method.

Energy-Based Legged Robots Terrain Traversability Modeling via Deep Inverse Reinforcement Learning

This work reports ondeveloping a deep inverse reinforcement learning method for legged robots terrain traversability modeling that incorporates both exteroceptive and proprioceptive sensory data. Existing works use robot-agnostic exteroceptive environmental features or handcrafted kinematic features; instead, we propose to also learn robot-specific inertial features from proprioceptive sensory data for reward approximation in a single deep neural network. Incorporating the inertial features can improve the model fidelity and provide a reward that depends on the robot’s state during deployment. We train the reward network using the Maximum Entropy Deep Inverse Reinforcement Learning (MEDIRL) algorithm and propose simultaneously minimizing a trajectory ranking loss to deal with the suboptimality of legged robot demonstrations. The demonstrated trajectories are ranked by locomotion energy consumption, in order to learn an energy-aware reward function and a more energy-efficient policy than demonstration. We evaluate our method using a dataset collected by an MIT Mini-Cheetah robot and a Mini-Cheetah simulator. The code is publicly available. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref>

Multi-objective deep inverse reinforcement learning for weight estimation of objectives

Multi-objective deep inverse reinforcement learning for weight estimation of objectives

Personalized route recommendation for ride-hailing with deep inverse reinforcement learning and real-time traffic conditions

Personalized route recommendation aims to recommend routes based on users’ route preference. The vast amount of GPS trajectories tracking driving behavior has made deep learning, especially inverse reinforcement learning (IRL), a popular choice for personalized route recommendation. However, current IRL studies assume that the traffic condition is static and approximate the expected state visitation frequencies to update the neural network. This study improves the IRL to recommend personalized routes considering real-time traffic conditions. We also improve the expected state visitation frequency calculation based on characteristics of ride-hailing and taxi trajectories to calculate the gradient of the neural network. In addition, the graph attention network is employed to capture the spatial dependencies between road segments. Numerical experiments using real ride-hailing trajectories in Chengdu, China validate our model. At last, a statistical test is conducted, and route preferences reflected by the same driver’s empty trajectories and occupied trajectories are found to have significant differences.

AdaBoost maximum entropy deep inverse reinforcement learning with truncated gradient

Studying the representational capacity of neural networks to learn nonlinear rewards is necessary in a complex and nonlinear environment. Over recent years, the maximum entropy deep inverse reinforcement learning algorithm (ME-DIRL) has been increasingly applied to the learning of nonlinear rewards. However, under cases of limited and imbalanced expert demonstration data, complex calculations, or overfitting, the learning nonlinear rewards remains a challenging problem. A novel ME-DIRL with AdaBoost algorithm (AME-DIRL) is our proposed solution. The focus of AME-DIRL is to utilize the AdaBoost algorithm. This combines multiple ME-DIRL processes to form a strong learner and thus overcome the imbalance of the data set. Furthermore, to deal with the complex calculations in AME-DIRL, a truncated gradient (TG) method is applied for getting the sparse rewards obtained by the strong learner, thus reducing the model complexity. To prevent overfitting, a correction factor is then added to the linear combination of weak learners. AME-DIRL models the relationship between input features and output rewards. Rewards are approximated by means of a convolutional neural network (CNN) with scaled exponential linear units (SELUs). Numerical results indicate that our proposed AME-DIRL shows higher accuracy in learning rewards when compared with several classical inverse reinforcement learning (IRL) algorithms.

Inducing structure in reward learning by learning features

Reward learning enables robots to learn adaptable behaviors from human input. Traditional methods model the reward as a linear function of hand-crafted features, but that requires specifying all the relevant features a priori, which is impossible for real-world tasks. To get around this issue, recent deep Inverse Reinforcement Learning (IRL) methods learn rewards directly from the raw state but this is challenging because the robot has to implicitly learn the features that are important and how to combine them, simultaneously. Instead, we propose a divide-and-conquer approach: focus human input specifically on learning the features separately, and only then learn how to combine them into a reward. We introduce a novel type of human input for teaching features and an algorithm that utilizes it to learn complex features from the raw state space. The robot can then learn how to combine them into a reward using demonstrations, corrections, or other reward learning frameworks. We demonstrate our method in settings where all features have to be learned from scratch, as well as where some of the features are known. By first focusing human input specifically on the feature(s), our method decreases sample complexity and improves generalization of the learned reward over a deep IRL baseline. We show this in experiments with a physical 7-DoF robot manipulator, and in a user study conducted in a simulated environment.