Inverse Reinforcement Learning Approach Research Articles

How do active road users act around autonomous vehicles? An inverse reinforcement learning approach

The inevitable impact of autonomous vehicles (AV) on traffic safety is becoming a reality with the progressive deployment of these vehicles in different parts of the world. Still, many questions linger in the minds of road users that will share the road and interact with these AVs on a daily basis. To answer some of these questions, this study utilized recently collected real-world AV data from the United States, with the focus mainly targeting active road users. Specifically, the 1,492 h of recorded trips were processed to extract AV-pedestrian and AV-cyclist interactions of different movement types. The interactions were then investigated to gain a better understanding of the active road users’ behavior, while excluding any interactions that involved intervention from the AVs’ human test drivers. Through deep maximum entropy inverse reinforcement learning (DME-IRL), the reward functions describing the utility of active road users were retrieved and assessed for five interaction scenarios, including parallel, opposing, crossing, and turning (left and right) interactions. In addition, the policies developed as part of the solution were used to simulate the behavior of active road users and validate the resulting conflicts in terms of safety and evasive actions. Overall, the utilized approach demonstrated high accuracy in mimicking the interaction behavior of active road users when encountering an AV, with 81–84% accuracy in predicting evasive actions in parallel and opposing interactions and 12–17% mean absolute error for safety indicators in crossing and turning interactions. In addition, the resulting reward functions provided reliable insight onto the preferences and considerations of the active road users in these situations. Overall, cyclists tend to be less cautious around AVs compared to pedestrians, while the AVs tend to slow down and leave sufficient distance from all active road users in most cases. Still, the robotic behavior of the AVs, which can sometimes be inconsistent, leads to risky evasive actions by the active road users, which can affect other road users in busy intersections.

Uncovering drone intentions using control physics informed machine learning

Unmanned Autonomous Vehicle (UAV) or drones are increasingly used across diverse application areas. Uncooperative drones do not announce their identity/flight plans and can pose a potential risk to critical infrastructures. Understanding drone’s intention is important to assigning risk and executing countermeasures. Intentions are often intangible and unobservable, and a variety of tangible intention classes are often inferred as a proxy. However, inference of drone intention classes using observational data alone is inherently unreliable due to observational and learning bias. Here, we developed a control-physics informed machine learning (CPhy-ML) that can robustly infer across intention classes. The CPhy-ML couples the representation power of deep learning with the conservation laws of aerospace models to reduce bias and instability. The CPhy-ML achieves a 48.28% performance improvement over traditional trajectory prediction methods. The reward inference results outperforms conventional inverse reinforcement learning approaches, decreasing the root mean squared spectral norm error from 3.3747 to 0.3229.

Modelling motorized and non-motorized vehicle conflicts using multiagent inverse reinforcement learning approach

Microsimulation models are effective for analysing road users’ interaction behaviour and assessing different facilities’ performance. However, only a few studies have developed simulation models for studying motorized and non-motorized vehicles conflicts. This is likely due to mixed traffic’s complexity and heterogeneity and the difficulty in accurately capturing road users’ avoidance maneuver. This study aims to adopt a multiagent simulation model to replicate road users’ microscopic behaviour and collision avoidance mechanisms in traffic conflict scenarios. Road users’ reward functions are recovered by the multiagent inverse reinforcement learning approach. The multiagent Actor-Critic deep learning algorithm is used to predict road users’ evasive action and assess their optimal policies. The findings demonstrate that the multiagent simulation model provides highly accurate predictions of road users’ trajectories and collision avoidance strategies. Furthermore, the results demonstrate a strong correlation between the predicted traffic conflict indicator from the simulated trajectories and that from the actual trajectories.

Trajectory modeling via random utility inverse reinforcement learning

We consider the problem of modeling trajectories of drivers in a road network from the perspective of inverse reinforcement learning. Cars are detected by sensors placed on sparsely distributed points on the street network of a city. As rational agents, drivers are trying to maximize some reward function unknown to an external observer. We apply the concept of random utility from econometrics to model the unknown reward function as a function of observed and unobserved features. In contrast to current inverse reinforcement learning approaches, we do not assume that agents act according to a stochastic policy; rather, we assume that agents act according to a deterministic optimal policy and show that randomness in data arises because the exact rewards are not fully observed by an external observer. We introduce the concept of extended state to cope with unobserved features and develop a Markov decision process formulation of drivers decisions. We present theoretical results which guarantee the existence of solutions and show that maximum entropy inverse reinforcement learning is a particular case of our approach. Finally, we illustrate Bayesian inference on model parameters through a case study with real trajectory data from a large city in Brazil.

A state-based inverse reinforcement learning approach to model activity-travel choices behavior with reward function recovery

Behaviorally oriented activity-travel choices (ATC) modeling is a principal part of travel demand analysis. Traditional econometric and rule-based methods require explicit model structures and complex domain knowledge. While several recent studies used machine learning models, especially adversarial inverse reinforcement learning (IRL) models, to learn potential ATC patterns with less expert-designed settings, they lack a clear representation of rational ATC behavior. In this study, we propose a data-driven IRL framework based on the maximum causal approach to minimize f-divergences between expert and agent state marginal distributions, which provides a more sample-efficient measurement. In addition, we specify a separate state-only reward function and derive an analytical gradient of the f-divergence objective with respect to reward parameters to ensure good convergences. The method can recover a stationary reward function, which assures the agent to get close to the expert behavior when training from scratch. We validate the proposed model using cellular signaling data from Chongqing, China by comparing with baseline models (behavior cloning, policy-based, and reward-based models) in aspects of policy performance comparison, reward recovery, and reward transfer tasks. The experiment results indicate that the proposed model outperforms existing methods and is relatively less sensitive to the number of expert demonstrations. Qualitative analyses are provided on the fundamental ATC preferences on different features given the reward function recovered from the observed mobility trajectories, and on the learning behaviors under different choices of f-divergence.

An Incremental Inverse Reinforcement Learning Approach for Motion Planning with Separated Path and Velocity Preferences

Humans often demonstrate diverse behaviors due to their personal preferences, for instance, related to their individual execution style or personal margin for safety. In this paper, we consider the problem of integrating both path and velocity preferences into trajectory planning for robotic manipulators. We first learn reward functions that represent the user path and velocity preferences from kinesthetic demonstration. We then optimize the trajectory in two steps, first the path and then the velocity, to produce trajectories that adhere to both task requirements and user preferences. We design a set of parameterized features that capture the fundamental preferences in a pick-and-place type of object transportation task, both in the shape and timing of the motion. We demonstrate that our method is capable of generalizing such preferences to new scenarios. We implement our algorithm on a Franka Emika 7-DoF robot arm and validate the functionality and flexibility of our approach in a user study. The results show that non-expert users are able to teach the robot their preferences with just a few iterations of feedback.

A deep inverse reinforcement learning approach to route choice modeling with context-dependent rewards

Route choice modeling is a fundamental task in transportation planning and demand forecasting. Classical methods generally adopt the discrete choice model (DCM) framework with linear utility functions and high-level route characteristics. While several recent studies have started to explore the applicability of deep learning for route choice modeling, they are all path-based with relatively simple model architectures and require choice set generation. Existing link-based models can capture the dynamic nature of link choices within the trip without the need for choice set generation, but still assume linear relationships and link-additive features. To address these issues, this study proposes a general deep inverse reinforcement learning (IRL) framework for link-based route choice modeling, which is capable of incorporating diverse features (of the state, action and trip context) and capturing complex relationships. Specifically, we adapt an adversarial IRL model to the route choice problem for efficient estimation of context-dependent reward functions without value iteration. Experiment results based on taxi GPS data from Shanghai, China validate the superior prediction performance of the proposed model over conventional DCMs and other imitation learning baselines, even for destinations unseen in the training data. Further analysis shows that the model exhibits competitive computational efficiency and reasonable interpretability. The proposed methodology provides a new direction for future development of route choice models. It is general and can be adaptable to other route choice problems across different modes and networks.

Estimating Link Flows in Road Networks With Synthetic Trajectory Data Generation: Inverse Reinforcement Learning Approach

While traffic volume data from loop detectors have been the common data source for link flow estimation, the detectors only cover a subset of links. These days, other data sources such as vehicle trajectory data collected from vehicle tracking sensors are also incorporated. However, trajectory data are often sparse in that the observed trajectories only represent a small subset of the whole population, where the exact route sampling rate is unknown and may vary over space and time. In this paper, we develop a method that leverage these two limited data sources to enhance link flow estimation. This study proposes a novel generative modelling framework, where we formulate a vehicle’s link-to-link movements as a sequential decision-making problem using the Markov Decision Process framework. We propose an Inverse Reinforcement Learning-based method, based on which synthetic population vehicle trajectories can be generated to estimate link flows across the whole network. The proposed method ensures the generated population vehicle trajectories are consistent with the observed traffic volume and trajectory data. The proposed generative modelling framework is compared to two existing methods in a synthetic road network and validated in a real road network.

Modeling Motorcyclist–Pedestrian Near Misses: A Multiagent Adversarial Inverse Reinforcement Learning Approach

Modeling Motorcyclist–Pedestrian Near Misses: A Multiagent Adversarial Inverse Reinforcement Learning Approach

An Inverse Reinforcement Learning Approach for Customizing Automated Lane Change Systems

Vehicle automation seeks to enhance road safety and improve the driving experience. However, a standard system does not account for variations in users and driving conditions. Customizing vehicle automation based on users’ preferences aims to improve the user experience and adoption of the technologies. This study introduces a systematic paradigm that starts with naturalistic driving data to identify the driving behaviors and styles for a customized automated lane change system. The driving behaviors are first extracted using Multivariate Functional Principal Component Analysis (MFPCA) with minimum prior expert knowledge. The driving styles are identified by clustering the extracted driving behaviors. An Inverse Reinforcement Learning (IRL) algorithm is then used to train the automated lane change system from grouped demonstrations of the identified driving styles to capture the preferences of a group of drivers with a similar driving style. The performance of the proposed customized automated lane change system is compared to (1) a non-customized system trained on all the sample trips, (2) customized systems built on expert-coded reward functions, and (3) customized systems trained using a Generative Adversarial Imitation Learning (GAIL) algorithm. The results show that our method outperforms all the other systems with respect to the prediction accuracy of the lane change actions. Additionally, our method gains insights on the representative behaviors of different driving styles to enable customization of automated lane change systems.

Inverse Resource Rational Based Stochastic Driver Behavior Model

Human drivers have limited and time-varying cognitive resources when making decisions in real-world traffic scenarios, which often leads to unique and stochastic behaviors that can not be explained by perfect rationality assumption, a widely accepted premise in modeling driving behaviors that presume drivers rationally make decisions to maximize their own rewards under all circumstances. To explicitly address this disadvantage, this study presents a novel driver behavior model that aims to capture the resource rationality and stochasticity of the human driver's behaviors in realistic longitudinal driving scenarios. The resource rationality principle can provide a theoretic framework to better understand the human cognition processes by modeling human's internal cognitive mechanisms as utility maximization subject to cognitive resource limitations, which can be represented as finite and time-varying preview horizons in the context of driving. An inverse resource rational-based stochastic inverse reinforcement learning approach (IRR-SIRL) is proposed to learn a distribution of the planning horizon and cost function of the human driver with a given series of human demonstrations. A nonlinear model predictive control (NMPC) with a time-varying horizon approach is used to generate driver-specific trajectories by using the learned distributions of the planning horizon and the cost function of the driver. The simulation experiments are carried out using human demonstrations gathered from the driver-in-the-loop driving simulator. The results reveal that the proposed inverse resource rational-based stochastic driver model can address the resource rationality and stochasticity of human driving behaviors in a variety of realistic longitudinal driving scenarios.

Customizing skills for assistive robotic manipulators, an inverse reinforcement learning approach with error-related potentials

Robotic assistance via motorized robotic arm manipulators can be of valuable assistance to individuals with upper-limb motor disabilities. Brain-computer interfaces (BCI) offer an intuitive means to control such assistive robotic manipulators. However, BCI performance may vary due to the non-stationary nature of the electroencephalogram (EEG) signals. It, hence, cannot be used safely for controlling tasks where errors may be detrimental to the user. Avoiding obstacles is one such task. As there exist many techniques to avoid obstacles in robotics, we propose to give the control to the robot to avoid obstacles and to leave to the user the choice of the robot behavior to do so a matter of personal preference as some users may be more daring while others more careful. We enable the users to train the robot controller to adapt its way to approach obstacles relying on BCI that detects error-related potentials (ErrP), indicative of the user’s error expectation of the robot’s current strategy to meet their preferences. Gaussian process-based inverse reinforcement learning, in combination with the ErrP-BCI, infers the user’s preference and updates the obstacle avoidance controller so as to generate personalized robot trajectories. We validate the approach in experiments with thirteen able-bodied subjects using a robotic arm that picks up, places and avoids real-life objects. Results show that the algorithm can learn user’s preference and adapt the robot behavior rapidly using less than five demonstrations not necessarily optimal.

Visual Navigation Using Inverse Reinforcement Learning and an Extreme Learning Machine

In this paper, we focus on the challenges of training efficiency, the designation of reward functions, and generalization in reinforcement learning for visual navigation and propose a regularized extreme learning machine-based inverse reinforcement learning approach (RELM-IRL) to improve the navigation performance. Our contributions are mainly three-fold: First, a framework combining extreme learning machine with inverse reinforcement learning is presented. This framework can improve the sample efficiency and obtain the reward function directly from the image information observed by the agent and improve the generation for the new target and the new environment. Second, the extreme learning machine is regularized by multi-response sparse regression and the leave-one-out method, which can further improve the generalization ability. Simulation experiments in the AI-THOR environment showed that the proposed approach outperformed previous end-to-end approaches, thus, demonstrating the effectiveness and efficiency of our approach.

Multi‐objective electrical demand response policy generation considering customer response behaviour learning: An enhanced inverse reinforcement learning approach

Demand response (DR) is an effective load management method. To attract customers to participate, DR policies need to both satisfy customers' individual DR habits and be economically profitable. However, customers’ individual DR habits are hard to be formulated with few hypotheses when other objectives are simultaneously considered. To tackle this challenge, a novel DR behavioural learning method is proposed. We learn customers’ DR habits by an inverse reinforcement learning (IRL) method to reduce the subjectivity in DR model formulation. Meanwhile, in contrast to traditional learning-based methods, the proposed method can adapt to multiple DR objectives more than just following customers’ DR habits, like obtaining higher economic revenues. Additionally, we consider the diversity and changes of customer DR behaviour patterns and offer an enhancement for the proposed DR behavioural learning method via building a DR pattern clustering and inference module. The proposed method can work with customer-side energy storage systems to diversify the DR policies and make the DR behaviours more flexible. Case studies show the proposed method can reduce about 10–20% behavioural learning deviations than the compared model-based methods, while daily charges of the proposed method can be further reduced by over 4% than the compared supervised-learning-based methods.

Markov-game modeling of cyclist-pedestrian interactions in shared spaces: A multi-agent adversarial inverse reinforcement learning approach

Understanding and modeling road user dynamics and their microscopic interaction behaviour at shared space facilities are curial for several applications including safety and performance evaluations. Despite the multi-agent nature of road user interactions, the majority of previous studies modeled their interactions as a single-agent modeling framework, i.e., considering the other interaction agents as part of the passive environment. However, this assumption is unrealistic and could limit the model's accuracy and transferability in non-stationary road user environments. This study proposes a novel Multi-Agent Adversarial Inverse Reinforcement Learning approach (MA-AIRL) to model and simulate road user interactions at shared space facilities. Unlike the traditional game-theoretic framework that models multi-agent systems as a single time-step payoff, the proposed approach is based on Markov Games (MG) which models road users' sequential decisions concurrently. Moreover, the proposed model can handle bounded rationality agents, e.g., limited information access, through the implementation of the Logistic Stochastic Best Response Equilibrium (LSBRE) solution concept. The proposed algorithm recovers road users' multi-agent reward functions using adversarial deep neural network discriminators and estimates their optimal policies using Multi-agent Actor-Critic with Kronecker factors (MACK) deep reinforcement learning. Data from three shared space locations in Vancouver, BC and New York City, New York are used in this study. The model's performance is compared to a baseline single-agent Gaussian Process Inverse Reinforcement Learning (GPIRL). The results show that the multi-agent modeling framework led to a significantly more accurate prediction of road users’ behaviour and their evasive action mechanisms. Moreover, the recovered reward functions based on the single-agent modeling approach failed to capture the equilibrium solution concept similar to the multi-agent approach.

Microscopic modeling of cyclists interactions with pedestrians in shared spaces: a Gaussian process inverse reinforcement learning approach

This study presents a microsimulation-oriented framework for modeling cyclists' interactions with pedestrians in shared spaces. The objectives of this study are to 1) infer how cyclists in head-on and crossing interactions rationally assess and make guidance decisions of acceleration and yaw rate, and 2) use advanced Artificial Intelligent (AI) techniques to model road-user interactions. The Markov Decision Process modeling framework is used to account for road-user rationality and intelligence. Road user trajectories from three shared spaces in North America are extracted by means of computer-vision algorithms. Inverse Reinforcement Learning (IRL) algorithms are utilized to recover continuous linear and nonlinear Gaussian-Process (GP) reward-functions (RFs). Deep Reinforcement Learning is used to estimate optimal cyclist policies. Results demonstrated that the GP-RF captures the more complex interaction behaviour and accounts for road-user heterogeneity. The GP-RF led to more consistent inferences of road-users behaviour and accurate predictions of their trajectories compared with the linear RF.

Social Trajectory Planning for Urban Autonomous Surface Vessels

In this article, we propose a trajectory planning algorithm that enables autonomous surface vessels to perform socially compliant navigation in a city's canal. The key idea behind the proposed algorithm is to adopt an optimal control formulation in which the deviation of movements of the autonomous vessel from nominal movements of human-operated vessels is penalized. Consequently, given a pair of origin and destination points, it finds vessel trajectories that resemble those of human-operated vessels. To formulate this, we adopt kernel density estimation (KDE) to build a nominal movement model of human-operated vessels from a prerecorded trajectory dataset, and use a Kullback-Leibler control cost to measure the deviation of the autonomous vessel's movements from the model. We establish an analogy between our trajectory planning approach and the maximum entropy inverse reinforcement learning (MaxEntIRL) approach to explain how our approach can learn the navigation behavior of human-operated vessels. On the other hand, we distinguish our approach from the MaxEntIRL approach in that it does not require well-defined bases, often referred to as features, to construct its cost function as required in many of inverse reinforcement learning approaches in the trajectory planning context. Through experiments using a dataset of vessel trajectories collected from the automatic identification system, we demonstrate that the trajectories generated by our approach resemble those of human-operated vessels and that using them for canal navigation is beneficial in reducing head-on encounters between vessels and improving navigation safety.

On the use of the policy gradient and Hessian in inverse reinforcement learning

Reinforcement Learning (RL) is an effective approach to solve sequential decision making problems when the environment is equipped with a reward function to evaluate the agent’s actions. However, there are several domains in which a reward function is not available and difficult to estimate. When samples of expert agents are available, Inverse Reinforcement Learning (IRL) allows recovering a reward function that explains the demonstrated behavior. Most of the classic IRL methods, in addition to expert’s demonstrations, require sampling the environment to evaluate each reward function, that, in turn, is built starting from a set of engineered features. This paper is about a novel model-free IRL approach that does not require to specify a function space where to search for the expert’s reward function. Leveraging on the fact that the policy gradient needs to be zero for an optimal policy, the algorithm generates an approximation space for the reward function, in which a reward is singled out employing a second-order criterion. After introducing our approach for finite domains, we extend it to continuous ones. The empirical results, on both finite and continuous domains, show that the reward function recovered by our algorithm allows learning policies that outperform those obtained with the true reward function, in terms of learning speed.

A knowledge-driven layered inverse reinforcement learning approach for recognizing human intents

ABSTRACT There is a rising trend in exploring the capability of inverse reinforcement learning (IRL) in high dimensional demonstrations. Our aim is to recognise human intents from video data within an IRL framework. For this, we present a two-layered maximum likelihood IRL model. The usefulness of knowledge representation (KR) schemes and availability of advisors at different layers is exploited through this model. Two main aspects are addressed: a. the importance of having abstract high-level information to the IRL framework in terms of semantic object affordance and b. deductively exploring the utility of a state at different temporal abstractions. The effectiveness of the proposed model has been evaluated with the help of standard Cornell Activity Dataset (CAD-120).

Customizing Skills for Assistive Robotic Manipulators: An Inverse Reinforcement Learning Approach with Error-Related Potentials

Customizing Skills for Assistive Robotic Manipulators: An Inverse Reinforcement Learning Approach with Error-Related Potentials