Inverse Reinforcement Learning Framework Research Articles

Reconstructing human gaze behavior from EEG using inverse reinforcement learning

Decoding eye movements from non-invasive electroencephalography (EEG) data is a challenging yet vital task for both scientific and practical purposes, especially for identifying neurodegenerative disorders like Alzheimer’s disease (AD). Our research tackles this complexity by adapting inverse reinforcement learning (IRL), a machine learning method, to infer decision-making strategies from observed behaviors. We implement this to understand the processes driving eye direction and movements during diverse cognitive tasks, providing new insights into this field. Our paper begins with a detailed description of the procedures for collecting and preprocessing EEG data related to gaze behavior. We then elaborate on the development of an IRL framework designed to predict the spatial and temporal dynamics of eye movements (scanpaths) in participants engaged in cognitive tasks of varying complexity. Our model is tailored to accommodate the complexities inherent in neural signals and the stochastic nature of human gaze patterns. Our research findings underscore IRL’s effectiveness in precisely forecasting gaze patterns based on a combination of EEG and image data. The correlation between the model’s predictions and the actual gaze behavior observed in controlled experiments reinforces the utility of IRL in cognitive neuroscience research. Notably, our IRL-EEG models demonstrated superior performance, especially in more complex cognitive tasks. We further delve into the implications of our results for enhancing the understanding of neural mechanisms that govern gaze behavior.

A state-based inverse reinforcement learning approach to model activity-travel choices behavior with reward function recovery

Behaviorally oriented activity-travel choices (ATC) modeling is a principal part of travel demand analysis. Traditional econometric and rule-based methods require explicit model structures and complex domain knowledge. While several recent studies used machine learning models, especially adversarial inverse reinforcement learning (IRL) models, to learn potential ATC patterns with less expert-designed settings, they lack a clear representation of rational ATC behavior. In this study, we propose a data-driven IRL framework based on the maximum causal approach to minimize f-divergences between expert and agent state marginal distributions, which provides a more sample-efficient measurement. In addition, we specify a separate state-only reward function and derive an analytical gradient of the f-divergence objective with respect to reward parameters to ensure good convergences. The method can recover a stationary reward function, which assures the agent to get close to the expert behavior when training from scratch. We validate the proposed model using cellular signaling data from Chongqing, China by comparing with baseline models (behavior cloning, policy-based, and reward-based models) in aspects of policy performance comparison, reward recovery, and reward transfer tasks. The experiment results indicate that the proposed model outperforms existing methods and is relatively less sensitive to the number of expert demonstrations. Qualitative analyses are provided on the fundamental ATC preferences on different features given the reward function recovered from the observed mobility trajectories, and on the learning behaviors under different choices of f-divergence.

Meta-IRLSOT++: A meta-inverse reinforcement learning method for fast adaptation of trajectory prediction networks

Recent research on pedestrian trajectory prediction based on deep learning has made significant progress. However, the previous methods do not deeply explore the relationship between scene information and trajectory. Moreover, the training model requires massive data and only targets specific scenes, so the prediction performance is poor when the new scene samples are limited. To solve the above problems, a meta-inverse reinforcement learning-based framework, dubbed Meta-IRLSOT++, is proposed in this work. IRLSOT++ improves the solid baseline IRLSOT, and the main contributions are as follows: (1) An inverse reinforcement learning framework is introduced to explore the trajectory–scene association to achieve task-level scene understanding, enhancing the correlation between trajectory and scene. The trajectory heat maps describing pedestrians dynamic characteristics are leveraged to align better the trajectory and scene semantic segmentation predicted by TopFormer. (2) A Transformer-based encoder–decoder network is proposed to fuse the trajectory and plan cues for better generating multi-modal trajectories with multi-head attention. The social graph attention and pseudo-oracle predictor are introduced to capture pedestrians’ social interactions and intent states, ameliorating trajectory prediction performance. (3) Meta-learning is leveraged to achieve collaborative training based on IRLSOT++ to improve the model generalization in new scenes, enabling fast adaptation of trajectory prediction. Experimental results on the Stanford Drone Dataset (SDD) indicate that IRLSOT++ can precisely forecast future trajectories by improving IRLSOT, decreasing Average Displacement Error/ Final Displacement Error (ADE/FDE) values from 9.66/13.05 to 8.36/12.28. Moreover, the meta-learning strategy quickly adapts IRLSOT++ to the new scene, achieving ADE/FDE values of 7.31/11.02 and 15.42/26.55 when Tpred is set to 12 and 24, respectively. Both quantitative and qualitative experimental results show that Meta-IRLSOT++ has accuracy and fast adaptability, which is beneficial for real-world trajectory prediction tasks.

An inverse reinforcement learning framework with the Q-learning mechanism for the metaheuristic algorithm

A reward function is learned from the expert examples by inverse reinforcement learning (IRL), which is more reliable than an artificial method. The moth–flame optimization algorithm (MFO), which is based on the navigation mechanism of a moth flying at night, has been extensively employed to address the complex optimization problem. An inverse reinforcement learning framework with the Q-learning mechanism (IRLMFO) is designed to strengthen the performance of the MFO algorithm in a large-scale real-parameter optimization problem. The right strategy is chosen by the Q-learning mechanism, using historical data provided by the relevant approach in the strategy pool, which stores strategies that include diverse functions. The competition mechanism is designed to strengthen the exploitation capability of the IRLMFO algorithm. The performance of the IRLMFO is verified on the benchmark test suite in CEC 2017. Experimental results illustrate that the IRLMFO outperforms state-of-the-art algorithms.

Driving Behavior Modeling Using Naturalistic Human Driving Data With Inverse Reinforcement Learning

Driving behavior modeling is of great importance for designing safe, smart, and personalized autonomous driving systems. In this paper, an internal reward function-based driving model that emulates the human’s decision-making mechanism is utilized. To infer the reward function parameters from naturalistic human driving data, we propose a structural assumption about human driving behavior that focuses on discrete latent driving intentions. It converts the continuous behavior modeling problem to a discrete setting and thus makes maximum entropy inverse reinforcement learning (IRL) tractable to learn reward functions. Specifically, a polynomial trajectory sampler is adopted to generate candidate trajectories considering high-level intentions and approximate the partition function in the maximum entropy IRL framework. An environment model considering interactive behaviors among the ego and surrounding vehicles is built to better estimate the generated trajectories. The proposed method is applied to learn personalized reward functions for individual human drivers from the NGSIM highway driving dataset. The qualitative results demonstrate that the learned reward functions are able to explicitly express the preferences of different drivers and interpret their decisions. The quantitative results reveal that the learned reward functions are robust, which is manifested by only a marginal decline in proximity to the human driving trajectories when applying the reward function in the testing conditions. For the testing performance, the personalized modeling method outperforms the general modeling approach, significantly reducing the modeling errors in human likeness (a custom metric to gauge accuracy), and these two methods deliver better results compared to other baseline methods. Moreover, it is found that predicting the response actions of surrounding vehicles and incorporating their potential decelerations caused by the ego vehicle are critical in estimating the generated trajectories, and the accuracy of personalized planning using the learned reward functions relies on the accuracy of the forecasting model.

Utilizing B-Spline Curves and Neural Networks for Vehicle Trajectory Prediction in an Inverse Reinforcement Learning Framework

The ability to accurately predict vehicle trajectories is essential in infrastructure-based safety systems that aim to identify critical events such as near-crash situations and traffic violations. In a connected environment, important information about these critical events can be communicated to road users or the infrastructure to avoid or mitigate potential crashes. Intersections require special attention in this context because they are hotspots for crashes and involve numerous and complex interactions between road users. In this work, we developed an advanced machine learning method for trajectory prediction using B-spline curve representations of vehicle trajectories and inverse reinforcement learning (IRL). B-spline curves were used to represent vehicle trajectories; a neural network model was trained to predict the coefficients of these curves. A conditional variational autoencoder (CVAE) was used to generate candidate trajectories from these predicted coefficients. These candidate trajectories were then ranked according to a reward function that was obtained by training an IRL model on the (spline smoothed) vehicle trajectories and the surroundings of the vehicles. In our experiments we found that the neural network model outperformed a Kalman filter baseline and the addition of the IRL ranking module further improved the performance of the overall model.

CGAIL: Conditional Generative Adversarial Imitation Learning—An Application in Taxi Drivers’ Strategy Learning

Smart passenger-seeking strategies employed by taxi drivers contribute not only to drivers’ incomes, but also higher quality of service passengers received. Therefore, understanding taxi drivers’ behaviors and learning the good passenger-seeking strategies are crucial to boost taxi drivers’ well-being and public transportation quality of service. However, we observe that drivers’ preferences of choosing which area to find the next passenger are diverse and dynamic across locations and drivers. It is hard to learn the location-dependent preferences given the partial data (i.e., an individual driver’s trajectory may not cover all locations). In this article, we make the first attempt to develop conditional generative adversarial imitation learning (cGAIL) model, as a unifying collective inverse reinforcement learning framework that learns the driver’s decision-making preferences and policies by transferring knowledge across taxi driver agents and across locations. Our evaluation results on three months of taxi GPS trajectory data in Shenzhen, China, demonstrate that the driver’s preferences and policies learned from cGAIL are on average 36.2 percent more accurate than those learned from other state-of-the-art baseline approaches.

Using Inverse Reinforcement Learning with Real Trajectories to Get More Trustworthy Pedestrian Simulations

Reinforcement learning is one of the most promising machine learning techniques to get intelligent behaviors for embodied agents in simulations. The output of the classic Temporal Difference family of Reinforcement Learning algorithms adopts the form of a value function expressed as a numeric table or a function approximator. The learned behavior is then derived using a greedy policy with respect to this value function. Nevertheless, sometimes the learned policy does not meet expectations, and the task of authoring is difficult and unsafe because the modification of one value or parameter in the learned value function has unpredictable consequences in the space of the policies it represents. This invalidates direct manipulation of the learned value function as a method to modify the derived behaviors. In this paper, we propose the use of Inverse Reinforcement Learning to incorporate real behavior traces in the learning process to shape the learned behaviors, thus increasing their trustworthiness (in terms of conformance to reality). To do so, we adapt the Inverse Reinforcement Learning framework to the navigation problem domain. Specifically, we use Soft Q-learning, an algorithm based on the maximum causal entropy principle, with MARL-Ped (a Reinforcement Learning-based pedestrian simulator) to include information from trajectories of real pedestrians in the process of learning how to navigate inside a virtual 3D space that represents the real environment. A comparison with the behaviors learned using a Reinforcement Learning classic algorithm (Sarsa(λ)) shows that the Inverse Reinforcement Learning behaviors adjust significantly better to the real trajectories.

Regularising neural networks for future trajectory prediction via inverse reinforcement learning framework

Predicting distant future trajectories of agents in a dynamic scene is challenging because the future trajectory of an agent is affected not only by their past trajectory but also the scene contexts. To tackle this problem, the authors propose a model based on recurrent neural networks, and a novel method for training this model. The proposed model is based on an encoder–decoder architecture where the encoder encodes inputs (past trajectory and scene context information), while the decoder produces a future trajectory from the context vector given by the encoder. To make the proposed model better utilise the scene context information, the authors let the encoder predict the positions in the past trajectory and a reward function evaluate the positions along with the scene context information generated by the positions. The reward function, which is simultaneously trained with the proposed model, plays the role of a regulariser for the model during the simultaneous training. The authors evaluate the proposed model on several public benchmark datasets. The experimental results show that the prediction performance of the proposed model is greatly improved by the proposed regularisation method, which outperforms the‐state‐of‐the‐art models in terms of accuracy.

A knowledge-driven layered inverse reinforcement learning approach for recognizing human intents

ABSTRACT There is a rising trend in exploring the capability of inverse reinforcement learning (IRL) in high dimensional demonstrations. Our aim is to recognise human intents from video data within an IRL framework. For this, we present a two-layered maximum likelihood IRL model. The usefulness of knowledge representation (KR) schemes and availability of advisors at different layers is exploited through this model. Two main aspects are addressed: a. the importance of having abstract high-level information to the IRL framework in terms of semantic object affordance and b. deductively exploring the utility of a state at different temporal abstractions. The effectiveness of the proposed model has been evaluated with the help of standard Cornell Activity Dataset (CAD-120).

Robust Bayesian Inverse Reinforcement Learning with Sparse Behavior Noise

Inverse reinforcement learning (IRL) aims to recover the reward function underlying a Markov Decision Process from behaviors of experts in support of decision-making. Most recent work on IRL assumes the same level of trustworthiness of all expert behaviors, and frames IRL as a process of seeking reward function that makes those behaviors appear (near)-optimal. However, it is common in reality that noisy expert behaviors disobeying the optimal policy exist, which may degrade the IRL performance significantly. To address this issue, in this paper, we develop a robust IRL framework that can accurately estimate the reward function in the presence of behavior noise. In particular, we focus on a special type of behavior noise referred to as sparse noise due to its wide popularity in real-world behavior data. To model such noise, we introduce a novel latent variable characterizing the reliability of each expert action and use Laplace distribution as its prior. We then devise an EM algorithm with a novel variational inference procedure in the E-step, which can automatically identify and remove behavior noise in reward learning. Experiments on both synthetic data and real vehicle routing data with noticeable behavior noise show significant improvement of our method over previous approaches in learning accuracy, and also show its power in de-noising behavior data.