Inverse Reinforcement Learning Research Articles

Analyzing the Strategy of Propaganda using Inverse Reinforcement Learning: Evidence from the 2022 Russian Invasion of Ukraine

The 2022 Russian invasion of Ukraine was accompanied by a large-scale, pro-Russian propaganda campaign on social media. However, the strategy behind the dissemination of propaganda has remained unclear, particularly how the online discourse was strategically shaped by the propagandists' community. Here, we analyze the strategy of the Twitter/X community using an inverse reinforcement learning (IRL) approach. Specifically, IRL allows us to model online behavior as a Markov decision process, where the goal is to infer the underlying reward structure that guides propagandists when interacting with users with a supporting or opposing stance toward the invasion. Thereby, we aim to understand empirically whether and how between-user interactions are strategically used to promote the proliferation of Russian propaganda. For this, we leverage a large-scale dataset with 349,455 posts with pro-Russian propaganda from 132,131 users. We show that bots and humans follow a different strategy: bots respond predominantly to pro-invasion messages, suggesting that they seek to drive virality; in contrast, messages indicating opposition primarily elicit responses from humans, suggesting that they tend to engage in critical discussions. To the best of our knowledge, this is the first study analyzing the strategy behind propaganda from the 2022 Russian invasion of Ukraine through the lens of IRL.

Variable impedance control on contact-rich manipulation of a collaborative industrial mobile manipulator: An imitation learning approach

Variable impedance control (VIC) endows robots with the ability to adjust their compliance, enhancing safety and adaptability in contact-rich tasks. However, determining suitable variable impedance parameters for specific tasks remains challenging. To address this challenge, this paper proposes an imitation learning-based VIC policy that employs observations integrated with RGBD and force/torque (F/T) data enabling a collaborative mobile manipulator to execute contact-rich tasks by learning from human demonstrations. The VIC policy is learned through training the robot in a customized simulation environment, utilizing an inverse reinforcement learning (IRL) algorithm. High-dimensional demonstration data is encoded by integrating a 16-layer convolutional neural network (CNN) into the IRL environment. To minimize the sim-to-real gap, contact dynamic parameters in the training environment are calibrated. Then, the learning-based VIC policy is comprehensively trained in the customized environment and its transferability is validated through an industrial production case involving a high precision peg-in-hole task using a collaborative mobile manipulator. The training and testing results indicate that the proposed imitation learning-based VIC policy ensures robust performance for contact-rich tasks.

Eliciting patient preferences and predicting behaviour using Inverse Reinforcement Learning for telehealth use in outpatient clinics.

Non-attendance (NA) causes additional burden on the outpatient services due to clinician time and other resources being wasted, and it lengthens wait lists for patients. Telehealth, the delivery of health services remotely using digital technologies, is one promising approach to accommodate patient needs while offering more flexibility in outpatient services. However, there is limited evidence about whether offering telehealth consults as an option can change NA rates, or about the preferences of hospital outpatients for telehealth compared to in-person consults. We model patient preferences with a Maximum Entropy Inverse Reinforcement Learning (IRL) behaviour model, allowing for the calculation of general population- and demographic-specific relative preferences for consult modality. The aim of this research is to use real-world data to model patient preferences for consult modality using Maximum Entropy IRL behaviour model. Retrospective data were collected from an immunology outpatient clinic associated with a large metropolitan hospital in Brisbane, Australia. We used IRL with the Maximum Entropy behaviour model to learn outpatient preferences for appointment modality (telehealth or in-person) and to derive demographic predictors of attendance or NA. IRL models patients as decision making agents interacting sequentially over multiple time-steps, allowing for present actions to impact future outcomes, unlike previous models applied in this domain. We found statistically significant (α = 0.05) within-group preferences for telehealth consult modality in privately paying patients, patients who both identify as First Nations individuals and those who do not, patients aged 50-60, who did not require an interpreter, for the general population, and for the female population. We also found significant within-group preferences for in-person consult modality for patients who require an interpreter and for patients younger than 30. Using the Maximum Entropy IRL sequential behaviour model, our results agree with previous evidence that non-attendance can be reduced when telehealth is offered in outpatient clinics. Our results complement previous studies using non-sequential modelling methodologies. Our preference and NA prediction results may be useful to outpatient clinic administrators to tailor services to specific patient groups, such as scheduling text message consult reminders if a given patient is predicted to be more likely to NA.

Human-like Decision Making for autonomous lane change driving: a hybrid inverse reinforcement learning with game-theoretical vehicle interaction model

The development of automated driving vehicles aims to provide safer, comfortable, and more efficient mobility options. However, the decision-making control of autonomous vehicles still embraces limitations on human performance mimicry. These limitations become particularly evident in complex and unfamiliar driving scenarios, where weak decisionmaking abilities and poor adaptation of vehicle behaviour are prominent issues. This paper proposes a game-theoretic decisionmaking algorithm for human-like driving in the vehicle lane change scenario. Firstly, an inverse reinforcement learning (IRL) model is used to quantitatively analyse the lane change trajectories of the natural driving dataset, establishing the human-like human cost function. Subsequently, joint safety, comfort to build the comprehensive decision cost function. Use the combined decision cost function to conduct a non-cooperative game of vehicle lane changing decision to solve the optimal decision of host vehicle lane changing. The host vehicle lane-changing decision problem is formulated as a Stackelberg game optimization problem. To verify the feasibility and effectiveness of the algorithm proposed in this study, a lane change test scenario has been established. Firstly, we analyse the human-like decision-making model derived by the maximum entropy inverse reinforcement learning algorithm to verify the effectiveness and robustness of the IRL algorithm. Secondly, the human-like game decisionmaking algorithm in this paper is validated by conducting an interactive lane-changing experiment with obstacle vehicles of different driving styles. The experimental results prove that the human-like driving decision-making model proposed in this study can make lane-changing behaviours in line with human driving patterns in lane-changing scenarios of expressway.

Comprehensive Preview Decision Method for Direction and Speed of Intelligent Vehicle Based on Rules and Learning

Decision-making is a key part of autonomous driving systems. Human-like decision-making in complex scenarios is important for enhancing drivers’ trust in and acceptance of autonomous driving systems. In order to improve their human-like features and their adaptability to complex scenarios, this paper proposes a comprehensive preview decision method for direction and speed based on rules and learning. First, a decision-making structure including maneuvering ability judgment, motion trajectory prediction, safety judgment, legitimacy judgment, comprehensive performance evaluation, and parameters learning is presented. Second, a learning method for the longitudinal safety distance threshold based on a BP neural network is introduced. Third, inverse reinforcement learning (IRL) is used to learn the comprehensive performance evaluation weight coefficient for different driving styles. Finally, the DJI AD4CHE dataset is processed and used to train the parameters. Car-following and lane-changing scenarios are simulated to verify the effectiveness of decision-making. The simulation results show that the proposed method can reflect human-like decision-making in multiple scenarios.

Residual integral inverse reinforcement learning for intelligent self-healing control of unknown systems with actuator faults

Residual integral inverse reinforcement learning for intelligent self-healing control of unknown systems with actuator faults

Inverse reinforcement learning methods for linear differential games

In this study, we considered the problem of inverse reinforcement learning or estimating the cost function of expert players in multi-player differential games. We proposed two online data-driven solutions for linear–quadratic games that are applicable to systems that fulfill a specific dimension criterion or whose unknown matrices in the cost function conform to a diagonal condition. The first method, which is partially model-free, utilizes the trajectories of expert agents to solve the problem. The second method is entirely model-free and employs the trajectories of both expert and learner agents. We determined the conditions under which the solutions are applicable and identified the necessary requirements for the collected data. We conducted numerical simulations to establish the effectiveness of the proposed methods.

Quantum Imitation Learning.

Despite remarkable successes in solving various complex decision-making tasks, training an imitation learning (IL) algorithm with deep neural networks (DNNs) suffers from the high-computational burden. In this work, we propose quantum IL (QIL) with a hope to utilize quantum advantage to speed up IL. Concretely, we develop two QIL algorithms: quantum behavioral cloning (Q-BC) and quantum generative adversarial IL (Q-GAIL). Q-BC is trained with a negative log-likelihood (NLL) loss in an offline manner that suits extensive expert data cases, whereas Q-GAIL works in an inverse reinforcement learning (IRL) scheme, which is online, on-policy, and is suitable for limited expert data cases. For both QIL algorithms, we adopt variational quantum circuits (VQCs) in place of DNNs for representing policies, which are modified with data reuploading and scaling parameters to enhance the expressivity. We first encode classical data into quantum states as inputs, then perform VQCs, and finally measure quantum outputs to obtain control signals of agents. Experiment results demonstrate that both Q-BC and Q-GAIL can achieve comparable performance compared to classical counterparts, with the potential of quantum speedup. To our knowledge, we are the first to propose the concept of QIL and conduct pilot studies, which paves the way for the quantum era.

Мультиагентное тестирование на проникновение на основе AIRL

This paper explores the application of a multi-agent approach based on the Adversarial Inverse Reinforcement Learning (AIRL) method for penetration testing in information systems. Theoretical aspects of multi-agent AIRL are discussed, including the ability to model complex, multi-stage attacks, coordinate agent actions, and learn with partial observability, which accounts for limitations in information access. The practical application of this approach will demonstrate its effectiveness in identifying vulnerabilities, providing a deeper and more accurate security analysis.

Joint path planning and power allocation of a cellular-connected UAV using apprenticeship learning via deep inverse reinforcement learning

This paper investigates an interference-aware joint path planning and power allocation mechanism for a cellular-connected unmanned aerial vehicle (UAV) in a sparse suburban environment. The UAV’s goal is to fly from an initial point and reach a destination point by moving along the cells to guarantee the required quality of service (QoS). In particular, the UAV aims to maximize its uplink throughput and minimize interference to the ground user equipment (UEs) connected to neighboring cellular base stations (BSs), considering both the shortest path and limitations on flight resources. Expert knowledge is used to experience the scenario and define the desired behavior for the sake of the agent (i.e., UAV) training. To solve the problem, an apprenticeship learning method is utilized via inverse reinforcement learning (IRL) based on both Q-learning and deep reinforcement learning (DRL). The performance of this method is compared to learning from a demonstration technique called behavioral cloning (BC) using a supervised learning approach. Simulation and numerical results show that the proposed approach can achieve expert-level performance. We also demonstrate that, unlike the BC technique, the performance of our proposed approach does not degrade in unseen situations.

Modeling framework of human driving behavior based on Deep Maximum Entropy Inverse Reinforcement Learning

Driving behavior modeling is extremely crucial for designing safe, intelligent, and personalized autonomous driving systems. In this paper, a modeling framework based on Markov Decision Processes (MDPs) is introduced that emulates drivers’ decision-making processes. The framework combines the Deep Maximum Entropy Inverse Reinforcement Learning (Deep MEIRL) and a reinforcement learning algorithm-proximal strategy optimization (PPO). A neural network structure is customized for Deep MEIRL, which uses the velocity of the ego vehicle, the pedestrian position, the velocity of surrounding vehicles, the lateral distance, the surrounding vehicles’ type, and the distance to the crosswalk to recover the nonlinear reward function. The dataset of drone-based video footage is collected in Xi’an (China) to train and validate the framework. The outcomes demonstrate that Deep MEIRL-PPO outperforms traditional modeling frameworks (Maximum Entropy Inverse Reinforcement Learning (MEIRL) - PPO) in modeling and predicting human driving behavior. Specifically, in predicting human driving behavior, Deep MEIRL-PPO outperforms MEIRL-PPO by 50.71% and 43.90% on the basis of the MAE and HD, respectively. Furthermore, it is discovered that Deep MEIRL-PPO accurately learns the behavior of human drivers avoiding potential conflicts when lines of sight are occluded. This research can contribute to aiding self-driving vehicles in learning human driving behavior and avoiding unforeseen risks.

A Game-Theoretic Framework for Generic Second-Order Traffic Flow Models Using Mean Field Games and Adversarial Inverse Reinforcement Learning

A traffic system can be interpreted as a multiagent system, wherein vehicles choose the most efficient driving approaches guided by interconnected goals or strategies. This paper aims to develop a family of mean field games (MFG) for generic second-order traffic flow models (GSOM), in which cars control individual velocity to optimize their objective functions. GSOMs do not generally assume that cars optimize self-interested objectives, so such a game-theoretic reinterpretation offers insights into the agents’ underlying behaviors. In general, an MFG allows one to model individuals on a microscopic level as rational utility-optimizing agents while translating rich microscopic behaviors to macroscopic models. Building on the MFG framework, we devise a new class of second-order traffic flow MFGs (i.e., GSOM-MFG), which control cars’ acceleration to ensure smooth velocity change. A fixed-point algorithm with fictitious play technique is developed to solve GSOM-MFG numerically. In numerical examples, different traffic patterns are presented under different cost functions. For real-world validation, we further use an inverse reinforcement learning approach (IRL) to uncover the underlying cost function on the next-generation simulation (NGSIM) data set. We formulate the problem of inferring cost functions as a min-max game and use an apprenticeship learning algorithm to solve for cost function coefficients. The results show that our proposed GSOM-MFG is a generic framework that can accommodate various cost functions. The Aw Rascle and Zhang (ARZ) and Light-Whitham-Richards (LWR) fundamental diagrams in traffic flow models belong to our GSOM-MFG when costs are specified. History: This paper has been accepted for the Transportation Science Special Issue on ISTTT25 Conference. Funding: X. Di is supported by the National Science Foundation [CAREER Award CMMI-1943998]. E. Iacomini is partially supported by the Italian Research Center on High-Performance Computing, Big Data and Quantum Computing (ICSC) funded by MUR Missione 4-Next Generation EU (NGEU) [Spoke 1 “FutureHPC & BigData”]. C. Segala and M. Herty thank the Deutsche Forschungsgemeinschaft (DFG) for financial support [Grants 320021702/GRK2326, 333849990/IRTG-2379, B04, B05, and B06 of 442047500/SFB1481, HE5386/18-1,19-2,22-1,23-1,25-1, ERS SFDdM035; Germany’s Excellence Strategy EXC-2023 Internet of Production 390621612; and Excellence Strategy of the Federal Government and the Länder]. Support through the EU DATAHYKING is also acknowledged. This work was also funded by the DFG [TRR 154, Mathematical Modelling, Simulation and Optimization Using the Example of Gas Networks, Projects C03 and C05, Project No. 239904186]. Moreover, E. Iacomini and C. Segala are members of the Indam GNCS (Italian National Group of Scientific Calculus).

Map matching on low sampling rate trajectories through deep inverse reinforcement learning and multi-intention modeling

Analyzing freight vehicle movements using GPS trajectory data presents challenges due to environmental conditions and hardware limitations impacting data accuracy. Map matching, the process of aligning GPS signals with road networks, facilitates accurate route reconstruction. However, existing methods have limitations, particularly with low and ultra-low sampling rates. They often assume the shortest path between points, overlook historical data insights and neglect diverse driving behaviors, which may not align with real-world scenarios where shortest paths are not always optimal and different drivers exhibit varied behaviors. These limitations affect existing methods’ reliability, especially when we face low sampling rate trajectories. In this study, we propose multi-intention deep inverse reinforcement learning for map matching (MIDIRL) to address these challenges. MIDIRL integrates deep neural networks and multi-intention capturing mechanisms with inverse reinforcement learning to model complex driving preferences from historical trajectories, improving map matching accuracy, especially in ultra-low-frequency trajectories. Our experiments on real-world datasets demonstrate MIDIRL’s improved accuracy and efficiency of map matching compared to previous methods, even with limited training data.

Human-in-the-Loop Behavior Modeling via an Integral Concurrent Adaptive Inverse Reinforcement Learning.

One goal of artificial intelligence (AI) research is to teach machines how to learn from humans, such that they can perform a certain task in a natural human-like way. In this article, an online adaptive inverse reinforcement learning (IRL) approach to human behavior modeling is proposed to enhance machine intelligence for a class of linear human-in-the-loop (HiTL) systems using the state data only, where the human behavior is described by a linear quadratic optimal control model with an unknown weighting matrix for the quadratic cost function. First, an integral concurrent adaptive law is developed to learn the human feedback gain matrix online using the demonstrated state data only, which removes the persistent excitation (PE) conditions required by traditional adaptive estimation approaches and thus is more in line with real applications. Then, with the learned feedback gain matrix, the IRL problem is formulated as a linear matrix inequality (LMI) optimization problem, which can be efficiently solved to retrieve the weighting matrix of the human cost function. Finally, a simulation example is provided to illustrate the effectiveness of the proposed approach.

A New AI Approach by Acquisition of Characteristics in Human Decision-Making Process

Planning and decision making are closely interconnected processes that often occur in tandem, influence and informing each other. Planning usually precedes decision making in the chronological sequence, and it can be viewed as a strategy to make decisions. A comprehensive planning or decision strategy can facilitate effective decisions. Thus, understanding and learning human decision-making strategies has drawn intensive attention from the AI community. For example, applying planning algorithms into reinforcement leaning (RL) can simulate the consequence of different actions and select optimal decisions based on learned models, while inverse reinforcement learning (IRL) learns a reward function and policy from expert demonstration and applies them into new scenarios. Most of these methods work based on learning human decision strategies by using modeling of a Markovian decision-making process (MDP). In this paper, we argue that the property of MDP is not fit for human decision-making processes in the real-world and it is insufficient to capture human decision strategies. To tackle this challenge, we propose a new approach to identify the characteristics of human decision-making processes as a decision map, where the decision strategy is defined by the probability distribution of human decisions that are adaptive to the dynamic changes in the environment. The proposed approach was inspired by imitation learning (IL) but with fundamental differences: (a) Instead of aiming to learn an optimal policy based on expert’s demonstrations, we aimed to estimate the distribution of decisions of any group of people. (b) Instead of modeling the environment by an MDP, we used an ambiguity probability model to consider the uncertainty of each decision. (c) The participant trajectory was obtained by categorizing each decision of a participant to a certain cluster based on the commonness in the distribution of decisions. The result shows a feasible way to capture human long-term decision dependency, which provides a complement to the existing machine learning methods for understanding and learning human decision strategies.

A two-stage framework for parking search behavior prediction through adversarial inverse reinforcement learning and transformer

Parking scenarios are spatially dense and have a lot of interactions, making predicting vehicles’ search behavior crucial and challenging for autonomous driving. Existing data-driven prediction methods struggle to determine vehicles’ intents and consider the surrounding environment accurately. This study proposes a novel two-stage framework for parking search behavior prediction, involving parking intent and vehicle trajectory predictions based on imitation learning and deep learning. First, we develop an adversarial inverse reinforcement learning model for parking search intent (PSI-AIRL) learning from measured trajectory data in an actual parking lot. Then, we design an integrated convolutional neural network (CNN) and transformer model to forecast vehicle trajectory using historical observations and the predicted parking search intents. This two-stage framework achieves parking search intent prediction by applying global information about the parking lot while improving vehicle trajectory prediction’s accuracy and robustness. Finally, the experiments are conducted on the Dragon Lake Parking (DLP) dataset to compare our framework with state-of-the-art models. The results show that our model outperforms other baseline models in accuracy for parking intent and vehicle trajectory predictions. Moreover, our model shows exceptional accuracy and robustness in predicting across diverse parking scenarios.

Modelling two-dimensional driving behaviours at unsignalised intersection using multi-agent imitation learning

Traffic at unsignalised intersection usually involves complex interactions. It is critical to explicitly model the two-dimensional driving behaviours and capture the interactions among vehicles. However, little effort has been made to develop microscopic traffic simulation models at unsignalised intersections, which can generate realistic vehicle trajectories. To fill this gap, this study aims to develop a two-dimensional data-driven simulation model at an unsignalised intersection based on multi-agent imitation learning for generating realistic trajectories of vehicles crossing the intersection and macroscopic traffic characteristics. We propose a multi-agent adversarial inverse reinforcement learning model with four policies (MA-AIRL-4) to separately learn the driving behaviours with different directions (East-bound, West-bound, South-bound, and North–bound) by capturing the interactions between the vehicles. Using the vehicle trajectories data extracted from an unsignalised intersection of the open-source Interaction Dataset, we evaluate the performance of the proposed model by comparing it with several bench-marking models, i.e., MA-AIRL-2 model with two policies (i.e., one policy for West-East and North-South direction, respectively), MA-AIRL-1 model with one policy for all vehicles, three corresponding multi-agent generative adversarial imitation learning (MA-GAIL) models with four policies, two policies and one policy, respectively, and long short-term memory (LSTM) model. The results demonstrate the superiority of the proposed MA-AIRL-4 model in generating accurate vehicle trajectories and reproducing realistic speed distributions. The interpretation of the recovered reward function also indicates the effective learning of the proposed model and provides insights into the learned behaviours.

Autonomous navigation of catheters and guidewires in mechanical thrombectomy using inverse reinforcement learning

PurposeAutonomous navigation of catheters and guidewires can enhance endovascular surgery safety and efficacy, reducing procedure times and operator radiation exposure. Integrating tele-operated robotics could widen access to time-sensitive emergency procedures like mechanical thrombectomy (MT). Reinforcement learning (RL) shows potential in endovascular navigation, yet its application encounters challenges without a reward signal. This study explores the viability of autonomous guidewire navigation in MT vasculature using inverse reinforcement learning (IRL) to leverage expert demonstrations.MethodsEmploying the Simulation Open Framework Architecture (SOFA), this study established a simulation-based training and evaluation environment for MT navigation. We used IRL to infer reward functions from expert behaviour when navigating a guidewire and catheter. We utilized the soft actor-critic algorithm to train models with various reward functions and compared their performance in silico.ResultsWe demonstrated feasibility of navigation using IRL. When evaluating single- versus dual-device (i.e. guidewire versus catheter and guidewire) tracking, both methods achieved high success rates of 95% and 96%, respectively. Dual tracking, however, utilized both devices mimicking an expert. A success rate of 100% and procedure time of 22.6 s were obtained when training with a reward function obtained through ‘reward shaping’. This outperformed a dense reward function (96%, 24.9 s) and an IRL-derived reward function (48%, 59.2 s).ConclusionsWe have contributed to the advancement of autonomous endovascular intervention navigation, particularly MT, by effectively employing IRL based on demonstrator expertise. The results underscore the potential of using reward shaping to efficiently train models, offering a promising avenue for enhancing the accessibility and precision of MT procedures. We envisage that future research can extend our methodology to diverse anatomical structures to enhance generalizability.

Simultaneously learning intentions and preferences during physical human-robot cooperation

The advent of collaborative robots allows humans and robots to cooperate in a direct and physical way. While this leads to amazing new opportunities to create novel robotics applications, it is challenging to make the collaboration intuitive for the human. From a system’s perspective, understanding the human intentions seems to be one promising way to get there. However, human behavior exhibits large variations between individuals, such as for instance preferences or physical abilities. This paper presents a novel concept for simultaneously learning a model of the human intentions and preferences incrementally during collaboration with a robot. Starting out with a nominal model, the system acquires collaborative skills step-by-step within only very few trials. The concept is based on a combination of model-based reinforcement learning and inverse reinforcement learning, adapted to fit collaborations in which human and robot think and act independently. We test the method and compare it to two baselines: one that imitates the human and one that uses plain maximum entropy inverse reinforcement learning, both in simulation and in a user study with a Franka Emika Panda robot arm.

Inverse Reinforcement Learning With Graph Neural Networks for Full-Dimensional Task Offloading in Edge Computing

Inverse Reinforcement Learning With Graph Neural Networks for Full-Dimensional Task Offloading in Edge Computing