Inverse Reinforcement Learning Algorithm Research Articles

Underactuated MSV path following control via stable adversarial inverse reinforcement learning

Model-based control approaches are inadequate to solve the marine surface vehicle (MSV) path-following problem, especially under adverse environments. To effectively deal with the MSV path-following problem, model-free deep reinforcement learning (DRL) based methods have been developed. However, defining an efficient reward function for DRL in path following tasks is rather difficult. Providing expert demonstration is often easier than designing effective reward functions. Thus, we propose a model-free stable adversarial inverse reinforcement learning (SAIRL) algorithm that only adopts the state of MSV and reconstructs the reward function from the expert demonstration. The SAIRL algorithm is designed to guarantee the prescribed MSV path following accuracy and training stability. It utilizes an alternative loss function and dual-discriminator framework to dissolve the issue of policy collapse, which arises due to the vanishing gradient of the discriminator. Simulations and experiments have validated that the SAIRL algorithm outperforms other baseline algorithms in terms of path-following accuracy and stability of convergence.

Bridging the Gap Between Self-Report and Behavioral Laboratory Measures: A Real-Time Driving Task With Inverse Reinforcement Learning

A major challenge in assessing psychological constructs such as impulsivity is the weak correlation between self-report and behavioral task measures that are supposed to assess the same construct. To address this issue, we developed a real-time driving task called the “highway task,” in which participants often exhibit impulsive behaviors mirroring real-life impulsive traits captured by self-report questionnaires. Here, we show that a self-report measure of impulsivity is highly correlated with performance in the highway task but not with traditional behavioral task measures of impulsivity (47 adults aged 18–33 years). By integrating deep neural networks with an inverse reinforcement learning (IRL) algorithm, we inferred dynamic changes of subjective rewards during the highway task. The results indicated that impulsive participants attribute high subjective rewards to irrational or risky situations. Overall, our results suggest that using real-time tasks combined with IRL can help reconcile the discrepancy between self-report and behavioral task measures of psychological constructs.

Inverse Q-Learning Using Input-Output Data.

This article addresses the problem of learning the objective function of linear discrete-time systems that use static output-feedback (OPFB) control by designing inverse reinforcement learning (RL) algorithms. Most of the existing inverse RL methods require the availability of states and state-feedback control from the expert or demonstrated system. In contrast, this article considers inverse RL in a more general case where the demonstrated system uses static OPFB control with only input-output measurements available. We first develop a model-based inverse RL algorithm to reconstruct an input-output objective function of a demonstrated discrete-time system using its system dynamics and the OPFB gain. This objective function infers the demonstrations and OPFB gain of the demonstrated system. Then, an input-output Q -function is built for the inverse RL problem upon the state reconstruction technique. Given demonstrated inputs and outputs, a data-driven inverse Q -learning algorithm reconstructs the objective function without the knowledge of the demonstrated system dynamics or the OPFB gain. This algorithm yields unbiased solutions even though exploration noises exist. Convergence properties and the nonunique solution nature of the proposed algorithms are studied. Numerical simulation examples verify the effectiveness of the proposed methods.

Federated Inverse Reinforcement Learning for Smart ICUs With Differential Privacy

Clinical decision-making models have been developed to support therapeutic interventions based on medical data from either a single hospital or multiple hospitals. However, models based on multi-hospital data require collaboration among hospitals to integrate local data, which can result in information leakage and violate patient privacy. To address this challenge, we propose a novel approach that combines federated learning with inverse reinforcement learning to create an efficient medical decision-making support tool while preserving patient privacy. Our approach uses an inverse reinforcement learning algorithm with differential privacy to train a neural network-based agent on local data containing clinician trajectories, which learns a private treatment policy by observing patients’ conditions. Additionally, we integrate federated learning into the proposed algorithm to learn a global optimal action policy collaboratively among various smart ICUs, overcoming data limitations at each hospital. We evaluate our approach using real-world medical data and demonstrate that it achieves superior performance in a distributed manner.

A novel Congestion Control algorithm based on inverse reinforcement learning with parallel training

With the growing impact of the Internet, computer network communication has become an essential component for various industries. Congestion Control (CC) algorithms serve as the backbone of network communication, significantly affecting network quality. However, designing a CC algorithm that performs optimally across diverse network environments presents substantial challenges. The task of redesigning and optimizing CC algorithms for specific network environments demands both expert experience and a substantial workforce. In this paper, we proposed an inverse reinforcement learning (IRL) algorithm that can use expert data to guide the self-optimization of the CC model in specific network environments. To enhance model training efficiency, we propose a parallel training framework and a visualization analysis tool, enabling distributed training and real-time control level analysis. In the experiments, we assess the performance of 16 algorithms across 3 network scenarios using Pantheon. Our IRL model achieves the optimal level of network performance in satellite network, enhancing throughput by 10%–23%. For delay performance, it ranks 2nd in wired network and achieves a 21%–67% improvement over traditional TCP algorithms in regular network.

A Mutual Information-Based Assessment of Reverse Engineering on Rewards of Reinforcement Learning

Rewards are critical hyperparameters in reinforcement learning (RL), since in most cases different reward values will lead to greatly different performance. Due to their commercial value, RL rewards become the target of reverse engineering by the inverse reinforcement learning (IRL) algorithm family. Existing efforts typically utilize two metrics to measure the IRL performance: the expected value difference and the mean reward loss, which we call them EVD and MRL respectively. Unfortunately, in some cases, EVD and MRL can give completely opposite results, due to MRL focusing on whole state-space rewards while EVD only considering partly sampled rewards. Such situation naturally rises to one fundamental question: whether current metrics and assessment are sufficient and accurate for more general use. Thus, in this paper, based on the metric called normalized mutual information of reward clusters (C-NMI) we propose a novel IRL assessment; we aim to fill this research gap by considering a middle-granularity state space between the entire state space and the specific sampling space. We utilize the agglomerative nesting algorithm (AGNES) to control dynamical C-NMI computing via a 4-order tensor model with injected manipulated trajectories. With such a model, we can uniformly capture different-dimension values of MRL, EVD, and C-NMI, and perform more comprehensive and accurate assessment and analyses. Extensive experiments on several mainstream IRLs are experimented in Object World, hence revealing that the assessing accuracy of our method increases 110.13% and 116.59% respectively when compared with the EVD and MRL. Meanwhile, C-NMI is more robust than EVD and MRL under different demonstrations.

Inverse Reinforcement Learning for Adversarial Apprentice Games.

This article proposes new inverse reinforcement learning (RL) algorithms to solve our defined Adversarial Apprentice Games for nonlinear learner and expert systems. The games are solved by extracting the unknown cost function of an expert by a learner using demonstrated expert's behaviors. We first develop a model-based inverse RL algorithm that consists of two learning stages: an optimal control learning and a second learning based on inverse optimal control. This algorithm also clarifies the relationships between inverse RL and inverse optimal control. Then, we propose a new model-free integral inverse RL algorithm to reconstruct the unknown expert cost function. The model-free algorithm only needs online demonstration of the expert and learner's trajectory data without knowing system dynamics of either the learner or the expert. These two algorithms are further implemented using neural networks (NNs). In Adversarial Apprentice Games, the learner and the expert are allowed to suffer from different adversarial attacks in the learning process. A two-player zero-sum game is formulated for each of these two agents and is solved as a subproblem for the learner in inverse RL. Furthermore, it is shown that the cost functions that the learner learns to mimic the expert's behavior are stabilizing and not unique. Finally, simulations and comparisons show the effectiveness and the superiority of the proposed algorithms.

Towards Human-Robot Collaborative Surgery: Trajectory and Strategy Learning in Bimanual Peg Transfer

While the traditional control of surgical robots relies on fully manual teleoperations, human-robot collaborative systems promise to address issues such as workspace constrains and laborious tasks. In particular, shared control can reduce the surgeon's workload and improve the overall surgical performance by supporting the surgeon effort during movements while keeping them in charge of complex control phases. In this letter, we propose a segmentation of the bimanual peg transfer task that alternates manual and autonomous control correspondingly. The authority allocation in this shared control framework considers both the limitation of learning-based methods and the higher dexterity of humans during physical interaction. The motion and strategies are transferred from an expert human to a da Vinci Research Kit (dVRK) [1] using an epsilon-greedy on a maximum entropy inverse reinforcement learning algorithm. The model generated enables to train an intelligent agent that can skillfully collaborate with the human operator during the surgical task. The proposed shared control framework is verified both on a virtual platform and then on a real dVRK to assess its usability and robustness. The results show that, compared to traditional teleoperation, our method can achieve faster and more consistent peg transfers. An analysis of the participants' effort also reveals a significantly lower perception of the workload.

Off-policy inverse Q-learning for discrete-time antagonistic unknown systems

This paper proposes a data-driven model-free inverse reinforcement learning (RL) algorithm to reconstruct the unknown cost function of the demonstrated discrete-time (DT) dynamical systems with antagonistic disturbances. We propose an inverse RL policy iteration scheme that uses system dynamics and the input policies, for deriving our main result of a data-driven off-policy inverse Q-learning algorithm using only demonstrated trajectories of the antagonistic system without knowing system dynamics and the control policy gain. This data-driven algorithm consists of Q-function evaluation, state-penalty weight improvement, and action policies update. We guarantee unbiased estimates in the data-driven algorithm when exploration noises exist for the persistence of the excitation. An example verifies the proposed algorithm.

Quantum generative adversarial imitation learning

Investigating quantum advantage in the NISQ era is a challenging problem whereas quantum machine learning becomes the most promising application that can be resorted to. However, no proposal has been investigated for arguably challenging inverse reinforcement learning to demonstrate the potential advantage. In this work, we propose a hybrid quantum–classical inverse reinforcement learning algorithm based on the variational quantum circuit with the generative adversarial framework. We find an important connection between the quantum gradient anomaly and the performance degradation, which suggest a gradient clipping strategy to stabilize the training process. In light of the algorithm, we study three classic control problems and the Hamiltonian parameter estimation in quantum sensing with shallow quantum circuits. The numerical results showcase that the control-enhanced quantum sensor can saturate quantum Cramér-Rao bound only with a single variational layer, empirically demonstrating a parameter complexity advantage over the classical learning control. The proposed generative adversarial reinforcement learning algorithm achieves state-of-the-art performance in classical and quantum sensor control in terms of required number of parameters.

Inverse Reinforcement Learning as the Algorithmic Basis for Theory of Mind: Current Methods and Open Problems

Theory of mind (ToM) is the psychological construct by which we model another’s internal mental states. Through ToM, we adjust our own behaviour to best suit a social context, and therefore it is essential to our everyday interactions with others. In adopting an algorithmic (rather than a psychological or neurological) approach to ToM, we gain insights into cognition that will aid us in building more accurate models for the cognitive and behavioural sciences, as well as enable artificial agents to be more proficient in social interactions as they become more embedded in our everyday lives. Inverse reinforcement learning (IRL) is a class of machine learning methods by which to infer the preferences (rewards as a function of state) of a decision maker from its behaviour (trajectories in a Markov decision process). IRL can provide a computational approach for ToM, as recently outlined by Jara-Ettinger, but this will require a better understanding of the relationship between ToM concepts and existing IRL methods at the algorthmic level. Here, we provide a review of prominent IRL algorithms and their formal descriptions, and discuss the applicability of IRL concepts as the algorithmic basis of a ToM in AI.

Reinforcement Learning for Stock Prediction and High-Frequency Trading With T+1 Rules

The high-frequency trading framework for the price trend prediction model and trading strategy has been a popular approach for T+0 trading in the stock market. The prediction model is used to predict price trends, and the trading strategy is used to determine the price and volume of the order. Most trading strategies consist of multiple trading logic associated with certain tuning parameters. These parameters significantly affect the profitability of high-frequency trading frameworks. There are two main disadvantages of this framework: 1) the price trend prediction model can not adapt to the current market data distribution, and 2) the trading strategy can not adapt to the current market conditions automatically. Thus, the framework cannot always maintain positive revenue. To address this problem, we propose a novel dynamic parameter optimization algorithm based on reinforcement learning for stock prediction and trading, and to generate an adaptive trading framework. First, we use a rolling model training method for stock price trend prediction. Second, we regard each set of strategy parameters as action and devise an inverse reinforcement learning algorithm for the reward function to accurately estimate the reward of each action. Because of the T+1 trading rules of the Chinese stock market, we consider the constraint of limited short position in the reward function. Finally, a reward-enhanced upper confidence bound (UCB) selection algorithm is proposed to automatically optimize the parameters of the trading logic in real-time trading. The experimental results show that our method achieves competitive performance in the Chinese stock market.

Inverse Reinforcement Learning Based: Segmented Lane-Change Trajectory Planning With Consideration of Interactive Driving Intention

One of the most challenging problems in autonomous driving is trajectory planning for lane changes. Conventional trajectory planning is generally realized by optimizing a specific cost function. However, the model performance required for different scenarios makes it repetitive and time-consuming to tune the cost function. Furthermore, to ensure driving safety, comfort as well as traffic efficiency, it is essential to interact with other vehicles by predicting their driving intentions. Therefore, this paper proposes a segmented-updated lane-change trajectory planning method for autonomous vehicles on closed highways. The maximum entropy inverse reinforcement learning (IRL) algorithm for human demonstration learning and the long short-term memory and Bayesian network (LSTM-BN)-based model for target vehicles' driving intentions are applied. Three contributions are made in this work: 1) The LSTM-BN is used to judge the intentions of surrounding vehicles based on its historical information to inform lane-change trajectory planning. 2) Different feature-based cost functions of the IRL algorithm are designed according to different lane-change scenarios, which enhances its adaptability to different scenarios. 3) Segmented-updated trajectory planning is used to dynamically optimize trajectories via an online optimization process, which reduces the model's limitations and improves overall lane-change performance. Finally, simulation experiments demonstrate that the proposed method has strong generalization ability, reliable accuracy, and can adjust lane-change trajectories reasonably according to the driving intentions of surrounding vehicles.

Energy-Based Legged Robots Terrain Traversability Modeling via Deep Inverse Reinforcement Learning

This work reports ondeveloping a deep inverse reinforcement learning method for legged robots terrain traversability modeling that incorporates both exteroceptive and proprioceptive sensory data. Existing works use robot-agnostic exteroceptive environmental features or handcrafted kinematic features; instead, we propose to also learn robot-specific inertial features from proprioceptive sensory data for reward approximation in a single deep neural network. Incorporating the inertial features can improve the model fidelity and provide a reward that depends on the robot’s state during deployment. We train the reward network using the Maximum Entropy Deep Inverse Reinforcement Learning (MEDIRL) algorithm and propose simultaneously minimizing a trajectory ranking loss to deal with the suboptimality of legged robot demonstrations. The demonstrated trajectories are ranked by locomotion energy consumption, in order to learn an energy-aware reward function and a more energy-efficient policy than demonstration. We evaluate our method using a dataset collected by an MIT Mini-Cheetah robot and a Mini-Cheetah simulator. The code is publicly available. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref>

An Inverse Reinforcement Learning Approach for Customizing Automated Lane Change Systems

Vehicle automation seeks to enhance road safety and improve the driving experience. However, a standard system does not account for variations in users and driving conditions. Customizing vehicle automation based on users’ preferences aims to improve the user experience and adoption of the technologies. This study introduces a systematic paradigm that starts with naturalistic driving data to identify the driving behaviors and styles for a customized automated lane change system. The driving behaviors are first extracted using Multivariate Functional Principal Component Analysis (MFPCA) with minimum prior expert knowledge. The driving styles are identified by clustering the extracted driving behaviors. An Inverse Reinforcement Learning (IRL) algorithm is then used to train the automated lane change system from grouped demonstrations of the identified driving styles to capture the preferences of a group of drivers with a similar driving style. The performance of the proposed customized automated lane change system is compared to (1) a non-customized system trained on all the sample trips, (2) customized systems built on expert-coded reward functions, and (3) customized systems trained using a Generative Adversarial Imitation Learning (GAIL) algorithm. The results show that our method outperforms all the other systems with respect to the prediction accuracy of the lane change actions. Additionally, our method gains insights on the representative behaviors of different driving styles to enable customization of automated lane change systems.

Inverse reinforcement learning for multi-player noncooperative apprentice games

In this paper, we devise inverse reinforcement learning (RL) algorithms for nonlinear continuous-time systems described by multiplayer differential equations. We define a new class of Multi-player Noncooperative Apprentice Games, in which both the expert and the learner have N-player control inputs. The games are solved by the learners reconstructing unknown performance reward functions of the experts from the experts’ trajectories, i.e., states and optimal control inputs. We first develop a model-based inverse RL algorithm that involves two learning stages: an optimal control learning stage and a second inverse optimal control (IOC) learning stage. Our algorithm solves IOC as a subproblem. We therefore provide one possible unified framework for inverse RL and IOC in multiplayer differential dynamic systems. We then develop two inverse RL algorithms using neural networks: completely model-free for homogeneous control inputs; and partially model-free for heterogeneous control inputs. Finally, we present the results of simulations, which verify the validity of our proposed algorithms.

Sparse online maximum entropy inverse reinforcement learning via proximal optimization and truncated gradient

Maximum entropy (ME) algorithms have been broadly studied towards the learning of rewards and obtaining of optimal policies for inverse reinforcement learning (IRL) problems. However, the tendency for the avoidance of maximum entropy IRL (ME IRL) algorithms, due to issues such as complicated computations, overfitting, and low convergence, has inspired studies aimed at the development of newer ME IRL algorithms of improved performance. Here, a new maximum entropy IRL based on proximal optimization is proposed. The follow-the-proximally-regularized-leader (FTPRL) method, with its good sparse solutions, is taken as proximal optimization to improve the generalization performance of ME IRL algorithm, resulting in ME-FTPRL IRL. With the help of l1/l22-regularization and adaptive per-state learning rates, our proposed algorithm can select features and correct the update direction of reward weights to reduce model complexity and avoid overfitting, which also speeds up convergence. During each iteration, the truncated gradient (TG) method is applied for the ME-FTPRL IRL (named ME-TFTPRL IRL) to update reward weights. This avoids the floating-point problem of the FTPRL method. Q-learning algorithm is then used to obtain the optimal strategies with the learned rewards. Subsequently, the sparsity and convergence of ME-TFTPRL IRL are proven based on regularization, TG method and regret bound. Simulation results demonstrate that our proposed ME-TFTPRL IRL has better sparsity, better generalization, and faster convergence than either existing or our own initially proposed ME IRL algorithms.

How Private Is Your RL Policy? An Inverse RL Based Analysis Framework

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep-Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL analysis framework (PRIL) that involves performing reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using the Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.

AdaBoost maximum entropy deep inverse reinforcement learning with truncated gradient

Studying the representational capacity of neural networks to learn nonlinear rewards is necessary in a complex and nonlinear environment. Over recent years, the maximum entropy deep inverse reinforcement learning algorithm (ME-DIRL) has been increasingly applied to the learning of nonlinear rewards. However, under cases of limited and imbalanced expert demonstration data, complex calculations, or overfitting, the learning nonlinear rewards remains a challenging problem. A novel ME-DIRL with AdaBoost algorithm (AME-DIRL) is our proposed solution. The focus of AME-DIRL is to utilize the AdaBoost algorithm. This combines multiple ME-DIRL processes to form a strong learner and thus overcome the imbalance of the data set. Furthermore, to deal with the complex calculations in AME-DIRL, a truncated gradient (TG) method is applied for getting the sparse rewards obtained by the strong learner, thus reducing the model complexity. To prevent overfitting, a correction factor is then added to the linear combination of weak learners. AME-DIRL models the relationship between input features and output rewards. Rewards are approximated by means of a convolutional neural network (CNN) with scaled exponential linear units (SELUs). Numerical results indicate that our proposed AME-DIRL shows higher accuracy in learning rewards when compared with several classical inverse reinforcement learning (IRL) algorithms.

Spatiotemporal Costmap Inference for MPC Via Deep Inverse Reinforcement Learning

It can be difficult to autonomously produce driver behavior so that it appears natural to other traffic participants. Through Inverse Reinforcement Learning (IRL), we can automate this process by learning the underlying reward function from human demonstrations. We propose a new IRL algorithm that learns a goal-conditioned spatio-temporal reward function. The resulting costmap is used by Model Predictive Controllers (MPCs) to perform a task without any hand-designing or hand-tuning of the cost function. We evaluate our proposed Goal-conditioned SpatioTemporal Zeroing Maximum Entropy Deep IRL (GSTZ)-MEDIRL framework together with MPC in the CARLA simulator for autonomous driving, lane keeping, and lane changing tasks in a challenging dense traffic highway scenario. Our proposed methods show higher success rates compared to other baseline methods including behavior cloning, state-of-the-art RL policies, and MPC with a learning-based behavior prediction model.