Inverse Reinforcement Learning Research Articles

Estimation of pedestrian-vehicle behavior using adversarial inverse reinforcement learning

Estimation of pedestrian-vehicle behavior using adversarial inverse reinforcement learning

A novel Congestion Control algorithm based on inverse reinforcement learning with parallel training

With the growing impact of the Internet, computer network communication has become an essential component for various industries. Congestion Control (CC) algorithms serve as the backbone of network communication, significantly affecting network quality. However, designing a CC algorithm that performs optimally across diverse network environments presents substantial challenges. The task of redesigning and optimizing CC algorithms for specific network environments demands both expert experience and a substantial workforce. In this paper, we proposed an inverse reinforcement learning (IRL) algorithm that can use expert data to guide the self-optimization of the CC model in specific network environments. To enhance model training efficiency, we propose a parallel training framework and a visualization analysis tool, enabling distributed training and real-time control level analysis. In the experiments, we assess the performance of 16 algorithms across 3 network scenarios using Pantheon. Our IRL model achieves the optimal level of network performance in satellite network, enhancing throughput by 10%–23%. For delay performance, it ranks 2nd in wired network and achieves a 21%–67% improvement over traditional TCP algorithms in regular network.

No more road bullying: An integrated behavioral and motion planner with proactive right-of-way acquisition capability

This research proposes an integrated behavioral and motion planner with proactive right-of-way acquisition capability. The proposed planner overcomes the shortcomings of conventional planner and is able to: (i) integrate both behavioral and motion planning for enhanced optimal maneuver; (ii) support real-time implementation; (iii) proactively acquire right-of-way for improved travel efficiency; (iv) take a person-by-person driving countermeasure for strengthened safety. In order to realize the proactive right-of-way acquisition, the game-based interaction-aware decision-making is designed, and Stackelberg competition is adopted as the game rule. The problem is formulated as a differential-game-based Receding Horizon Control (RHC) to integrate both behavioral and motion planning. An online Inverse Reinforcement Learning (IRL) is leveraged to identify the individual driving behavior of the competing vehicle in order for the enabling of person-by-person countermeasures. The problem is solved by a highly-efficient quadratic programming algorithm. A simulation evaluation is conducted on the proposed planner. The results demonstrate that the proposed planner is with millisecond-level computational efficiency. It improves travel efficiency by 4.78–25.77% with the proactive right-of-way acquisition, and enhances driving safety by 5.22–36.08% via taking person-by-person countermeasures. The range is caused by different congestion levels.

A Mutual Information-Based Assessment of Reverse Engineering on Rewards of Reinforcement Learning

Rewards are critical hyperparameters in reinforcement learning (RL), since in most cases different reward values will lead to greatly different performance. Due to their commercial value, RL rewards become the target of reverse engineering by the inverse reinforcement learning (IRL) algorithm family. Existing efforts typically utilize two metrics to measure the IRL performance: the expected value difference and the mean reward loss, which we call them EVD and MRL respectively. Unfortunately, in some cases, EVD and MRL can give completely opposite results, due to MRL focusing on whole state-space rewards while EVD only considering partly sampled rewards. Such situation naturally rises to one fundamental question: whether current metrics and assessment are sufficient and accurate for more general use. Thus, in this paper, based on the metric called normalized mutual information of reward clusters (C-NMI) we propose a novel IRL assessment; we aim to fill this research gap by considering a middle-granularity state space between the entire state space and the specific sampling space. We utilize the agglomerative nesting algorithm (AGNES) to control dynamical C-NMI computing via a 4-order tensor model with injected manipulated trajectories. With such a model, we can uniformly capture different-dimension values of MRL, EVD, and C-NMI, and perform more comprehensive and accurate assessment and analyses. Extensive experiments on several mainstream IRLs are experimented in Object World, hence revealing that the assessing accuracy of our method increases 110.13% and 116.59% respectively when compared with the EVD and MRL. Meanwhile, C-NMI is more robust than EVD and MRL under different demonstrations.

A Hierarchical Vehicle Behavior Prediction Framework With Traffic Signals and Interactive Agents

Vehicle behavior prediction in complex urban scenarios with traffic signals and interactive agents is an important yet complicated task for autonomous vehicles (AVs). In this work, a hierarchical vehicle behavior prediction framework is proposed to incorporate the traffic signal information and model the interaction between vehicles. The framework predicts vehicle behaviors in two stages, discrete intention prediction and continuous trajectory prediction. In the discrete intention prediction stage, Bayesian network is adopted to provide a high-level behavior prediction of the principle other vehicle. The discrete prediction results are forwarded to the second stage, where a continuous trajectory is predicted with maximum entropy inverse reinforcement learning and potential game. The framework is designed to be able to capture the difference among human drivers with parameterized driver characteristics. The proposed predictor is validated in two scenarios: the yellow light running scenario and the right-turn scenario. The trajectory prediction average displacement error of the yellow light running scenario is 0.695m for a 3-second prediction interval, and the prediction accuracy of the right-turn vehicle in the right-turn scenario is 0.51m for a 2-second prediction interval.

Generative inverse reinforcement learning for learning 2-opt heuristics without extrinsic rewards in routing problems

Deep reinforcement learning (DRL) has shown promise in solving challenging combinatorial optimization (CO) problems, such as the traveling salesman problem (TSP) and vehicle routing problem (VRP). However, existing DRL methods rely on manually designed reward functions, which may be inaccurate or unrealistic. Moreover, traditional DRL algorithms suffer from unstable training and sparse reward problems. This paper proposes GIRL (Generative Inverse Reinforcement Learning), a method to learn 2-opt heuristics without explicit extrinsic rewards to address these limitations. GIRL combines generative adversarial networks (GANs) and DRL to learn effective policies and reward functions in a reverse end-to-end fashion, improving generalization capabilities. Furthermore, we introduce a self-attentional policy network tailored for 2-opt heuristics and train the framework using a soft actor-critic algorithm along with a discriminator in the GAN.Extensive experiments on various TSP and VRP instances demonstrate superior performance compared to state-of-the-art methods. Moreover, integrating GANs and DRL enables data-driven reward functions, improving accuracy and realism. Using self-attentional networks and the soft actor-critic algorithm enhances training stability and addresses the sparse reward problem. This work advances reinforcement learning techniques in CO, enabling more accurate and practical optimization methods in real-world applications.

Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference

Recent advances in reinforcement learning have inspired increasing interest in learning user modeling adaptively through dynamic interactions, e.g., in reinforcement learning based recommender systems. In most reinforcement learning applications, reward functions provide the critical guideline for optimization. However, current reinforcement learning-based methods rely on manually-defined reward functions, which cannot adapt to dynamic, noisy environments. Moreover, they generally use task-specific reward functions that sacrifice generalization ability. We propose a generative inverse reinforcement learning for user behavioral preference modeling to address the above issues. Instead of using predefined reward functions, our model can automatically learn the rewards from user's actions based on discriminative actor-critic network and Wasserstein GAN. Our model provides a general approach to characterizing and explaining underlying behavioral tendencies. Our experiments show our method outperforms state-of-the-art methods in several scenarios, namely traffic signal control, online recommender systems, and scanpath prediction.

Transaction-aware inverse reinforcement learning for trading in stock markets

Transaction-aware inverse reinforcement learning for trading in stock markets

Control-oriented imitation learning for atmospheric flight control of launch vehicles under wind disturbance

The atmospheric flight control of launch vehicle is required to track the nominal trajectory while obeying path constraints on the aerodynamic load peak. Due to the wind disturbance, wind-induced angle of attack may increase the aerodynamic load peak, and reducing of it will lead to larger trajectory tracking error. Because of the cumbersome tuning of gain schedule and the dependency on accurate wind prediction, it is difficult for existing methods to handle the trade-off problem between reduction of the aerodynamic load peak and trajectory tracking error under uncertain wind disturbance. The presented research is intended to remedy this difficulty through learning based methods. The ascent flight phase is modelled as a multiple objective Markov decision process, where the trade-off under wind disturbance is precisely modelled as the optimization of a multi-objective vector under the stochastic state transition. A neural network flight control policy is then trained by a novel control-oriented imitation learning (CoIL) scheme. The CoIL scheme incorporates two control-oriented design within the standard reinforcement learning scheme: the inverse reinforcement learning that infers a proper reward from demonstration trajectories to represent the multi-objective vector of atmospheric flight control, as well as the policy hierarchy that simplifies interaction with the high order vehicle dynamics. Compared with previous learning-based attempts in the literature, this scheme uses orders of magnitude fewer demonstration trajectories and avoids hand-programmed reward design. Evaluations are performed under an ascent flight mission, indicating that the obtained policy outperforms the prevalent baseline controllers with reduction of the normalized maximum aerodynamic load by 28.3%, altitude error by 38.6%, velocity error by 17.1% and path angle error by 42.7% under unpredicted wind disturbance.

Learning by Successful or Unsuccessful Experiences?

Humans learn from both successful and unsuccessful experiences, because useful information about how to solve complex problems can be gleaned not only from success but also from failure. In this paper, we propose a method for investigating this difference by applying Preference based Inverse Reinforcement Learning to Double Transition Models built from replays of StarCraft II. Our method provides two advantages: (1) the ability to identify integrated reward distributions from computational models composed of multiple experiences, and (2) the ability to discern differences between learning by successes and failures. Our experimental results demonstrate that reward distributions are shaped depending on the trajectories utilized to build models. Reward distributions based on successful episodes were skewed to the left, while those based on unsuccessful episodes were skewed to the right. Furthermore, we found that players with symmetric triple reward distributions had a high probability of winning the game.

Danger-zone and maximum entropy deep inverse reinforcement learning for human-robot navigation

As a result of the development of artificial intelligence in recent years, scientists have gone further in the field of human-robot interaction (HRI), and one of the remaining problems is how to achieve a safe and human comfort-satisfying navigation when it comes to crowd-robot interaction (CRI). This article explores two existing deep learning reinforcement (DRL) methods, Danger-zone (DZ) and Maximum Entropy Deep Inverse Reinforcement Learning (MEDIRL). The former method proposes a Danger-Zone to predict the trajectory of pedestrians, using the DRL network to achieve obstacle avoidance and combining advanced algorithms with it to summarize the most effective combination. The latter method collects expert demonstrations in an uncontrolled environment. It uses DNN networks to predict human behavior and compares the results with the actual trajectories to demonstrate their validity. This article summarizes and evaluates these two methods. Moreover, this article also gives various outlooks on the direction of human-computer interaction.

Enhancing construction robot learning for collaborative and long-horizon tasks using generative adversarial imitation learning

The development and deployment of robots on construction sites are integral to the industrialization of construction, known as Construction 4.0. Tele-operated and pre-programmed robots have enhanced construction efficiency and safety. However, their utilization on-site remains limited due to the need for expert remote control and the lack of adaptability in dynamic environments. Reinforcement learning (RL) has emerged as a promising solution, as RL-controlled robots possess inherent self-learning abilities to adapt to diverse situations. Nevertheless, manual design of RL reward functions for complex tasks poses challenges. To address this issue, inverse reinforcement learning (IRL) methods, such as Generative Adversarial Imitation Learning (GAIL), have been proposed to learn optimal actions through expert demonstration and self-exploration, without explicitly defined reward functions. In this study, we propose an innovative approach integrating GAIL and virtual reality (VR) integrated robot control approach to control robots for long-horizon collaborative construction tasks involving multiple sub-tasks. We employ VR expert demonstrations as input for GAIL training, enabling a team of robots, including an Unmanned Ground Vehicle (UGV) and two robot arms, to interact with the designed RL environment and perform tasks such as transporting, picking, and installing window panels. Handle long-horizon collaborative construction tasks (i.e., a long sequence of several sub-tasks performed by multiple robots). For evaluation, we compare the performance of our VR-GAIL model with a prevalent and robust RL baseline model, Proximal Policy Optimization (PPO). The results demonstrate that our reward-free VR-GAIL model achieves, on average, a 4.5% higher success rate than the PPO counterpart equipped with carefully designed reward functions across all three sub-tasks and their randomized variations. Furthermore, the performance gap between GAIL and PPO widens as the task difficulty increases. These findings indicate that our approach effectively enhances RL agent performance in tackling complex construction tasks while expediting development by eliminating reward function design requirements.

Inverse Reinforcement Learning for Adversarial Apprentice Games.

This article proposes new inverse reinforcement learning (RL) algorithms to solve our defined Adversarial Apprentice Games for nonlinear learner and expert systems. The games are solved by extracting the unknown cost function of an expert by a learner using demonstrated expert's behaviors. We first develop a model-based inverse RL algorithm that consists of two learning stages: an optimal control learning and a second learning based on inverse optimal control. This algorithm also clarifies the relationships between inverse RL and inverse optimal control. Then, we propose a new model-free integral inverse RL algorithm to reconstruct the unknown expert cost function. The model-free algorithm only needs online demonstration of the expert and learner's trajectory data without knowing system dynamics of either the learner or the expert. These two algorithms are further implemented using neural networks (NNs). In Adversarial Apprentice Games, the learner and the expert are allowed to suffer from different adversarial attacks in the learning process. A two-player zero-sum game is formulated for each of these two agents and is solved as a subproblem for the learner in inverse RL. Furthermore, it is shown that the cost functions that the learner learns to mimic the expert's behavior are stabilizing and not unique. Finally, simulations and comparisons show the effectiveness and the superiority of the proposed algorithms.

Inverse Reinforcement Learning of Pedestrian–Robot Coordination

Inverse Reinforcement Learning of Pedestrian–Robot Coordination

Towards Human-Robot Collaborative Surgery: Trajectory and Strategy Learning in Bimanual Peg Transfer

While the traditional control of surgical robots relies on fully manual teleoperations, human-robot collaborative systems promise to address issues such as workspace constrains and laborious tasks. In particular, shared control can reduce the surgeon's workload and improve the overall surgical performance by supporting the surgeon effort during movements while keeping them in charge of complex control phases. In this letter, we propose a segmentation of the bimanual peg transfer task that alternates manual and autonomous control correspondingly. The authority allocation in this shared control framework considers both the limitation of learning-based methods and the higher dexterity of humans during physical interaction. The motion and strategies are transferred from an expert human to a da Vinci Research Kit (dVRK) [1] using an epsilon-greedy on a maximum entropy inverse reinforcement learning algorithm. The model generated enables to train an intelligent agent that can skillfully collaborate with the human operator during the surgical task. The proposed shared control framework is verified both on a virtual platform and then on a real dVRK to assess its usability and robustness. The results show that, compared to traditional teleoperation, our method can achieve faster and more consistent peg transfers. An analysis of the participants' effort also reveals a significantly lower perception of the workload.

Interactive cognitive maps support flexible behavior under threat

In social environments, survival can depend upon inferring and adapting to other agents' goal-directed behavior. However, it remains unclear how humans achieve this, despite the fact that many decisions must account for complex, dynamic agents acting according to their own goals. Here, we use a predator-prey task (total n= 510) to demonstrate that humans exploit an interactive cognitive map of the social environment to infer other agents' preferences and simulate their future behavior, providing for flexible, generalizable responses. A model-based inverse reinforcement learning model explained participants' inferences about threatening agents' preferences, with participants using this inferred knowledge to enact generalizable, model-based behavioral responses. Using tree-search planning models, we then found that behavior was best explained by a planning algorithm that incorporated simulations of the threat's goal-directedbehavior. Our results indicate that humans use a cognitive map to determine other agents' preferences, facilitating generalized predictions of their behavior and effective responses.

AdaBoost-Bagging deep inverse reinforcement learning for autonomous taxi cruising route and speed planning

Taxi cruising route planning has attracted considerable attention, and relevant studies can be broadly categorized into three main streams: recommending one or multiple areas, providing a detailed cruising route, and deriving the optimal routing policy. However, these studies depend on accurate pick-up/drop-off information, and seldom pay attention to cruising speed planning. In view of the rapid development of autonomous taxis, this study proposes AdaBoost-Bagging maximum entropy deep inverse reinforcement learning to learn cruising policy from experienced taxi drivers’ trajectories. Moreover, we develop a trajectory-based self-attention bidirectional LSTM model to adjust cruising speeds on different roads. Numerical experiments using real taxi trajectories in Chengdu, China demonstrate the effectiveness of our approach in learning taxi drivers’ policies and improving taxis’ operational efficiency.

Retracted: Real-Time Generation Method of Oil Painting Style Brushstrokes Based on Inverse Reinforcement Learning

Retracted: Real-Time Generation Method of Oil Painting Style Brushstrokes Based on Inverse Reinforcement Learning

Off-policy inverse Q-learning for discrete-time antagonistic unknown systems

This paper proposes a data-driven model-free inverse reinforcement learning (RL) algorithm to reconstruct the unknown cost function of the demonstrated discrete-time (DT) dynamical systems with antagonistic disturbances. We propose an inverse RL policy iteration scheme that uses system dynamics and the input policies, for deriving our main result of a data-driven off-policy inverse Q-learning algorithm using only demonstrated trajectories of the antagonistic system without knowing system dynamics and the control policy gain. This data-driven algorithm consists of Q-function evaluation, state-penalty weight improvement, and action policies update. We guarantee unbiased estimates in the data-driven algorithm when exploration noises exist for the persistence of the excitation. An example verifies the proposed algorithm.

Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planning

This paper focuses on inverse reinforcement learning for autonomous navigation using distance and semantic category observations. The objective is to infer a cost function that explains demonstrated behavior while relying only on the expert’s observations and state-control trajectory. We develop a map encoder, that infers semantic category probabilities from the observation sequence, and a cost encoder, defined as a deep neural network over the semantic features. Since the expert cost is not directly observable, the model parameters can only be optimized by differentiating the error between demonstrated controls and a control policy computed from the cost estimate. We propose a new model of expert behavior that enables error minimization using a closed-form subgradient computed only over a subset of promising states via a motion planning algorithm. Our approach allows generalizing the learned behavior to new environments with new spatial configurations of the semantic categories. We analyze the different components of our model in a minigrid environment. We also demonstrate that our approach learns to follow traffic rules in the autonomous driving CARLA simulator by relying on semantic observations of buildings, sidewalks, and road lanes.