Expert Trajectories Research Articles

Reinforcement learning from suboptimal demonstrations based on Reward Relabeling

Deep reinforcement learning (DRL) has achieved remarkable milestones in the field of artificial intelligence. However, the reward functions for most real-world tasks are characterized by delays and sparsity, posing significant challenges for DRL methods. To tackle the issues of delayed and sparse rewards, there have been many approaches based on the prior knowledge of expert trajectories proposed, such as GAIL and its variants. However, if only suboptimal demonstrations available, they usually struggle to overcome the performance disadvantage due to the complexity and fragility of adversarial training. To address these problems, this paper introduces a novel framework combining Self-Imitation learning with Reward Relabeling based Reinforcement learning, thus dubbed SIR3. It is capable of accelerating online learning using suboptimal demonstrations in environments even with extremely sparse rewards and meanwhile encouraging exploration of better policies. SIR3 devises a task-independent reward relabeling mechanism to generate reward signals for both the expert examples and online experience. This design provides the agent with more informative guidance, even when the number of suboptimal demonstrations is minimal. During the training process, the integration of imitation learning and RL losses enables the agent to dynamically mimic rewarding trajectories, possibly collected from experts or self-explored. Experimental findings on widely recognized MuJoCo benchmarks reveal that SIR3 can efficiently learn excellent policies surpassing suboptimal demonstrations, achieving superior training efficiency and performance relative to SOTA methods. Notably, in some environments, it secures a performance edge over an order of magnitude.

A Soft Actor-Critic Deep Reinforcement-Learning-Based Robot Navigation Method Using LiDAR

When there are dynamic obstacles in the environment, it is difficult for traditional path-generation algorithms to achieve desired obstacle-avoidance results. To solve this problem, we propose a robot navigation control method based on SAC (Soft Actor-Critic) Deep Reinforcement Learning. Firstly, we use a fast path-generation algorithm to control the robot to generate expert trajectories when the robot encounters danger as well as when it approaches a target, and we combine SAC reinforcement learning with imitation learning based on expert trajectories to improve the safety of training. Then, for the hybrid data consisting of agent data and expert data, we use an improved prioritized experience replay method to improve the learning efficiency of the policies. Finally, we introduce RNN (Recurrent Neural Network) units into the network structure of the SAC Deep Reinforcement-Learning navigation policy to improve the agent’s transfer inference ability in a new environment and obstacle-avoidance ability in dynamic environments. Through simulation and practical experiments, it is fully verified that our method has a higher training efficiency and navigation success rate compared to state-of-the-art reinforcement-learning algorithms, which further enhances the obstacle-avoidance capability of the robot system.

Robust Visual Imitation Learning with Inverse Dynamics Representations

Imitation learning (IL) has achieved considerable success in solving complex sequential decision-making problems. However, current IL methods mainly assume that the environment for learning policies is the same as the environment for collecting expert datasets. Therefore, these methods may fail to work when there are slight differences between the learning and expert environments, especially for challenging problems with high-dimensional image observations. However, in real-world scenarios, it is rare to have the chance to collect expert trajectories precisely in the target learning environment. To address this challenge, we propose a novel robust imitation learning approach, where we develop an inverse dynamics state representation learning objective to align the expert environment and the learning environment. With the abstract state representation, we design an effective reward function, which thoroughly measures the similarity between behavior data and expert data not only element-wise, but also from the trajectory level. We conduct extensive experiments to evaluate the proposed approach under various visual perturbations and in diverse visual control tasks. Our approach can achieve a near-expert performance in most environments, and significantly outperforms the state-of-the-art visual IL methods and robust IL methods.

Energy-Based Continuous Inverse Optimal Control.

The problem of continuous inverse optimal control (over finite time horizon) is to learn the unknown cost function over the sequence of continuous control variables from expert demonstrations. In this article, we study this fundamental problem in the framework of energy-based model (EBM), where the observed expert trajectories are assumed to be random samples from a probability density function defined as the exponential of the negative cost function up to a normalizing constant. The parameters of the cost function are learned by maximum likelihood via an "analysis by synthesis" scheme, which iterates: 1) synthesis step: sample the synthesized trajectories from the current probability density using the Langevin dynamics via backpropagation through time and 2) analysis step: update the model parameters based on the statistical difference between the synthesized trajectories and the observed trajectories. Given the fact that an efficient optimization algorithm is usually available for an optimal control problem, we also consider a convenient approximation of the above learning method, where we replace the sampling in the synthesis step by optimization. Moreover, to make the sampling or optimization more efficient, we propose to train the EBM simultaneously with a top-down trajectory generator via cooperative learning, where the trajectory generator is used to fast initialize the synthesis step of the EBM. We demonstrate the proposed methods on autonomous driving tasks and show that they can learn suitable cost functions for optimal control.

A Fast Anti-Jamming Algorithm Based on Imitation Learning for WSN.

Wireless sensor networks (WSNs), integral components underpinning the infrastructure of the internet of things (IoT), confront escalating threats originating from attempts at malicious jamming. Nevertheless, the limited nature of the hardware resources in distributed, low-cost WSNs, such as those for computing power and storage, poses a challenge when implementing complex and intelligent anti-jamming algorithms like deep reinforcement learning (DRL). Hence, in this paper a rapid anti-jamming method is proposed based on imitation learning in order to address this issue. First, on-network nodes obtain expert anti-jamming trajectories using heuristic algorithms, taking historical experiences into account. Second, an RNN neural network that can be used for anti-jamming decision making is trained by mimicking these expert trajectories. Finally, the late-access network nodes receive anti-jamming network parameters from the existing nodes, allowing them to obtain a policy network directly applicable to anti-jamming decision making and thus avoiding redundant learning. Experimental results demonstrate that, compared with traditional Q-learning and random frequency-hopping (RFH) algorithms, the imitation learning-based algorithm empowers late-access network nodes to swiftly acquire anti-jamming strategies that perform on par with expert strategies.

Anti-Jamming Communication Using Imitation Learning.

The communication reliability of wireless communication systems is threatened by malicious jammers. Aiming at the problem of reliable communication under malicious jamming, a large number of schemes have been proposed to mitigate the effects of malicious jamming by avoiding the blocking interference of jammers. However, the existing anti-jamming schemes, such as fixed strategy, Reinforcement learning (RL), and deep Q network (DQN) have limited use of historical data, and most of them only pay attention to the current state changes and cannot gain experience from historical samples. In view of this, this manuscript proposes anti-jamming communication using imitation learning. Specifically, this manuscript addresses the problem of anti-jamming decisions for wireless communication in scenarios with malicious jamming and proposes an algorithm that consists of three steps: First, the heuristic-based Expert Trajectory Generation Algorithm is proposed as the expert strategy, which enables us to obtain the expert trajectory from historical samples. The trajectory mentioned in this algorithm represents the sequence of actions undertaken by the expert in various situations. Then obtaining a user strategy by imitating the expert strategy using an imitation learning neural network. Finally, adopting a functional user strategy for efficient and sequential anti-jamming decisions. Simulation results indicate that the proposed method outperforms the RL-based anti-jamming method and DQN-based anti-jamming method regarding solving continuous-state spectrum anti-jamming problems without causing "curse of dimensionality" and providing greater robustness against channel fading and noise as well as when the jamming pattern changes.

Human-like route planning for automatic collision avoidance using generative adversarial imitation learning

Automatic collision avoidance systems are expected to eliminate maritime accidents caused by human error. Recent studies have shown that ships can prevent collisions in complex situations by using deep reinforcement learning (DRL). However, rewards must be designed to implement DRL, which can be challenging for ambiguous tasks, such as following the Convention on the International Regulations for Preventing Collisions at Sea (COLREGs). While some studies have used inverse reinforcement learning (IRL) to derive an appropriate reward for limited problems with small, discrete state spaces, this study introduced generative adversarial imitation learning (GAIL) and proposed a route planning algorithm that addresses the above limitations. We applied DRL to generate sample collision avoidance trajectories and demonstrate that the imitative route planner based on GAIL can reproduce these trajectories. Also, expert trajectories were collected in simulation experiments involving a well-experienced captain, and we attempted to generate collision avoidance routes that mimic human expert performance. By applying a convolutional neural network and other techniques to improve the planner's imitation capability, we established a sophisticated route planner that is consistent with captain's risk assessments and tolerant of noise.

A vision-based virtual fixture with robot learning for teleoperation

Teleoperation plays a key role for semi-automated tasks with high complexity in remote working environment. By integrating the interaction information and control strategy, the control performance can be guaranteed by the skilled operator manipulation in terms of stability and precision. However, due to a lack of prolonged specialized training, the manipulation characteristics, such as operation habits and tremor for green hands, the control performance of teleoperation cannot be guaranteed, especially for complicated and refined tasks. To this end, a vision-based virtual fixture with robot learning approach is proposed for teleoperation. In the proposed method, a dynamic movement primitives method is utilized to learn the human tutor or skilled operator manipulation skill and then generates the expert trajectories for training of green hands. Additionally, considering the instantaneity of manipulation, a vision-based virtual fixture is utilized to generate a force selector based on position error and provides a force guidance to the green hands in order to enhance the precision of control with expert level and reduce the operation pressure. Comparative experimental results demonstrated the performance of the developed approach for teleoperation.

Deep-PANTHER: Learning-Based Perception-Aware Trajectory Planner in Dynamic Environments

This letter presents Deep-PANTHER, a learning-based perception-aware trajectory planner for unmanned aerial vehicles (UAVs) in dynamic environments. Given the current state of the UAV, and the predicted trajectory and size of the obstacle, Deep-PANTHER generates multiple trajectories to avoid a dynamic obstacle while simultaneously maximizing its presence in the field of view (FOV) of the onboard camera. To obtain a computationally tractable real-time solution, imitation learning is leveraged to train a Deep-PANTHER policy using demonstrations provided by a multimodal optimization-based expert. Extensive simulations show replanning times that are two orders of magnitude faster than the optimization-based expert, while achieving a similar cost. By ensuring that each expert trajectory is assigned to one distinct student trajectory in the loss function, Deep-PANTHER can also capture the multimodality of the problem and achieve a mean squared error (MSE) loss with respect to the expert that is up to 18 times smaller than state-of-the-art (Relaxed) Winner-Takes-All approaches. Deep-PANTHER is also shown to generalize well to obstacle trajectories that differ from the ones used in training.

Optimizing pedestrian simulation based on expert trajectory guidance and deep reinforcement learning

Most traditional pedestrian simulation methods suffer from short-sightedness, as they often choose the best action at the moment without considering the potential congesting situations in the future. To address this issue, we propose a hierarchical model that combines Deep Reinforcement Learning (DRL) and Optimal Reciprocal Velocity Obstacle (ORCA) algorithms to optimize the decision process of pedestrian simulation. For certain complex scenarios prone to local optimality, we include expert trajectory imitation degree in the reward function, aiming to improve pedestrian exploration efficiency by designing simple expert trajectory guidance lines without constructing databases of expert examples and collecting priori datasets. The experimental results show that the proposed method presents great stability and generalizability, evidenced by its capability to adjust the behavioral strategy earlier for the upcoming congestion situations. The overall simulation time for each scenario is reduced by approximately 8-44% compared to traditional methods. After including the expert trajectory guidance, the convergence speed of the model is greatly improved, evidenced by the reduced 56-64% simulation time from the first exploration to the global maximum cumulative reward value. The expert trajectory establishes the macro rules while preserving the space for free exploration, avoiding local dilemmas, and achieving optimized training efficiency.

Learning Flight Control Systems from Human Demonstrations and Real-Time Uncertainty-Informed Interventions⋆

This paper describes a methodology for learning flight control systems from human demonstrations and interventions while considering the estimated uncertainty in the learned models. The proposed approach uses human demonstrations to train an initial model via imitation learning and then iteratively, improve its performance by using real-time human interventions. The aim of the interventions is to correct undesired behaviors and adapt the model to changes in the task dynamics. The learned model uncertainty is estimated in real-time via Monte Carlo Dropout and the human supervisor is cued for intervention via an audiovisual signal when this uncertainty exceeds a predefined threshold. This proposed approach is validated in an autonomous quadrotor landing task on both fixed and moving platforms. It is shown that with this algorithm, a human can rapidly teach a flight task to an unmanned aerial vehicle via demonstrating expert trajectories and then adapt the learned model by intervening when the learned controller performs any undesired maneuver, the task changes, and/or the model uncertainty exceeds a threshold.

Cross Apprenticeship Learning Framework: Properties and Solution Approaches

Apprenticeship learning is a framework in which an agent learns a policy to perform a given task in an environment using example trajectories provided by an expert. In the real world, one might have access to expert trajectories in different environments where system dynamics is different while the learning task is the same. For such scenarios, two types of learning objectives can be defined. One where the learned policy performs very well in one specific environment and another when it performs well across all environments. To balance these two objectives in a principled way, our work presents the cross apprenticeship learning (CAL) framework. This consists of an optimization problem where an optimal policy for each environment is sought while ensuring that all policies remain close to each other. This nearness is facilitated by one tuning parameter in the optimization problem. We derive properties of the optimizers of the problem as the tuning parameter varies. We identify conditions under which an agent prefers using the policy obtained from CAL over the traditional apprenticeship learning. Since the CAL problem is nonconvex, we provide a convex outer approximation. Finally, we demonstrate the attributes of our framework in the context of a navigation task in a windy gridworld environment.

Imitation and Transfer Learning for LQG Control

In this paper we study an imitation and transfer learning setting for Linear Quadratic Gaussian (LQG) control, where (i) the system dynamics, noise statistics and cost function are unknown and expert data is provided (that is, sequences of optimal inputs and outputs) to learn the LQG controller, and (ii) multiple control tasks are performed for the same system but with different LQG costs. We show that the LQG controller can be learned from a set of expert trajectories of length $n(l+2)-1$, with $n$ and $l$ the dimension of the system state and output, respectively. Further, the controller can be decomposed as the product of an estimation matrix, which depends only on the system dynamics, and a control matrix, which depends on the LQG cost. This data-based separation principle allows us to transfer the estimation matrix across different LQG tasks, and to reduce the length of the expert trajectories needed to learn the LQG controller to~$2n+m-1$ with $m$ the dimension of the inputs (for single-input systems with $l=2$, this yields approximately a $50\%$ reduction of the required expert data).

Autonomous underwater vehicle formation control and obstacle avoidance using multi-agent generative adversarial imitation learning

Autonomous underwater vehicle (AUV) is widely used in complex underwater missions such as bottom survey and data collection. Multiple AUVs can cooperatively complete tasks that single AUV cannot accomplish. Recently, multi-agent reinforcement learning (MARL) has been introduced to improve multi-AUV control in uncertain marine environments. However, it is very difficult and even unpractical to design effective and efficient reward functions for various tasks. In this paper, we implemented multi-agent generative adversarial imitation learning (MAGAIL) from expert demonstrated trajectories for formation control and obstacle avoidance of multi-AUV. In addition, decentralized training with decentralized execution framework was adopted to alleviate the communication problem in underwater environments. Moreover, to facilitate the discriminator to accurately judge the quality of AUV’s trajectory in the two tasks and increase the convergence speed, we improved upon MAGAIL by dividing the state–action pairs of expert trajectory for each AUV into two groups and updating discriminator by randomly selecting equal number of state–action pairs from both groups. Our experimental results on a simulated AUV system modeling Sailfish 210 of our lab in the Gazebo simulation environment show that MAGAIL allows control policies of multi-AUV to obtain a better performance than traditional multi-agent deep reinforcement learning from fine-tuned reward function — IPPO. Moreover, control policies trained via MAGAIL in simple tasks can generalize better to complex tasks than those trained via IPPO.

Online Apprenticeship Learning

In Apprenticeship Learning (AL), we are given a Markov Decision Process (MDP) without access to the cost function. Instead, we observe trajectories sampled by an expert that acts according to some policy. The goal is to find a policy that matches the expert's performance on some predefined set of cost functions. We introduce an online variant of AL (Online Apprenticeship Learning; OAL), where the agent is expected to perform comparably to the expert while interacting with the environment. We show that the OAL problem can be effectively solved by combining two mirror descent based no-regret algorithms: one for policy optimization and another for learning the worst case cost. By employing optimistic exploration, we derive a convergent algorithm with O(sqrt(K)) regret, where K is the number of interactions with the MDP, and an additional linear error term that depends on the amount of expert trajectories available. Importantly, our algorithm avoids the need to solve an MDP at each iteration, making it more practical compared to prior AL methods. Finally, we implement a deep variant of our algorithm which shares some similarities to GAIL, but where the discriminator is replaced with the costs learned by OAL. Our simulations suggest that OAL performs well in high dimensional control problems.

Energy management of hybrid electric vehicles based on inverse reinforcement learning

Many scholars have conducted research on reinforcement learning in energy management, and verified that reinforcement learning methods have certain advantages. The guiding direction of the interaction between the agent and environment is determined by the reward function. However, the current design of the reward function still has drawbacks. In most reinforcement learning energy management strategies, the design of the reward function is subjective and empirical, and describing the intention of the expert objectively is difficult. However, this condition cannot guarantee that the agent learns the optimal driving strategy under a given reward function. To address these problems, an energy management strategy based on reverse reinforcement learning is proposed to obtain the reward function weight under the expert trajectory, and then it is used to guide the behavior of the engine agent and the battery agent. Finally, the revised weights are re-entered into the forward reinforcement learning training. From the engine operating point, fuel consumption value, SOC change curve, and reward training process, indicate that the algorithm has certain advantages. In summary, the fuel-saving effect of the proposed algorithm is improved between 5%–10%.

Multigoal Visual Navigation With Collision Avoidance via Deep Reinforcement Learning

Learning to map the images acquired by a moving agent equipped with a camera sensor to motion commands for multigoal navigation is challenging. Most existing approaches are still struggling against collision avoidance, faster convergence, and generalization. In this article, a novel actor–critic architecture is presented to learn the optimal navigation policy. We introduce single-step reward observation and collision penalty to reshape the reinforcement learning (RL) reward function. The collision perception can be obtained by the reshaped reward function and treated as measurement information from the visual observation to avoid obstacles. Besides, expert trajectories are used to generate subgoals. A subgoal reward shaping is then proposed to accelerate policy learning with the expert knowledge of subgoals. In order to generate human-aware navigation policies, an observation-action consistency (OAC) model is introduced to ensure that the agent reaches the subgoals in turn, and moves toward the target. The whole training process is performed on a self-supervised RL approach, accompanied by an expert supervision signal. This method balances the exploration and exploitation, helping the proposed model to generalize to unseen goals. The training experiments on AI2-THOR show better performance and faster convergence speed, compared with the existing approaches. For the generalization capacity to unseen goals, the proposed method achieves the state-of-the-art success rate, with at least a 30% improvement of average episode collision.

Validation of virtual reality arthroscopy simulator relevance in characterising experienced surgeons

BackgroundVirtual reality (VR) simulation is particularly suitable for learning arthroscopy skills. Despite significant research, one drawback often outlined is the difficulty in distinguishing performance levels (Construct Validity) in experienced surgeons. Therefore, it seems adequate to search new methods of performance measurements using probe trajectories instead of commonly used metrics. HypothesisIt was hypothesized that a larger experience in surgical shoulder arthroscopy would be correlated with better performance on a VR shoulder arthroscopy simulator and that experienced operators would share similar probe trajectories. Materials & methodsAfter answering to standardized questionnaires, 104 trajectories from 52 surgeons divided into 2 cohorts (26 intermediates and 26 experts) were recorded on a shoulder arthroscopy simulator. The procedure analysed was the “loose body removal” in a right shoulder joint. 10 metrics were computed on the trajectories including procedure duration, overall path length, economy of motion and smoothness. Additionally, Dynamic Time Warping (DTW) was computed on the trajectories for unsupervised hierarchical clustering of the surgeons. ResultsExperts were significantly faster (Median 70.9s Interquartile range [56.4–86.3] vs. 116.1s [82.8–154.2], p<0.01), more fluid (4.6.105mm.s−3 [3.1.105–7.2.105] vs. 1.5.106mm.s−3 [2.6.106–3.5.106], p=0.05), and economical in their motion (19.3mm2 [9.1–25.9] vs. 33.8mm2 [14.8–50.5], p<0.01), but there was no significant difference in performance for path length (671.4mm [503.8–846.1] vs 694.6mm [467.0–1090.1], p=0.62). The DTW clustering differentiates two expertise related groups of trajectories with performance similarities, respectively including 48 expert trajectories for the first group and 52 intermediates and 4 expert trajectories for the second group (Sensitivity of 92%, Specificity of 100%). Hierarchical clustering with DTW significantly identified expert operators from intermediate operators and found trajectory similarities among 24/26 experts. ConclusionThis study demonstrated the Construct Validity of the VR shoulder arthroscopy simulator within groups of experienced surgeons. With new types of metrics simply based on the simulator's raw trajectories, it was possible to significantly distinguish levels of expertise. We demonstrated that clustering analysis with Dynamic Time Warping was able to reliably discriminate between expert operators and intermediate operators. Clinical RelevanceThe results have implications for the future of arthroscopic surgical training or post-graduate accreditation programs using virtual reality simulation. Level of evidenceIII; prospective comparative study.

Development of the methods for the class teacher effectiveness assessment

The problem of developing a methodology for assessing the effectiveness of the class teacher’s educational activity and the possibility of its implementation in the context of an educational establishment has been getting more vital. The developed methodology for assessing the class teacher performance efficiency was tested on the basis of four educational establishments in Nizhny Novgorod, which are the clinical internship bases to train future teachers and class teachers at Minin University (Nizhny Novgorod), who were experts carrying out the examination of the methodology for assessing the performance efficiency according to the proposed indicators. The developed methodology for assessing the class teacher performance efficiency can be used in the practice of educational organizations to develop a system for assessing the encouragement of the class teacher performance, supervising his professional growth, developing individual trajectories of professional improvement, analytical conclusions, and expertise.

Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft

We present Hierarchical Deep Q-Network (HDQfD) that won first place in the MineRL competition. The HDQfD works on imperfect demonstrations and utilizes the hierarchical structure of expert trajectories. We introduce the procedure of extracting an effective sequence of meta-actions and subgoals from the demonstration data. We present a structured task-dependent replay buffer and an adaptive prioritizing technique that allow the HDQfD agent to gradually erase poor-quality expert data from the buffer. In this paper, we present the details of the HDQfD algorithm and give the experimental results in the Minecraft domain.