Imitation Learning Method Research Articles

Solving hard-exploration problems with counting and replay approach

The reinforcement learning agent has been very successful in many Atari 2600 games. However, while applied to a more complex and challenging environment, it is crucial to avoid falling into the local optimum, especially when the games contain many traps, ample action space, challenging scenarios, and sporadic successful episodes. In this case, using the intrinsic motivation method can easily fall into the local optimum. If the domain knowledge is excessively used, it is not applicable when encountering different game designs. Therefore, to enhance the agent’s ability to explore and avoid catastrophic forgetting due to the fades of intrinsic motivation, a Trajectory Evaluation Module is developed and integrated with ideas from the Count-Based Exploration and Trajectory Replay method.Moreover, our approach is integrated very well with the Self Imitation Learning method and works effectively for hard-exploration video games. Our policy is also evaluated with two video games: Super Mario Bros and Sonic the Hedgehog. The experiment results show that our Trajectory Evaluation Module can help the agent pass through various obstacles and scenarios, and successfully break through all levels of Super Mario Bros.

Learning and Assessing Optimal Dynamic Treatment Regimes Through Cooperative Imitation Learning

Dynamic Treatment Regimes (DTRs) are sets of sequential decision rules that can be adapted over time to treat patients with a specific pathology. DTR consists of alternative treatment paths and any of these treatments can be adapted depending on the patient’s characteristics. Reinforcement Learning (RL) and Imitation Learning (IL) approaches have been deployed for obtaining optimal treatment for a patient but, these approaches rely only on positive trajectories (i.e., treatments that concluded with positive responses of the patient). In contrast, negative trajectories (i.e., samples of non-responding treatments) are discarded, although these have valuable information content. We propose a Cooperative Imitation Learning (CIL) method that exploits information from both negative and positive trajectories to learn the optimal DTR. The proposed method reduces the chance of selecting any treatment which results in a negative outcome (negative response of the patient) during the medical examination. To validate our approach, we have considered a well-known DTR which is defined for the treatment of patients with alcohol addiction. Results show that our approach outperforms those that rely only on positive trajectories.

Active Inference Integrated With Imitation Learning for Autonomous Driving

Classical imitation learning methods suffer substantially from the learning hierarchical policies when the imitative agent faces an unobserved state by the expert agent. To address these drawbacks, we propose an online active learning through active inference approach that encodes the expert’s demonstrations based on observation-action to improve the learner’s future motion prediction. For this purpose, we provide a switching Dynamic Bayesian Network based on the dynamic interaction between the expert agent and another object in its surrounding as a reference model, which we exploit to initialize an incremental probabilistic learning model. This learning model grows and matures based on the free-energy formulation and message passing of active inference dynamically at discrete and continuous levels in an online active learning phase. In this scheme, generalized states of the learning world are represented as distance-vector, where it is the learner’s observation concerning its interaction with a moving object. Considering the distance vector entail intentions, it enables action prediction evaluation in a prospective sense. We illustrate these points using simulations of driving intelligent agents. The learning agent is trained by using long-term predictions from the generative learning model to reproduce the expert’s motion while learning how to select a suitable action through new experiences. Our results affirm that a Dynamic Bayesian optimal approach provides a principled framework and outperforms conventional reinforcement learning methods. Furthermore, it endorses the general formulation of action prediction as active inference.

Generative Adversarial Network for Imitation Learning from Single Demonstration

Imitation learning is an effective method for training an autonomous agent to accomplish a task by imitating expert behaviors in their demonstrations. However, traditional imitation learning methods require a large number of expert demonstrations in order to learn a complex behavior. Such a disadvantage has limited the potential of imitation learning in complex tasks where the expert demonstrations are not sufficient. In order to address the problem, we propose a Generative Adversarial Network-based model which is designed to learn optimal policies using only a single demonstration. The proposed model is evaluated on two simulated tasks in comparison with other methods. The results show that our proposed model is capable of completing considered tasks despite the limitation in the number of expert demonstrations, which clearly indicate the potential of our model.

Two-Dimensional Task Offloading for Mobile Networks: An Imitation Learning Framework

Mobile computing network is envisioned as a powerful framework to support the growing computation-intensive applications in the era of the Internet of Things (IoT). In this paper, we exploit the potential of a multi-layer network via a two-dimensional (2-D) task offloading scheme, which enables horizontal cooperations among the edge nodes. To minimize the average task offloading delay for all the mobile users, we formulate a mixed non-linear programming (MINLP) by jointly optimizing the 2-D offloading decisions and communication/computational resource allocation. To address this very challenging problem, we exploit the unique algorithmic structure of the optimal branch-and-bound (B&B) algorithm, and propose a novel Gaussian process imitation learning (GPIL) method to learn how to discover the shortcut for node searching in the B&B enumeration tree and significantly accelerate the B&B algorithm. When the network key parameters change, we further propose a novel recursive GPIL (RGPIL) method to agilely adapt to the new scenario with a fast policy update, where the new posterior distribution can be recursively updated based on a few new training data. Our simulation results show that the proposed method can achieve a near optimal solution with a significantly reduced complexity (e.g., a reduction of 98.7% in the number of searched nodes for a typical case). On this basis, the advantage of 2-D offloading scheme over the conventional schemes is also verified.

Deep Feature Representation Based Imitation Learning for Autonomous Helicopter Aerobatics

The drastic changes in flight parameters during aerobatics and the high instability of the system make the control of autonomous aerobatics unusually difficult. In this paper, we propose a deep feature representation based imitation learning method for autonomous aerobatics, which leverages expert demonstrations to efficiently learn the mapping of high-dimensional flight observations onto continuous actions (pitch, tail, and thrust). Different from the existing methods, our proposed method requires neither trajectory specification and alignment nor any assumptions and processing of system uncertainties, so it can greatly simplify the controller solution steps and reduce the computational burden. Particularly, our method uses the proposed deep feature representation network (DFR-network) to directly map the experts’ demonstration trajectory to a deep representation space spanned by a set of learned subspaces which represent the motion patterns with the same statistical property among demonstration trajectories. Various aerobatic maneuvers can be encoded in the deep representation space through a simple combination of embedding features. Accordingly, the proposed method can perform arbitrary aerobatic maneuvers by observing a limit set of expert demonstrations. The effectiveness of the deep feature representation based imitation learning method is verified on the real-world flight data. Experiments show that compared with the existing methods, our proposed method has higher control accuracy, stronger robustness and anti-interference ability.

Domain Adaptation for Imitation Learning Using Generative Adversarial Network.

Imitation learning is an effective approach for an autonomous agent to learn control policies when an explicit reward function is unavailable, using demonstrations provided from an expert. However, standard imitation learning methods assume that the agents and the demonstrations provided by the expert are in the same domain configuration. Such an assumption has made the learned policies difficult to apply in another distinct domain. The problem is formalized as domain adaptive imitation learning, which is the process of learning how to perform a task optimally in a learner domain, given demonstrations of the task in a distinct expert domain. We address the problem by proposing a model based on Generative Adversarial Network. The model aims to learn both domain-shared and domain-specific features and utilizes it to find an optimal policy across domains. The experimental results show the effectiveness of our model in a number of tasks ranging from low to complex high-dimensional.

Augmenting Policy Learning with Routines Discovered from a Single Demonstration

Humans can abstract prior knowledge from very little data and use it to boost skill learning. In this paper, we propose routine-augmented policy learning (RAPL), which discovers routines composed of primitive actions from a single demonstration and uses discovered routines to augment policy learning. To discover routines from the demonstration, we first abstract routine candidates by identifying grammar over the demonstrated action trajectory. Then, the best routines measured by length and frequency are selected to form a routine library. We propose to learn policy simultaneously at primitive-level and routine-level with discovered routines, leveraging the temporal structure of routines. Our approach enables imitating expert behavior at multiple temporal scales for imitation learning and promotes reinforcement learning exploration. Extensive experiments on Atari games demonstrate that RAPL improves the state-of-the-art imitation learning method SQIL and reinforcement learning method A2C. Further, we show that discovered routines can generalize to unseen levels and difficulties on the CoinRun benchmark.

Hybrid Imitation Learning Framework for Robotic Manipulation Tasks.

This study proposes a novel hybrid imitation learning (HIL) framework in which behavior cloning (BC) and state cloning (SC) methods are combined in a mutually complementary manner to enhance the efficiency of robotic manipulation task learning. The proposed HIL framework efficiently combines BC and SC losses using an adaptive loss mixing method. It uses pretrained dynamics networks to enhance SC efficiency and performs stochastic state recovery to ensure stable learning of policy networks by transforming the learner’s task state into a demo state on the demo task trajectory during SC. The training efficiency and policy flexibility of the proposed HIL framework are demonstrated in a series of experiments conducted to perform major robotic manipulation tasks (pick-up, pick-and-place, and stack tasks). In the experiments, the HIL framework showed about a 2.6 times higher performance improvement than the pure BC and about a four times faster training time than the pure SC imitation learning method. In addition, the HIL framework also showed about a 1.6 times higher performance improvement and about a 2.2 times faster training time than the other hybrid learning method combining BC and reinforcement learning (BC + RL) in the experiments.

다관절 로봇을 위한 TD3-GAIL 기반의 가상 모방 학습 방법

As robots replace human work, interests in robot teaching are increasing to implement various tasks and to quickly generate task motions. However, there are some difficulties to apply into articulated robots with multiple degrees of freedom. To solve this problem, we developed a virtual environment of robot manipulator for imitation learning and proposed a twin delayed deterministic generative adversarial imitation learning method which provides the imitation of expert’s demonstration. The developed virtual environment provides an efficient way to gather expert’s demonstration data. The learning performance of the proposed method was verified by comparing it with the conventional one based on the virtual environment for robot manipulators.

Research on sports action training method based on generative confrontation network model and artificial intelligence

In order to improve the effect of sports movement training, this paper builds a sports movement training model based on artificial intelligence technology based on the generation of confrontation network model. Moreover, in order to achieve the combination of model and model-free deep reinforcement learning algorithm, this paper implements the model’s guidance and constraints on deep reinforcement learning algorithm from the perspective of reward value and behavior strategy and divides the model into two situations. In one case, the existing or manually established expert rules are used as model constraints, which is equivalent to online learning by experts. In another case, expert samples are used as model constraints, and an imitation learning method based on generative adversarial networks is introduced. Moreover, using expert samples as training data, the mechanism that the model is guided by the reward value is combined with the model-free algorithm by generating a confrontation network structure. Finally, this paper studies the performance of the model through experimental research. The research results show that the model constructed in this paper has a certain effect.

Generative Adversarial Imitation Learning from Failed Experiences (Student Abstract)

Imitation learning provides a family of promising methods that learn policies from expert demonstrations directly. As a model-free and on-line imitation learning method, generative adversarial imitation learning (GAIL) generalizes well to unseen situations and can handle complex problems. In this paper, we propose a novel variant of GAIL called GAIL from failed experiences (GAILFE). GAILFE allows an agent to utilize failed experiences in the training process. Moreover, a constrained optimization objective is formalized in GAILFE to balance learning from given demonstrations and from self-generated failed experiences. Empirically, compared with GAIL, GAILFE can improve sample efficiency and learning speed over different tasks.

Vision-Language Navigation Policy Learning and Adaptation.

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms baseline methods by 10 percent on Success Rate weighted by Path Length (SPL) and achieves the state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore and adapt to unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7 to 11.7 percent).

Intelligent Edge: Leveraging Deep Imitation Learning for Mobile Edge Computation Offloading

In this work, we propose a new deep imitation learning (DIL)-driven edge-cloud computation offloading framework for MEC networks. A key objective for the framework is to minimize the offloading cost in time-varying network environments through optimal behavioral cloning. Specifically, we first introduce our computation offloading model for MEC in detail. Then we make fine-grained offloading decisions for a mobile device, and the problem is formulated as a multi-label classification problem, with local execution cost and remote network resource usage consideration. To minimize the offloading cost, we train our decision making engine by leveraging the deep imitation learning method, and further evaluate its performance through an extensive numerical study. Simulation results show that our proposal outperforms other benchmark policies in offloading accuracy and offloading cost reduction. At last, we discuss the directions and advantages of applying deep learning methods to multiple MEC research areas, including edge data analytics, dynamic resource allocation, security, and privacy, respectively.

Deterministic generative adversarial imitation learning

This paper proposes a deterministic generative adversarial imitation learning method which allows the robot to implement the motion planning task rapidly by learning from the demonstration data without reward function. In our method, the deep deterministic policy gradient method is used as the generator for learning the action policy on the basis of discriminator, and the demonstration data is input into the generator to ensure its stability. Three experiments on the push and pick-and-place tasks are conducted in the gym robotic environment. Results show that the learning speed of our method is much faster than the stochastic generative adversarial imitation learning method, and it can effectively learn from the demonstration data in different states of the task with higher learning stability. The proposed method can complete the motion planning task without environmental reward quickly and improve the stability of the training process.

Extraction of Driver Skill and Attention for Shared Autonomy

Recently, autonomous driving has been developed due to increase of traffic accident, traffic congestion and tailgating. In this study, we mainly focus on shared autonomy, where an autonomous system shares control of a vehicle with a driver. The system in shared autonomy has a role to help the driver avoid dangerous situations without interference in the driver’s control. To this end, the system should know the driver’s skill and attention when driving. This paper, therefore, integrates imitation learning and attribution method for this purpose. The proposed method demonstrated to visualize the visual field that driver especially paid attention.

Learning to Branch: Accelerating Resource Allocation in Wireless Networks

Resource allocation in wireless networks, such as device-to-device (D2D) communications, is usually formulated as mixed integer nonlinear programming (MINLP) problems, which are generally NP-hard and difficult to get the optimal solutions. Traditional methods to solve these MINLP problems are all based on mathematical optimization techniques, such as the branch-and-bound (B&B) algorithm that converges slowly and has forbidding complexity for real-time implementation. Therefore, machine leaning (ML) has been used recently to address the MINLP problems in wireless communications. In this paper, we use imitation learning method to accelerate the B&B algorithm. With invariant problem-independent features and appropriate problem-dependent feature selection for D2D communications, a good auxiliary prune policy can be learned in a supervised manner to speed up the most time-consuming branch process of the B&B algorithm. Moreover, we develop a mixed training strategy to further reinforce the generalization ability and a deep neural network (DNN) with a novel loss function to achieve better dynamic control over optimality and computational complexity. Extensive simulation demonstrates that the proposed method can achieve good optimality and reduce computational complexity simultaneously.

Two-Stage Model-Agnostic Meta-Learning With Noise Mechanism for One-Shot Imitation

Given that humans and animals can learn new behaviors in a short time by observing others, the question we need to consider is how to make robots behave like humans or animals, that is, through effective demonstration, robots can quickly understand and learn a new ability. One possible solution is imitation based meta-learning, but most of the related approaches are limited in a particular network structure or a specific task. Particularly, meta-learning methods based on gradient-update are prone to overfit. In this article, we propose a generic meta-learning algorithm that divides the learning process into two independent stages ( skill cloning and skill transfer ) with a noise mechanism which is compatible with any model. The skill cloning stage enables a good understanding of the demonstration, which helps the skill transfer stage when the robot applies the learned experience into new tasks. The experimental results show that our algorithm can alleviate the phenomenon of overfitting by introducing a noise mechanism . Our method not only performs well on the regression task but is significantly better than the existing state-of-the-art one-shot imitation learning methods in the same simulation environments (i.e., simulated pushing and simulated reaching).

Human Skeleton Detection, Modeling and Gesture Imitation Learning for a Social Purpose

Gesture recognition is topical in computer science and aims at interpreting human gestures via mathematical algorithms. Among the numerous applications are physical rehabilitation and imitation games. In this work, we suggest performing human gesture recognition within the context of a serious imitation game, which would aim at improving social interactions with teenagers with autism spectrum disorders. We use an artificial intelligence algorithm to detect the skeleton of the participant, then model the human pose space and describe an imitation learning method using a Gaussian Mixture Model in the Riemannian manifold.

Gated-Attention Architectures for Task-Oriented Language Grounding

To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using a Gated-Attention mechanism and learns a policy to execute the natural language instruction using standard reinforcement and imitation learning methods. We show the effectiveness of the proposed model on unseen instructions as well as unseen maps, both quantitatively and qualitatively. We also introduce a novel environment based on a 3D game engine to simulate the challenges of task-oriented language grounding over a rich set of instructions and environment states.