Imitation Policy Research Articles

Enhancing human–robot communication with a comprehensive language-conditioned imitation policy for embodied robots in smart cities

Integrating Embodied Robots into a smart city’s networked system can significantly enhance the city’s operational efficiency. These robots can be connected to the city’s network, receiving and transmitting data in real time for enhancing human–robot communications. Language-conditioned robot behavior plays a vital role in executing complex tasks by associating human commands or instructions with perception and actions. However, most language-conditioned policy-related research is limited to specific datasets and cannot generalize across different environments. In this study, we propose a novel imitation learning framework tailored for language-conditioned robotic tasks. Our framework includes specialized encoders designed for various benchmarks and utilizes two distinct models: the Transformer and Diffusion models. We rigorously evaluate this framework in three different robotic environments. Our findings indicate that the framework consistently delivers superior performance across multiple domains. Notably, we observe that the Transformer model is particularly effective in managing tasks with long trajectories, whereas the Diffusion model demonstrates enhanced proficiency in generating trajectories from limited training datasets. Our approach showcases remarkable generalization capabilities across a range of tasks and achieves significantly higher success rates in task completion.

Generative Upper-Level Policy Imitation Learning With Pareto-Improvement for Energy-Efficient Advanced Machining Systems.

The potential intelligence behind advanced machining systems (AMSs) offers positive contributions toward process improvement. Imitation learning (IL) offers an appealing approach to accessing this intelligence by observing demonstrations from skilled technologists. However, existing IL algorithms that implement single policy strategies have yet to consider realistic scenarios for complex AMS tasks, where the available demonstrations may have come from various experts. Moreover, most IL assumes that the expert's policy is optimal, preventing the learning from fulfilling the previously ignored green missions. This article introduces a novel three-phase policy search algorithm based on IL, enabling the learning of heterogeneous expert policies while balancing energy preferences. The first phase equips the agent with machining basics through upper-level policy learning, generating an imitation policy distribution with various decision-making principles. The second phase enhances energy conservation capabilities by employing Pareto-improvement learning and fine-tuning the agent's policies to a Pareto-policy manifold. The third phase produces outcomes and amplifies the efficacy of human feedback by utilizing ensemble policies. The experimental results indicate that the proposed method outperforms meta-heuristics, exhibiting superior solution quality and faster computation times compared to four diverse baseline methods, each with diverse samples.

Learning Physically Simulated Tennis Skills from Broadcast Videos

We present a system that learns diverse, physically simulated tennis skills from large-scale demonstrations of tennis play harvested from broadcast videos. Our approach is built upon hierarchical models, combining a low-level imitation policy and a high-level motion planning policy to steer the character in a motion embedding learned from broadcast videos. When deployed at scale on large video collections that encompass a vast set of examples of real-world tennis play, our approach can learn complex tennis shotmaking skills and realistically chain together multiple shots into extended rallies, using only simple rewards and without explicit annotations of stroke types. To address the low quality of motions extracted from broadcast videos, we correct estimated motion with physics-based imitation, and use a hybrid control policy that overrides erroneous aspects of the learned motion embedding with corrections predicted by the high-level policy. We demonstrate that our system produces controllers for physically-simulated tennis players that can hit the incoming ball to target positions accurately using a diverse array of strokes (serves, forehands, and backhands), spins (topspins and slices), and playing styles (one/two-handed backhands, left/right-handed play). Overall, our system can synthesize two physically simulated characters playing extended tennis rallies with simulated racket and ball dynamics. Code and data for this work is available at https://research.nvidia.com/labs/toronto-ai/vid2player3d/.

Evaluation of MPC-based Imitation Learning for Human-like Autonomous Driving

This work evaluates and analyzes the combination of imitation learning (IL) and differentiable model predictive control (MPC) for the application of human-like autonomous driving. We combine MPC with a hierarchical learning-based policy, and measure its performance in open-loop and closed-loop with metrics related to safety, comfort and similarity to human driving characteristics. We also demonstrate the value of augmenting open-loop behavioral cloning with closed-loop training for a more robust learning, approximating the policy gradient through time with the state space model used by the MPC. We perform experimental evaluations on a lane keeping control system, learned from demonstrations collected on a fixed-base driving simulator, and show that our imitative policies approach the human driving style preferences.

A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control

We present a simple and intuitive approach for interactive control of physically simulated characters. Our work builds upon generative adversarial networks (GAN) and reinforcement learning, and introduces an imitation learning framework where an ensemble of classifiers and an imitation policy are trained in tandem given pre-processed reference clips. The classifiers are trained to discriminate the reference motion from the motion generated by the imitation policy, while the policy is rewarded for fooling the discriminators. Using our GAN-like approach, multiple motor control policies can be trained separately to imitate different behaviors. In runtime, our system can respond to external control signal provided by the user and interactively switch between different policies. Compared to existing method, our proposed approach has the following attractive properties: 1) achieves state-of-the-art imitation performance without manually designing and fine tuning a reward function; 2) directly controls the character without having to track any target reference pose explicitly or implicitly through a phase state; and 3) supports interactive policy switching without requiring any motion generation or motion matching mechanism. We highlight the applicability of our approach in a range of imitation and interactive control tasks, while also demonstrating its ability to withstand external perturbations as well as to recover balance. Overall, our approach has low runtime cost and can be easily integrated into interactive applications and games.

Abstraction-Guided Policy Recovery from Expert Demonstrations

Behavior cloning is a method of automated decision-making that aims to extract meaningful information from expert demonstrations and reproduce the same behavior autonomously. It is unlikely that demonstrations will exhaustively cover the potential problem space, compromising the quality of automation when out-of-distribution states are encountered. Our approach RECO jointly learns both an imitation policy and recovery policy from expert data. The recovery policy steers the agent from unknown states back to the demonstrated states in the data set. While there is, per definition, no data available to learn the recovery policy, we exploit abstractions to generalize beyond the available data and simulate the recovery problem. When the most appropriate abstraction for the given data is unknown, our method selects the best recovery policy from a set generated by several candidate abstractions. In tabular domains, where we assume an agent must call to a human supervisor for help if it is in an unknown state, we show how RECO results in drastically fewer calls without compromising solution quality and with relatively few trajectories provided by an expert. We also introduce a continuous adaptation of our method and demonstrate the ability of RECO to recover an agent from states where its supervised learning-based imitation policy would otherwise fail.

Interactive-Imitation-Based Distributed Coordination Scheme for Smart Manufacturing

Concordant operations among automatic industrial devices (i.e., agents) play a significant role in achieving efficient smart manufacturing. Existing multiagent cooperation methods focus on centralized training with decentralized execution, which is unsuitable for the edge-based distributed industrial scenarios. To harmonize distributed devices’ actions and enhance the production efficiency, in this article, we propose an Interactive-Imitation-based Distributed Coordination (IIDC) algorithm. Specifically, we leverage the generative adversarial imitation learning (GAIL) model to direct one agent's actions by following external professional demonstrations. We also adopt the self-imitation learning (SIL) model to direct one agent's potential actions by following its own previous good experiences. As the imperfect demonstrations in the GAIL-based interagent imitation process may degrade the imitation accuracy, we present a confidence-based matching method to reduce the gap between the professional and imitative behaviors. Furthermore, during the SIL-based intra-agent imitation process, the rewardless explorations also lead to nonoptimal imitation policy. We then present a Stein variational policy gradient based self-imitation method to learn an expected optimal policy. We validate the IIDC algorithm's effectiveness via the sequential assembly task. Evaluation results demonstrate that the IIDC algorithm can enhance the production efficiency evidently.

Deep Bayesian Nonparametric Learning of Rules and Plans from Demonstrations with a Learned Automaton Prior

We introduce a method to learn imitative policies from expert demonstrations that are interpretable and manipulable. We achieve interpretability by modeling the interactions between high-level actions as an automaton with connections to formal logic. We achieve manipulability by integrating this automaton into planning, so that changes to the automaton have predictable effects on the learned behavior. These qualities allow a human user to first understand what the model has learned, and then either correct the learned behavior or zero-shot generalize to new, similar tasks. We build upon previous work by no longer requiring additional supervised information which is hard to collect in practice. We achieve this by using a deep Bayesian nonparametric hierarchical model. We test our model on several domains and also show results for a real-world implementation on a mobile robotic arm platform.

Bayesian Gaussian Mixture Model for Robotic Policy Imitation

A common approach to learn robotic skills is to imitate a demonstrated policy. Due to the compounding of small errors and perturbations, this approach may let the robot leave the states in which the demonstrations were provided. This requires the consideration of additional strategies to guarantee that the robot will behave appropriately when facing unknown states. We propose to use a Bayesian method to quantify the action uncertainty at each state. The proposed Bayesian method is simple to set up, computationally efficient, and can adapt to a wide range of problems. Our approach exploits the estimated uncertainty to fuse the imitation policy with additional policies. It is validated on a Panda robot with the imitation of three manipulation tasks in the continuous domain using different control input/state pairs.

Imitation Learning via Kernel Mean Embedding

Imitation learning refers to the problem where an agent learns a policy that mimics the demonstration provided by the expert, without any information on the cost function of the environment. Classical approaches to imitation learning usually rely on a restrictive class of cost functions that best explains the expert's demonstration, exemplified by linear functions of pre-defined features on states and actions. We show that the kernelization of a classical algorithm naturally reduces the imitation learning to a distribution learning problem, where the imitation policy tries to match the state-action visitation distribution of the expert. Closely related to our approach is the recent work on leveraging generative adversarial networks (GANs) for imitation learning, but our reduction to distribution learning is much simpler, robust to scarce expert demonstration, and sample efficient. We demonstrate the effectiveness of our approach on a wide range of high-dimensional control tasks.

TAX COMPETITION, RELATIVE PERFORMANCE, AND POLICY IMITATION

Rather than about their absolute payoffs, governments in fiscal competition often seem to care about their performance relative to other governments. Moreover, they often appear to mimic policies observed elsewhere. I study such behavior in a standard tax competition game. Both with relative payoff concerns and for imitative policies, evolutionary stability for games with finitely many players is the appropriate solution concept. Independently of the number of jurisdictions involved, an evolutionarily stable tax policy coincides with the competitive outcome of a tax competition game with infinitely many players. It, thus, involves drastic efficiency losses.

Surviving in the Global Market: ‘Americanisation’ and the Relaunch of Italy's Car Industry after the Second World War

AbstractThis contribution sheds light on the successful recovery strategies developed by the Italian car industry after the Second World War and in particular the ‘innovation through creative imitation policy’ that characterised its relationship with American producers. This policy was feasible because the Italian car sector, and Fiat in particular, was fairly well prepared to measure itself against the American model and aimed at technology-transfer-based Americanisation in order to make itself competitive in the global market. Mass production adapted to Italian plants and necessities, combined with Italian style and design, was seen as a path towards international relaunch. Fiat's subsequent development of its own technological resources and innovation strategy enabled it to weather the ups and downs of its long-term business history. When in 2009 an alliance with Chrysler was signed, Fiat's technological capabilities in the field of innovative ecological and fuel-saving technology and its long-standing attention to style and interior design were judged important inputs to help the American car-maker (and Fiat itself) to survive in the global market.

Tax Competition, Relative Performance and Policy Imitation

Rather than about absolute payoffs, governments in fiscal competition often seem to care about their performance relative to other governments. Moreover, they often appear to mimic policies observed elsewhere. We study such behaviour in a tax competition game with mobile capital a la Zodrow-Mieszkowski. Both with relative payoff concerns and for imitative policies, evolutionary stability is the appropriate solution concept. It renders tax competition more aggressive than with best-reply policies (Nash equilibrium). Whatever the number of jurisdictions involved, an evolutionary stable tax policy coincides with the competitive outcome of a tax competition game played among infinitely many governments. Tax competition among boundedly rational governments, thus, involves drastic efficiency losses.