Learn Control Policies Research Articles

Predictive feature selection for genetic policy search

Automatic learning of control policies is becoming increasingly important to allow autonomous agents to operate alongside, or in place of, humans in dangerous and fast-paced situations. Reinforcement learning (RL), including genetic policy search algorithms, comprise a promising technology area capable of learning such control policies. Unfortunately, RL techniques can take prohibitively long to learn a sufficiently good control policy in environments described by many sensors (features). We argue that in many cases only a subset of available features are needed to learn the task at hand, since others may represent irrelevant or redundant information. In this work, we propose a predictive feature selection framework that analyzes data obtained during execution of a genetic policy search algorithm to identify relevant features on-line. This serves to constrain the policy search space and reduces the time needed to locate a sufficiently good policy by embedding feature selection into the process of learning a control policy. We explore this framework through an instantiation called predictive feature selection embedded in neuroevolution of augmenting topology (NEAT), or PFS-NEAT. In an empirical study, we demonstrate that PFS-NEAT is capable of enabling NEAT to successfully find good control policies in two benchmark environments, and show that it can outperform three competing feature selection algorithms, FS-NEAT, FD-NEAT, and SAFS-NEAT, in several variants of these environments.

Local Models for data-driven learning of control policies for complex systems

An approach based on local learning, relying on Nadaraya–Watson models (NWMs), is introduced for the problem of deriving an automatic controller able to exploit data collected during the operation of some complex plant or system by a reference teacher (e.g., a human operator). Such learning approach is particularly useful when the system is too complex to be modeled accurately and/or the task cannot be easily formalized by a cost function, a situation which rules out classic approaches based, e.g., on dynamic programming. Here it is proved that local models are a suitable solution for a real-time employment, since they allow to incorporate new information directly and efficiently without the need of offline training, and new data immediately reflect in improvement of performance. To this purpose, convergence analysis of the method is provided, also considering the case where the reference controller introduces random variations in the training data. Finally, a simulation test, concerning the control of a mechanical system, is provided to showcase the use of local models in an applicative scenario.

Learning parametric dynamic movement primitives from multiple demonstrations

Learning from demonstration has shown to be a suitable approach for learning control policies (CPs). However, most previous studies learn CPs from a single demonstration, which results in limited scalability and insufficient generalization toward a wide range of applications in real environments. This paper proposes a novel approach to learn highly scalable CPs of basis movement skills from multiple demonstrations. In contrast to conventional studies with a single demonstration, i.e., dynamic movement primitives (DMPs), our approach efficiently encodes multiple demonstrations by shaping a parametric-attractor landscape in a set of differential equations. Assuming a certain similarity among multiple demonstrations, our approach learns the parametric-attractor landscape by extracting a small number of common factors in multiple demonstrations. The learned CPs allow the synthesis of novel movements with novel motion styles by specifying the linear coefficients of the bases as parameter vectors without losing useful properties of the DMPs, such as stability and robustness against perturbations. For both discrete and rhythmic movement skills, we present a unified learning procedure for learning a parametric-attractor landscape from multiple demonstrations. The feasibility and highly extended scalability of DMPs are demonstrated on an actual dual-arm robot.

Convergence Analysis of Ant Colony Learning

AbstractIn this paper, we study the convergence of the pheromone levels of Ant Colony Learning (ACL) in the setting of discrete state spaces and noiseless state transitions. ACL is a multi-agent approach for learning control policies that combines some of the principles found in ant colony optimization and reinforcement learning. Convergence of the pheromone levels in expected value is a necessary requirement for the convergence of the learning process to optimal control policies. In this paper, we derive upper and lower bounds for the pheromone levels and relate those to the learning parameters and the number of ants used in the algorithm. We also derive upper and lower bounds on the expected value of the pheromone levels.

A novel method for learning policies from variable constraint data

Many everyday human skills can be framed in terms of performing some task subject to constraints im- posed by the environment. Constraints are usually unobserv- able and frequently change between contexts. In this paper, we present a novel approach for learning (unconstrained) control policies from movement data, where observations come from movements under different constraints. As a key ingredient, we introduce a small but highly effective modi- fication to the standard risk functional, allowing us to make a meaningful comparison between the estimated policy and constrained observations. We demonstrate our approach on systems of varying complexity, including kinematic data from the ASIMO humanoid robot with 27 degrees of free- dom, and present results for learning from human demon- stration.

Closed-loop learning of visual control policies

In this paper we present a general, flexible framework for learning mappings from images to actions by interacting with the environment. The basic idea is to introduce a feature-based image classif...

Choice of approximator and design of penalty function for an approximate dynamic programming based control approach

This paper investigates the choice of function approximator for an approximate dynamic programming (ADP) based control strategy. The ADP strategy allows the user to derive an improved control policy given a simulation model and some starting control policy (or alternatively, closed-loop identification data), while circumventing the ‘curse-of-dimensionality’ of the traditional dynamic programming approach. In ADP, one fits a function approximator to state vs. ‘cost-to-go’ data and solves the Bellman equation with the approximator in an iterative manner. A proper choice and design of function approximator is critical for convergence of the iteration and the quality of final learned control policy, because an approximation error can grow quickly in the loop of optimization and function approximation. Typical classes of approximators used in related approaches are parameterized global approximators (e.g. artificial neural networks) and nonparametric local averagers (e.g. k-nearest neighbor). In this paper, we assert on the basis of some case studies and a theoretical result that a certain type of local averagers should be preferred over global approximators as the former ensures monotonic convergence of the iteration. However, a converged cost-to-go function does not necessarily lead to a stable control policy on-line due to the problem of over-extrapolation. To cope with this difficulty, we propose that a penalty term be included in the objective function in each minimization to discourage the optimizer from finding a solution in the regions of state space where the local data density is inadequately low. A nonparametric density estimator, which can be naturally combined with a local averager, is employed for this purpose.

Learning to Control the Performance of Batch Processes

The current shift in chemical industry interest away from high throughput production towards small amounts of high value added products has increased the industry's awareness of the issues associated with product and process performance. This fact poses a unique set of challenging control problems characterized by their ‘ill-definedness’, high process nonlinearities and imperfect modelling. In this work, a novel methodology is proposed for incremental learning of a control policy that can continuously improve product quality and operating performance.The new concept introduced here is the notion of a performance function that implicitly includes end-product quality constraints as a process goal and operational preferences which describe different modes of operation. Since plant information is often scarce and expensive to obtain, it is proposed that the performance function be learned by integrating together batch-to-batch data, intra-run measurements and a predictive process model. A new framework for this integration, called coarse code generalization, is proposed which revolves around the generation of an artificial set of batch runs using a hybrid process model. In this model, learning biases are incorporated through background knowledge that expresses run outcome sensitivities with respect to states and actions. Artificial batch runs provide an augmented data set which is used for inductive learning of the performance function. With a minimum amount of information on process performance available, a first approximation to the performance function is constructed and an optimization program is used to define a near-optimal control policy. As more plant data become available, the performance function refinement procedure permits also an increasing refinement of the learned control policy. Recipe changes that increase process performance can then be implemented on-line. A semi-batch reactor where an autocatalytic reaction takes place is used to present and test the methodology. Results obtained demonstrate that the methodology can cope successfully with the problems of both imperfect modelling and scarce information which are typical of the industrial environment. Also, coarse code generalization for performance control proves to be an ideal means of integrating inductive learning with first-principles models and other types of domain knowledge.