Grid World Research Articles

Clustered Autoencoded Variational Inverse Reinforcement Learning

Abstract Variational Auto-Encoder (VAE) is a handy and computationally friendly Bayesian tool for Inverse Reinforcement Learning (IRL) problems, with the native setting of absent reward functions in a Markov Decision Process (MDP). However, recent works mainly deal with single reward, which turn out to be insufficient for complex dynamic environments with multiple demonstrators of various characteristics (hence multiple reward functions). This paper extends the dimensionality of reward (from ℝ to ℝ K ) by incorporating a latent embedding and clustering step on top of a scalable Bayesian IRL model, which enhances her applicability to multi-reward scenarios. We introduce our method, Clustered Autoencoded Variational Inverse Reinforcement Learning (CAVIRL), which is able to approximate multiple posterior reward functions and learn the corresponding policies for experts of various characteristics and skills. As a by-product, the proposed model also thrives to determine the number of clusters K on her own, as opposed to the competing multi-reward imitation learning models that require K to be prespecified. We trained the proposed model within a grid world with multiple types of players, where we achieved 100% correctness in determining the number of players’ types and 80%-83.9% match between the model-learned policies and the players’ demonstrations from the data.

Robust learning for collision-free trajectory in space environment with limited a priori information

For on-orbit repair and space debris removal missions, a collision-free trajectory to the target is to be planned in a dynamically changing environment. Although such a trajectory can be learned in a ground simulator through repetitive trials, the established environment is often of limited precision due to space perturbations and measurement errors. Considering possible discrepancies between simulation and the real world, a tunnel, instead of a single trajectory, should be learned on the ground. In this paper, a robust planning method based on Q-learning is proposed for space missions with a priori environment information of limited precision. Based on a specific on-orbit repair scenario, reward functions in accordance with the multiple mission objectives are designed. The trajectories learned under different parameter randomization settings are combined and a robust tunnel is generated in the discrete grid world. By keeping the spacecraft inside the tunnel in the actual mission, collisions with the dynamic obstacles would be avoided and the goal of target rendezvous would be achieved. At last, a numerical simulation is carried out and the proposed method is validated under both nominal and randomized conditions.

Online Synthesis for Runtime Enforcement of Safety in Multiagent Systems

We study the problem of enforcing safety in multiagent systems at runtime by modifying the system behavior if a potential safety violation is detected. Traditional runtime enforcement methods that solve a reactive synthesis problem at design time have two significant drawbacks. First, these techniques do not scale as one has to take into account all possible behaviors from every agent, and this is computationally prohibitive. Second, these approaches require every agent to know the state of every other agent. We address these limitations through a new approach where online modifications to behavior are synthesized onboard every agent. There is an enforcer onboard every agent, which can modify the behavior of only the corresponding agent. In this approach, which is naturally decentralized, the enforcer on every agent has two components: 1) a pathfinder that corrects the behavior of the agent and 2) an ordering mechanism that dynamically modifies the priority of the agent. The current priority of an agent determines if the enforcer uses the pathfinder to modify the behavior of the agent. We derive an upper bound on the maximum deviation for any agent from its original behavior; that is, all agents make progress. We prove that the worst case synthesis time is quadratic in the number of agents at runtime as opposed to exponential at design time for the existing methods that rely on design-time computation merely. Additionally, we prove the completeness of the technique under some mild assumptions; that is, if the agents can progress safely, then enforcers will find this behavior. We test the technique in collision avoidance scenarios. For 50 agents in a 50 × 50 grid world modeling the common workspace for the agents, the online synthesis requires only a few seconds per agent whenever a potential collision is detected. In contrast, the centralized design-time synthesis of shields for a similar setting is intractable beyond four agents in a 5 × 5 grid world.

Top program construction and reduction for polynomial time Meta-Interpretive learning

Meta-Interpretive Learners, like most ILP systems, learn by searching for a correct hypothesis in the hypothesis space, the powerset of all constructible clauses. We show how this exponentially-growing search can be replaced by the construction of a Top program: the set of clauses in all correct hypotheses that is itself a correct hypothesis. We give an algorithm for Top program construction and show that it constructs a correct Top program in polynomial time and from a finite number of examples. We implement our algorithm in Prolog as the basis of a new MIL system, Louise, that constructs a Top program and then reduces it by removing redundant clauses. We compare Louise to the state-of-the-art search-based MIL system Metagol in experiments on grid world navigation, graph connectedness and grammar learning datasets and find that Louise improves on Metagol’s predictive accuracy when the hypothesis space and the target theory are both large, or when the hypothesis space does not include a correct hypothesis because of “classification noise” in the form of mislabelled examples. When the hypothesis space or the target theory are small, Louise and Metagol perform equally well.

CAPABILITY ITERATION NETWORK FOR ROBOT PATH PLANNING

Path planning is an important topic in robotics. Recently, value iteration based deep learning models have achieved good performance such as Value Iteration Network(VIN). However, previous methods suffer from slow convergence and low accuracy on large maps, hence restricted in path planning for agents with complex kinematics such as legged robots. Therefore, we propose a new value iteration based path planning method called Capability Iteration Network(CIN). CIN utilizes sparse reward maps and encodes the capability of the agent with state-action transition probability, rather than a convolution kernel in previous models. Furthermore, two training methods including end-to-end training and training capability module alone are proposed, both of which speed up convergence greatly. Several path planning experiments in various scenarios, including on 2D, 3D grid world and real robots with different map sizes are conducted. The results demonstrate that CIN has higher accuracy, faster convergence, and lower sensitivity to random seed compared to previous VI-based models, hence more applicable for real robot path planning.

Safe option-critic: learning safety in the option-critic architecture

AbstractDesigning hierarchical reinforcement learning algorithms that exhibit safe behaviour is not only vital for practical applications but also facilitates a better understanding of an agent’s decisions. We tackle this problem in the options framework (Sutton, Precup & Singh, 1999), a particular way to specify temporally abstract actions which allow an agent to use sub-policies with start and end conditions. We consider a behaviour assafethat avoids regions of state space with high uncertainty in the outcomes of actions. We propose an optimization objective that learnssafeoptions by encouraging the agent to visit states with higher behavioural consistency. The proposed objective results in a trade-off between maximizing the standard expected return and minimizing the effect of model uncertainty in the return. We propose a policy gradient algorithm to optimize the constrained objective function. We examine the quantitative and qualitative behaviours of the proposed approach in a tabular grid world, continuous-state puddle world, and three games from the Arcade Learning Environment: Ms. Pacman, Amidar, and Q*Bert. Our approach achieves a reduction in the variance of return, boosts performance in environments with intrinsic variability in the reward structure, and compares favourably both with primitive actions and with risk-neutral options.

Causal cognitive architecture 1: Integration of connectionist elements into a navigation-based framework

The brain-inspired Causal Cognitive Architecture 1 (CCA1) tightly integrates the sensory processing capabilities found in neural networks with many of the causal abilities found in human cognition. Causality emerges not from a central controlling stored program but directly from the architecture. Sensory input vectors are processed by robust association circuitry and then propagated to a navigational temporary map. Instinctive and learned objects and procedures are applied to the same temporary map, with a resultant navigation signal obtained. Navigation can similarly be for the physical world as well as for a landscape of higher cognitive concepts. There is good explainability for causal decisions. A simulation of the CCA1 controlling a search and rescue robot is presented with the goal of finding and rescuing a lost hiker within a grid world. A simulation of the CCA1 controlling a repair robot is presented that can predict the movement of a series of gears.

Integrated Double Estimator Architecture for Reinforcement Learning.

Estimation bias is an important index for evaluating the performance of reinforcement learning (RL) algorithms. The popular RL algorithms, such as Q -learning and deep Q -network (DQN), often suffer overestimation due to the maximum operation in estimating the maximum expected action values of the next states, while double Q -learning (DQ) and double DQN may fall into underestimation by using a double estimator (DE) to avoid overestimation. To keep the balance between overestimation and underestimation, we propose a novel integrated DE (IDE) architecture by combining the maximum operation and DE operation to estimate the maximum expected action value. Based on IDE, two RL algorithms: 1) integrated DQ (IDQ) and 2) its deep network version, that is, integrated double DQN (IDDQN), are proposed. The main idea of the proposed RL algorithms is that the maximum and DE operations are integrated to eliminate the estimation bias, where one estimator is stochastically used to perform action selection based on the maximum operation, and the convex combination of two estimators is used to carry out action evaluation. We theoretically analyze the reason of estimation bias caused by using nonmaximum operation to estimate the maximum expected value and investigate the possible reasons of underestimation existence in DQ. We also prove the unbiasedness of IDE and convergence of IDQ. Experiments on the grid world and Atari 2600 games indicate that IDQ and IDDQN can reduce or even eliminate estimation bias effectively, enable the learning to be more stable and balanced, and improve the performance effectively.

Relevance assignation feature selection method based on mutual information for machine learning

With the complication of the subjects and environment of the machine learning, feature selection methods have been used more frequently as an effective mean of dimension reduction. However, existing feature selection methods are deficient in striking a balance between the relevance evaluation accuracy with the searching efficiency. In this regard, the characteristics of the relevance between the feature set and the classification result are analyzed. Then, we propose our Relevance Assignation Feature Selection (RAFS) method based on the mutual information theory, which assigns the relevance evaluation according to the redundancy. With this method, we can estimate the contribution of each feature in a feature set, which is regarded as value of the feature and is used as the heuristic index in searching of the relevant features. A special dataset (“Grid World”) with strong interactive features is designed. Using the Grid World and six other natural datasets, the proposed method is compared with six other feature selection methods. Results show that in the Grid World dataset, the RAFS method can find correct relevant features with the probability above 90%, much higher than the others. In six other datasets, the RAFS method also has the best performance in the classification accuracy.

An effective maximum entropy exploration approach for deceptive game in reinforcement learning

Deceptive games are games that utilize the reward structure to keep the agent away from the global optimization and have been grown up to become a huge challenge in the field of deep reinforcement learning intelligent exploration. Most of the cutting-edge exploration approaches, such as count-based and curiosity-driven, even with intrinsic motivation, which achieves better performance in the sparse reward game, still easily fall into local optimal traps in the deceptive game. To address this shortfall, we introduce a further exploration approach called Maximum Entropy Explore (MEE). Based on entropy rewards and the off-policy actor-critic reinforcement learning algorithm, we divided the agent exploration policy into two independent parts, namely, the target policy and the explorer policy. The explorer policy, taking the maximum entropy of the target policy as the optimization goal, is used to interact with the environment and generated trajectories for the target policy. The target policy regards the maximization of external reward as the optimization goal to achieve the global solution. To alleviate the catastrophic forgetting problem which leads to the training of the agent not stabilized during the off-policy exploration phrase, the optimal experience replay is applied. An on-policy mode switch trick is used to validly prevent the unstable and diverge which caused by the deadly triad. We conduct experiments comparing our approach with state-of-the-art deep reinforcement learning algorithm and exploration methods in the grid world and StarCraft II environments with deceptive reward. The experiment indicates that the MME approach sets out to be in the present paper effectively avoids the deceptive reward trap and learns the global optimal strategy.

Reward Value-Based Goal Selection for Agents’ Cooperative Route Learning Without Communication in Reward and Goal Dynamism

This paper proposes a goal selection method to operate agents get maximum reward values per time by noncommunicative learning. In particular, that method aims to enable agents to cooperate along to dynamism of reward values and goal locations. Adaptation against to these dynamisms can enable agents to learn cooperative actions along to changing transportation tasks and changing incomes/rewards because of transporting tasks for heavy/valuable and light/valueless items in a storehouse. Concretely, this paper extends the previous noncommunicative cooperative action learning method (Profit minimizing reinforcement learning with oblivion of memory: PMRL-OM) and sets the two unified conditions combined of the number of time steps and the rewards. One of the unified conditions is calculated the approximated number of time steps if the expected reward values are the same each other for all purposes, and the other is the minimum number of time steps divided by the reward value. The proposed method makes all agents learn to achieve the purposes in the order in which they have the minimum number of the condition values. After that, each agent learns cooperative policy by PMRL-OM as the previous method. This paper analyzes the unified conditions and derives that the condition calculating the approximated time steps can be combined both evaluations with almost same weight unlike the value the other condition, that is, the condition can help the agents to select the appropriate purposes among them with the small difference in terms of the two evaluations. This paper tests empirically the performances of PMRL-OM with the two conditions by comparing with the PMRL-OM in three cases of grid world problems whose goal locations and reward values are changed dynamically. The results of this derive that the unified conditions perform better than PMRL-OM without some conditions in grid world problems. In particular, it is clear that the condition calculating the approximated time step can direct the appropriate goals for the agents.

How Should an Agent Practice?

We present a method for learning intrinsic reward functions to drive the learning of an agent during periods of practice in which extrinsic task rewards are not available. During practice, the environment may differ from the one available for training and evaluation with extrinsic rewards. We refer to this setup of alternating periods of practice and objective evaluation as practice-match, drawing an analogy to regimes of skill acquisition common for humans in sports and games. The agent must effectively use periods in the practice environment so that performance improves during matches. In the proposed method the intrinsic practice reward is learned through a meta-gradient approach that adapts the practice reward parameters to reduce the extrinsic match reward loss computed from matches. We illustrate the method on a simple grid world, and evaluate it in two games in which the practice environment differs from match: Pong with practice against a wall without an opponent, and PacMan with practice in a maze without ghosts. The results show gains from learning in practice in addition to match periods over learning in matches only.

A complementary learning systems approach to temporal difference learning

Complementary Learning Systems (CLS) theory suggests that the brain uses a ’neocortical’ and a ’hippocampal’ learning system to achieve complex behaviour. These two systems are complementary in that the ’neocortical’ system relies on slow learning of distributed representations while the ’hippocampal’ system relies on fast learning of pattern-separated representations. Both of these systems project to the striatum, which is a key neural structure in the brain’s implementation of Reinforcement Learning (RL). Current deep RL approaches share similarities with a ’neocortical’ system because they slowly learn distributed representations through backpropagation in Deep Neural Networks (DNNs). An ongoing criticism of such approaches is that they are data inefficient and lack flexibility. CLS theory suggests that the addition of a ’hippocampal’ system could address these criticisms. In the present study we propose a novel algorithm known as Complementary Temporal Difference Learning (CTDL), which combines a DNN with a Self-Organizing Map (SOM) to obtain the benefits of both a ’neocortical’ and a ’hippocampal’ system. Key features of CTDL include the use of Temporal Difference (TD) error to update a SOM and the combination of a SOM and DNN to calculate action values. We evaluate CTDL on Grid World, Cart–Pole and Continuous Mountain Car tasks and show several benefits over the classic Deep Q-Network (DQN) approach. These results demonstrate (1) the utility of complementary learning systems for the evaluation of actions, (2) that the TD error signal is a useful form of communication between the two systems and (3) that our approach extends to both discrete and continuous state and action spaces.

Engineered self-organization for resilient robot self-assembly with minimal surprise

In collective robotic systems, the automatic generation of controllers for complex tasks is still a challenging problem. Open-ended evolution of complex robot behaviors can be a possible solution whereby an intrinsic driver for pattern formation and self-organization may prove to be important. We implement such a driver in collective robot systems by evolving prediction networks as world models in pair with action–selection networks. Fitness is given for good predictions which causes a bias towards easily predictable environments and behaviors in the form of emergent patterns, that is, environments of minimal surprise. There is no task-dependent bias or any other explicit predetermination for the different qualities of the emerging patterns. A careful configuration of actions, sensor models, and the environment is required to stimulate the emergence of complex behaviors. We study self-assembly to increase the scenario’s complexity for our minimal surprise approach and, at the same time, limit the complexity of our simulations to a grid world to manage the feasibility of this approach. We investigate the impact of different swarm densities and the shape of the environment on the emergent patterns. Furthermore, we study how evolution can be biased towards the emergence of desired patterns. We analyze the resilience of the resulting self-assembly behaviors by causing damages to the assembled pattern and observe the self-organized reassembly of the structure. In summary, we evolved swarm behaviors for resilient self-assembly and successfully engineered self-organization in simulation. In future work, we plan to transfer our approach to a swarm of real robots.

The Nature of Decision-Making: Human Behavior vs. Machine Learning

Artificial agents have often been compared to humans in their ability to categorize images or play strategic games. However, comparisons between human and artificial agents are frequently based on the overall performance on a particular task, and not necessarily on the specifics of how each agent behaves. In this study, we directly compared human behaviour with a reinforcement learning (RL) model. Human participants and an RL agent navigated through different grid world environments with high- and low- value targets. The artificial agent consisted of a deep neural network trained to map pixel input of a 27x27 grid world into cardinal directions using RL. An epsilon greedy policy was used to maximize reward. Behaviour of both agents was evaluated on four different conditions. Results showed both humans and RL agents consistently chose the higher reward over a lower reward, demonstrating an understanding of the task. Though both humans and RL agents consider movement cost for reward, the machine agent considers the movement costs more, trading off the effort with reward differently than humans. We found humans and RL agents both consider long-term rewards as they navigate through the world, yet unlike humans, the RL model completely disregards limitations in movements (e.g. how many total moves received). Finally, we rotated pseudorandom grid arrangements to study how decisions change with visual differences. We unexpectedly found that the RL agent changed its behaviour due to visual rotations, yet remained less variable than humans. Overall, the similarities between humans and the RL agent shows the potential RL agents have of being an adequate model of human behaviour. Additionally, the differences between human and RL agents suggest improvements to RL methods that may improve their performance. This research compares the human mind with artificial intelligence, creating the opportunity for future innovation.

Inverse Risk-Sensitive Reinforcement Learning

This work addresses the problem of inverse reinforcement learning in Markov decision processes where the decision-making agent is risk-sensitive . In particular, a risk-sensitive reinforcement learning algorithm with convergence guarantees that makes use of coherent risk metrics and models of human decision-making which have their origins in behavioral psychology and economics is presented. The risk-sensitive reinforcement learning algorithm provides the theoretical underpinning for a gradient-based inverse reinforcement learning algorithm that seeks to minimize a loss function defined on the observed behavior. It is shown that the gradient of the loss function with respect to the model parameters is well defined and computable via a contraction map argument. Evaluation of the proposed technique is performed on a Grid World example, a canonical benchmark problem.

Adaptive Fuzzy Watkins: A New Adaptive Approach for Eligibility Traces in Reinforcement Learning

Reinforcement learning is one of the most reliable methods, which have been used to solve many problems. One of the best reinforcement learning family methods are temporal difference methods. The most important weakness of reinforcement learning methods, such as temporal difference methods, is that these methods have slow convergence rate. Many studies are devoted to solving this problem. One of the proposed solutions to this problem is eligibility traces. Owing to the nature of off-policy methods, combining eligibility traces with off-policy methods requires special attention. In the early learning process for Watkins method (one of the dominant eligibility traces methods), cutting eligibility traces during exploratory actions results in diminishing benefits of eligibility traces method. In this study, we propose a framework to combine eligibility traces with off-policy methods. This research attempts to properly use the information explored during action exploration of the agent; to this end, the decision about applying the eligibility traces during the exploratory actions of the agent is made by means of fuzzy adaptation. We apply this method to find the goal state in the static and dynamic grid world. We compare our approach against the state of the art techniques and show that it outperforms these techniques both in terms of averaged achieved reward and also the convergence time.

Rover-IRL: Inverse Reinforcement Learning With Soft Value Iteration Networks for Planetary Rover Path Planning

Planetary rovers, such as those currently on Mars, face difficult path planning problems, both before landing during the mission planning stages as well as once on the ground. In this work, we present a new approach to these planning problems based on inverse reinforcement learning using deep convolutional networks and value iteration networks (VIN) as important internal structures. VIN are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable. We propose a modification to the value iteration recurrence, referred to as the soft value iteration network (SVIN). SVIN is designed to produce more effective training gradients through the VIN. It relies on an internal soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action. We demonstrate the effectiveness of our proposed architecture in both a grid world dataset as well as a highly realistic synthetic dataset generated from currently deployed rover mission planning tools and real Mars imagery.

Stochastic Double Deep Q-Network

Estimation bias seriously affects the performance of reinforcement learning algorithms. The maximum operation may result in overestimation, while the double estimator operation often leads to underestimation. To eliminate the estimation bias, these two operations are combined together in our proposed algorithm named stochastic double deep Q-learning network (SDDQN), which is based on the idea of random selection. A tabular version of SDDQN is also given, named stochastic double Q-learning (SDQ). Both the SDDQN and SDQ are based on the double estimator framework. At each step, we choose to use either the maximum operation or the double estimator operation with a certain probability, which is determined by a random selection parameter. The theoretical analysis shows that there indeed exists a proper random selection parameter that makes SDDQN and SDQ unbiased. The experiments on Grid World and Atari 2600 games illustrate that our proposed algorithms can balance the estimation bias effectively and improve performance.

Multi-threading parallel reinforcement learning

With respect to the problem of the slow convergence of the traditional reinforcement learning algorithm in practical applications, we propose a novel multi-threading parallel reinforcement learning algorithm - MPRL. MPRL is mainly composed of two parts. One is the FCM-based reinforcement learning multi-threading partitioning method, which transforms the multi-threading partitioning problem into a clustering partition problem to obtain the optimal multi-threading partitioning solution. The other is the parallel reinforcement learning framework, which makes the parallel execution between the policy evaluation and the interaction with the environment. In the learning process, the experience replay is adopted to update the value function, which can also solve the problem of the non-convergence in the off-policy evaluation. Experimentally, the MPRL algorithm is applied to the windy grid world problem and the cart pole problem, and compared with Q-Learning, Sarsa and KCACL. The experimental results show that MPRL has a faster convergence rate and better convergence performance.