Cart-pole Problem Research Articles

Certifying Black-Box Policies With Stability for Nonlinear Control

Machine-learned black-box policies are ubiquitous for nonlinear control problems. Meanwhile, crude model information is often available for these problems from, e.g., linear approximations of nonlinear dynamics. We study the problem of certifying a black-box control policy with stability using model-based advice for nonlinear control on a single trajectory. We first show a general negative result that a naive convex combination of a black-box policy and a linear model-based policy can lead to instability, even if the two policies are both stabilizing. We then propose an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">adaptive <inline-formula><tex-math notation="LaTeX">$\lambda$</tex-math></inline-formula>-confident policy</i> , with a coefficient <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\lambda$</tex-math></inline-formula> indicating the confidence in a black-box policy, and prove its stability. With bounded nonlinearity, in addition, we show that the adaptive <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\lambda$</tex-math></inline-formula> -confident policy achieves a bounded competitive ratio when a black-box policy is near-optimal. Finally, we propose an online learning approach to implement the adaptive <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\lambda$</tex-math></inline-formula> -confident policy and verify its efficacy in case studies about the Cart-Pole problem and a real-world electric vehicle (EV) charging problem with covariate shift due to COVID-19.

Multi-Agent Reinforcement Learning with Information-sharing Constrained Policy Optimization for Global Cost Environment

Multi-agent Reinforcement Learning (MARL) is a machine learning method that solves problems by using multiple learning agents in a data-driven manner. Because of the advantage of utilizing multiple agents simultaneously, MARL has become an efficient solution to large-scale problems in a wide range of fields. However, as with general single-agent reinforcement learning, MARL requires trial and error to acquire the appropriate policies for each agent in the learning process. Therefore, how to guarantee performance and constraint satisfaction in MARL is a critical issue for application to real-world problems. In this study, we propose an Information-sharing Constrained Policy Optimization (IsCPO) method for MARL that guarantees constraint satisfaction during learning. In detail, IsCPO sequentially updates the policies of multiple agents in random order while sharing information of the surrogate costs and KL-divergence for evaluating the current and updated policies to the next agent. In addition, if there are no candidates of policies to be updated in accordance with the shared information, IsCPO skips updating the policies of the rest of the agents until the next iteration. As a result, IsCPO makes it possible to acquire the individual suboptimal policies of agents, satisfying constraints on global costs related to the state of the environment and the actions from multiple agents. We also introduce a practical algorithm for IsCPO that simplifies its implementation by adopting several mathematical approximations. Finally, we show the validity and effectiveness through simulation results on a multiple cart-pole problem and base station sleep control problem in a mobile network.

Policy-based optimization: single-step policy gradient method seen as an evolution strategy

This research reports on the recent development of black-box optimization methods based on single-step deep reinforcement learning and their conceptual similarity to evolution strategy (ES) techniques. It formally introduces policy-based optimization (PBO), a policy-gradient-based optimization algorithm that relies on a policy network to describe the density function of its forthcoming evaluations, and uses covariance estimation to steer the policy improvement process in the right direction. The specifics of the PBO algorithm are detailed, and the connections to evolutionary strategies are discussed. Relevance is assessed by benchmarking PBO against classical ES techniques on analytic functions minimization problems, and by optimizing various parametric control laws intended for the Lorenz attractor and the classical cartpole problem. Given the scarce existing literature on the topic, this contribution definitely establishes PBO as a valid, versatile black-box optimization technique, and opens the way to multiple future improvements building on the inherent flexibility of the neural networks approach.

Control with adaptive Q-learning: A comparison for two classical control problems

This paper evaluates adaptive Q-learning (AQL) and single-partition adaptive Q-learning (SPAQL), two algorithms for efficient model-free episodic reinforcement learning (RL), in two classical control problems (Pendulum and CartPole). AQL adaptively partitions the state–action space of a Markov decision process (MDP), while learning the control policy, i.e., the mapping from states to actions. The main difference between AQL and SPAQL is that the latter learns time-invariant policies, where the mapping from states to actions does not depend explicitly on the time step. This paper also proposes the SPAQL with terminal state (SPAQL-TS), an improved version of SPAQL tailored for the design of regulators for control problems. The time-invariant policies are shown to result in a better performance than the time-variant ones in both problems studied. These algorithms are particularly fitted to RL problems where the action space is finite, as is the case with the CartPole problem. SPAQL-TS solves the OpenAI Gym CartPole problem, while also displaying a higher sample efficiency than trust region policy optimization (TRPO), a standard RL algorithm for solving control tasks. Moreover, the policies learned by SPAQL are interpretable, while TRPO policies are typically encoded as neural networks, and therefore hard to interpret. Yielding interpretable policies while being sample-efficient are the major advantages of SPAQL.The code for the experiments is available at https://github.com/jaraujo98/SinglePartitionAdaptiveQLearning.

A Reconfigurable Two‐WSe2‐Transistor Synaptic Cell for Reinforcement Learning

Reward-modulated spike-timing-dependent plasticity (R-STDP) is a brain-inspired reinforcement learning (RL) rule, exhibiting potential for decision-making tasks and artificial general intelligence. However, the hardware implementation of the reward-modulation process in R-STDP usually requires complicated Si complementary metal-oxide-semiconductor (CMOS) circuit design that causes high power consumption and large footprint. Here, a design with two synaptic transistors (2T) connected in a parallel structure is experimentally demonstrated. The 2T unit based on WSe2 ferroelectric transistors exhibits reconfigurable polarity behavior, where one channel can be tuned as n-type and the other as p-type due to nonvolatile ferroelectric polarization. In this way, opposite synaptic weight update behaviors with multilevel (>6 bit) conductance states, ultralow nonlinearity (0.56/-1.23), and large Gmax /Gmin ratio of 30 are realized. By applying positive/negative reward to (anti-)STDP component of 2T cell, R-STDP learning rules are realized for training the spiking neural network and demonstrated to solve the classical cart-pole problem, exhibiting a way for realizing low-power (32 pJ per forward process) and highly area-efficient (100 µm2 ) hardware chip for reinforcement learning.

Variational quantum reinforcement learning via evolutionary optimization

Recent advances in classical reinforcement learning (RL) and quantum computation point to a promising direction for performing RL on a quantum computer. However, potential applications in quantum RL are limited by the number of qubits available in modern quantum devices. Here, we present two frameworks for deep quantum RL tasks using gradient-free evolutionary optimization. First, we apply the amplitude encoding scheme to the Cart-Pole problem, where we demonstrate the quantum advantage of parameter saving using amplitude encoding. Second, we propose a hybrid framework where the quantum RL agents are equipped with a hybrid tensor network-variational quantum circuit (TN-VQC) architecture to handle inputs of dimensions exceeding the number of qubits. This allows us to perform quantum RL in the MiniGrid environment with 147-dimensional inputs. The hybrid TN-VQC architecture provides a natural way to perform efficient compression of the input dimension, enabling further quantum RL applications on noisy intermediate-scale quantum devices.

Minibatch Recursive Least Squares Q-Learning.

The deep Q-network (DQN) is one of the most successful reinforcement learning algorithms, but it has some drawbacks such as slow convergence and instability. In contrast, the traditional reinforcement learning algorithms with linear function approximation usually have faster convergence and better stability, although they easily suffer from the curse of dimensionality. In recent years, many improvements to DQN have been made, but they seldom make use of the advantage of traditional algorithms to improve DQN. In this paper, we propose a novel Q-learning algorithm with linear function approximation, called the minibatch recursive least squares Q-learning (MRLS-Q). Different from the traditional Q-learning algorithm with linear function approximation, the learning mechanism and model structure of MRLS-Q are more similar to those of DQNs with only one input layer and one linear output layer. It uses the experience replay and the minibatch training mode and uses the agent's states rather than the agent's state-action pairs as the inputs. As a result, it can be used alone for low-dimensional problems and can be seamlessly integrated into DQN as the last layer for high-dimensional problems as well. In addition, MRLS-Q uses our proposed average RLS optimization technique, so that it can achieve better convergence performance whether it is used alone or integrated with DQN. At the end of this paper, we demonstrate the effectiveness of MRLS-Q on the CartPole problem and four Atari games and investigate the influences of its hyperparameters experimentally.

Hyper-parameter tuned deep Q network for area estimation of oil spills: a meta-heuristic approach

Oil Spills near shoreline are a major environmental hazard. Rapid estimation of spill perimeter provides a quick estimate of its area extent thus facilitating its quick removal. In this study; a meta-heuristic algorithm; “Gamma-Levy Hybrid Meta-heuristic with Conditional Evolution (GLHM-CE)” is proposed. The proposed algorithm is then used to evolve a distributed control strategy for a swarm of unmanned aerial vehicles for rapid confinement and estimation of spill perimeter. Every agent is controlled by a Deep Q Network whose Hyper-Parameters are tuned by GLHM-CE. Evaluation of GLHM-CE over 28 Blackbox Problems of CEC-2013,Special Session on Real-Parameter Optimization and its comparison with evolutionary algorithms like SHADE,Co-DE and JADE reveals that GLHM-CE successfully evades local minima and has a fast convergence. The effectiveness in hyper-parameter tuning of a Deep Q Network by GLHM-CE was evaluated over the quintessential CartPole problem from OpenAI Gym framework.

Deep Conservative Policy Iteration

Conservative Policy Iteration (CPI) is a founding algorithm of Approximate Dynamic Programming (ADP). Its core principle is to stabilize greediness through stochastic mixtures of consecutive policies. It comes with strong theoretical guarantees, and inspired approaches in deep Reinforcement Learning (RL). However, CPI itself has rarely been implemented, never with neural networks, and only experimented on toy problems. In this paper, we show how CPI can be practically combined with deep RL with discrete actions, in an off-policy manner. We also introduce adaptive mixture rates inspired by the theory. We experiment thoroughly the resulting algorithm on the simple Cartpole problem, and validate the proposed method on a representative subset of Atari games. Overall, this work suggests that revisiting classic ADP may lead to improved and more stable deep RL algorithms.

Multi-threading parallel reinforcement learning

With respect to the problem of the slow convergence of the traditional reinforcement learning algorithm in practical applications, we propose a novel multi-threading parallel reinforcement learning algorithm - MPRL. MPRL is mainly composed of two parts. One is the FCM-based reinforcement learning multi-threading partitioning method, which transforms the multi-threading partitioning problem into a clustering partition problem to obtain the optimal multi-threading partitioning solution. The other is the parallel reinforcement learning framework, which makes the parallel execution between the policy evaluation and the interaction with the environment. In the learning process, the experience replay is adopted to update the value function, which can also solve the problem of the non-convergence in the off-policy evaluation. Experimentally, the MPRL algorithm is applied to the windy grid world problem and the cart pole problem, and compared with Q-Learning, Sarsa and KCACL. The experimental results show that MPRL has a faster convergence rate and better convergence performance.

Multi-threading parallel reinforcement learning

With respect to the problem of the slow convergence of the traditional reinforcement learning algorithm in practical applications, we propose a novel multi-threading parallel reinforcement learning algorithm - MPRL. MPRL is mainly composed of two parts. One is the FCM-based reinforcement learning multi-threading partitioning method, which transforms the multi-threading partitioning problem into a clustering partition problem to obtain the optimal multi-threading partitioning solution. The other is the parallel reinforcement learning framework, which makes the parallel execution between the policy evaluation and the interaction with the environment. In the learning process, the experience replay is adopted to update the value function, which can also solve the problem of the non-convergence in the off-policy evaluation. Experimentally, the MPRL algorithm is applied to the windy grid world problem and the cart pole problem, and compared with Q-Learning, Sarsa and KCACL. The experimental results show that MPRL has a faster convergence rate and better convergence performance.

Self-Learning Control Using Dual Heuristic Programming with Global Laplacian Eigenmaps

In this paper, to solve nonlinear optimal control problems which can be modeled as Markov decision processes (MDPs), we present an online self-learning control algorithm called dual heuristic programming with global Laplacian Eigenmaps (GLEM-DHP). The GLEM-DHP algorithm makes use of GLEM, which is an improved manifold learning approach with global information, to learn the features for value function approximation of the MDP. Different from traditional feature representation methods using neural networks, the manifold-based features can be learned before the online learning process by collecting samples from the MDP. More importantly, in addition to local features, global information can also be utilized by using the geodesic minimum spanning tree (GMST) approach. Based on the theoretical property of GMST, it is shown that the GLEM-based features can represent the intrinsic geometric property of MDP states, which is beneficial to improve the performance of value function approximation and, hence, leads to better learning control properties. To compare the proposed method with previous learning control algorithms, the performance of GLEM-DHP is evaluated on two nonlinear control problems, which include the cart-pole problem and the ball-plate control problem. Simulation and experimental results show that the GLEM-DHP algorithm can obtain better learning control performance than previous learning control algorithms with manually designed features, as well as manifold features only with local information.

Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Primary-fuzzy-sets-based normal fuzzy reasoning methodology and its applications

Based on more useful and more heuristic primary fuzzy sets contained by conventionally used fuzzy sets, a novel normal fuzzy reasoning methodology is proposed in order to overcome the weakness of many conventional fuzzy systems. Analysis indicated that the compositional rule of inference Zadeh [IEEE Transactions on Systems, Man, and Cybernetics, 3, 28-44 (1973)] and related defuzzification schemes are not effective since they may generate unreasonable results such as the constant-speed problem and the unstable cart-pole problem. The new normal fuzzy system is a generalized framework of many conventional fuzzy systems such as Takagi-Sugeno's fuzzy system Sugeno and Yasukawa [IEEE Transactions on Fuzzy Systems, 1, 7-30 (1993)] and Takagi and Sugeno [IEEE Transactions on Systems, Man, and Cybemetics, SMC-15, 116-132 (1985)], Wang's fuzzy system Wang [Adaptive Fuzzy Systems and Control Design and Stability Analysis, Prentice-Hall, Englewood Cliffs, 1994], Lin's fuzzy system Lin [Neural Fuzzy Control Systems with Structure and Parameter Leaming, World Scientific, Singapore, 1994] and the fuzzy system using Yager's level set method Figueirdo et al. [IEEE Transactions on Fuzzy Systems, 1, 156-159 (1993)]. According to the normal fuzzy reasoning methodology, Takagi-Sugeno's fuzzy system can be constructed directly from high-level fuzzy rules without using the complex parameter estimation algorithm Lin [Neural Fuzzy Control Systems with Structure and Parameter Leaming, World Scientific, Singapore, 1994] and Sugeno and Yasukawa [IEEE Transactions on Fuzzy Systems, 1, 7-30 (1993)]. Simulations of the constant-speed problems, the fuzzy multiplications and the cart-pole system strongly indicated that the new normal fuzzy system is more effective and more robust than many conventional fuzzy systems. Therefore, the normal fuzzy reasoning methodology can be used to efficiently construct robust fuzzy systems.

New techniques for genetic development of a class of fuzzy controllers

Presents three novel techniques for enhancing the power of a genetic algorithm (GA) used to design fuzzy systems: a new context-dependent coding (CDC) technique, a simple chromosome reordering operator to maximize efficiency, and the coevolution of controller set tests to force competence in all areas of state space. These measures are shown to lead to a considerable improvement over conventional GAs when used to design controllers for a standard problem, such as the cart-pole problem. We use an analysis of GAs by L. Altenberg (1994) to determine a performance measure that demonstrates that our coding scheme and reordering operator improve the ability of the GA to organize itself and evolve chromosomal structures that not only produce high scores, but improve the search efficiency of the genetic operators. We investigate the algorithm in a controller to provide parallel parking maneuvers for mobile robots. It is shown that the controllers developed are robust to the systematic errors that inevitably arise when controllers are transferred from a simulated environment to the real world.

Artificial Neurogenesis: Applications to the Cart-Pole Problem and to an Autonomous Mobile Robot

A lot of recent research papers focus on the challenging problem of the combination of genetic algorithms and artificial neural networks. Developmental and molecular biology may be a source of inspiration for designing powerful artificial neurogenesis systems allowing the generation of complex modular structures. This paper describes in details such a neurogenesis model associated with an evolutionary process and its applications to the cart-pole problem and to the control of a mobile robot. Early results demonstrate the surprising efficiency of this methodology and give hints to continue the research towards the generation of more complex adaptive neural networks.

Automatic design of multidimensional controllers by means of an evolutionary algorithm

In this paper we outline some basic difficulties in the design of fuzzy controllers. We present a new controller concept, which avoids the drawbacks of classical fuzzy controllers. Our concept is suited for an automatic design especially. In order to show how one can develop such a new controller by means of an evolutionary algorithm, we use a simulation of the cart-pole problem with four input variables. In addition, we introduce a procedure, which allows a sequential optimization. Our approach is advantageously with respect to automotive applications.

A cartpole experiment benchmark for trainable controllers

The inverted pendulum problem, i.e., the cartpole, which is often used for demonstrating the success of neural network learning methods, is addressed. It is shown that a random search in weight space can quickly uncover coefficients (weights) for controllers that work over a wide range of initial conditions. This result indicates that success in finding a satisfactory neural controller is not sufficient proof for the effectiveness of unsupervised training methods. By analyzing the dynamics of the linear controller, the cartpole problem is reformulated to make it a more stringent test for neural training methods. A review of the literature on unsupervised training methods for cartpole controllers shows that the published results are difficult to compare and that for most of the methods there is not clear evidence of better performance than the random search method.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>