Model-free Reinforcement Learning Algorithm Research Articles

SQLR: Short-Term Memory Q-Learning for Elastic Provisioning

As a growing number of service and application providers choose cloud networks to deliver their services on a software-as-a-service (SaaS) basis, cloud providers need to make their provisioning systems agile enough to meet service level agreements (SLAs). At the same time, they should guard against over-provisioning, which limits their capacity to accommodate more tenants. To this end, we propose Shortterm memory Q-Learning pRovisioning (SQLR, pronounced as “scaler”), a system employing a customized variant of the modelfree reinforcement learning algorithm. It can reuse contextual knowledge learned from one workload to optimize the number of virtual machines (resources) allocated to serve other workload patterns. With minimal overhead, SQLR achieves comparable results to systems where resources are unconstrained. Our experiments show that we can reduce the amount of provisioned resources by about 20% with less than 1% overall service unavailability (due to blocking), while delivering similar response times to those of an over-provisioned system.

Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning

Model-based reinforcement learning (RL) is more sample efficient than model-free RL by using imaginary trajectories generated by the learned dynamics model. When the model is inaccurate or biased, imaginary trajectories may be deleterious for training the action-value and policy functions. To alleviate such problem, this paper proposes to adaptively reweight the imaginary transitions, so as to reduce the negative effects of poorly generated trajectories. More specifically, we evaluate the effect of an imaginary transition by calculating the change of the loss computed on the real samples when we use the transition to train the action-value and policy functions. Based on this evaluation criterion, we construct the idea of reweighting each imaginary transition by a well-designed meta-gradient algorithm. Extensive experimental results demonstrate that our method outperforms state-of-the-art model-based and model-free RL algorithms on multiple tasks. Visualization of our changing weights further validates the necessity of utilizing reweight scheme.

Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction

Value function is the central notion of Reinforcement Learning (RL). Value estimation, especially with function approximation, can be challenging since it involves the stochasticity of environmental dynamics and reward signals that can be sparse and delayed in some cases. A typical model-free RL algorithm usually estimates the values of a policy by Temporal Difference (TD) or Monte Carlo (MC) algorithms directly from rewards, without explicitly taking dynamics into consideration. In this paper, we propose Value Decomposition with Future Prediction (VDFP), providing an explicit two-step understanding of the value estimation process: 1) first foresee the latent future, 2) and then evaluate it. We analytically decompose the value function into a latent future dynamics part and a policy-independent trajectory return part, inducing a way to model latent dynamics and returns separately in value estimation. Further, we derive a practical deep RL algorithm, consisting of a convolutional model to learn compact trajectory representation from past experiences, a conditional variational auto-encoder to predict the latent future dynamics and a convex return model that evaluates trajectory representation. In experiments, we empirically demonstrate the effectiveness of our approach for both off-policy and on-policy RL in several OpenAI Gym continuous control tasks as well as a few challenging variants with delayed reward.

An Experience Replay Method Based on Tree Structure for Reinforcement Learning

Q-Learning, which is a well-known model-free reinforcement learning algorithm, a learning agent explores an environment to update a state-action function. In reinforcement learning, the agent does not require information about the environment in advance, so an interaction between the agent and the environment is for collecting the real experiences that is also an expensive and time-consuming process. Therefore, to reduce the burden of the interaction, sample efficiency becomes an important role in reinforcement learning. This study proposes an adaptive tree structure integrating with experience replay for Q-Learning, called ERTS-Q. In ERTS-Q method, Q-Learning is used for policy learning, a tree structure establishes a virtual model which perceives two different continuous states after each state transaction, and then the variations of the continuous state are calculated. After each state transition, all states with highly similar variation are aggregated into the same leaf nodes. Otherwise, new leaf nodes will be produced. For experience replay, the tree structure predicts the next state and reward based on the statistical information that is stored in tree nodes. The virtual experiences produced by the tree structure are used for achieving extra learning. Simulations of the mountain car and a maze environment are performed to verify the validity of the proposed modeling learning approach.

UAV Networks Against Multiple Maneuvering Smart Jamming With Knowledge-Based Reinforcement Learning

The unmanned aerial vehicles (UAVs) networks are very vulnerable to smart jammers that can choose their jamming strategy based on the ongoing channel state accordingly. Although reinforcement learning (RL) algorithms can give UAV networks the ability to make intelligent decisions, the high-dimensional state space makes it difficult for algorithms to converge quickly. This article proposes a knowledge-based RL method, which uses domain knowledge to compress the state space that the agent needs to explore and then improve the algorithm convergence speed. Specifically, we use the inertial law of the aircraft and the law of signal attenuation in free space to guide the highly efficient exploration of the UAVs in the state space. We incorporate the performance indicators of the receiver and the subjective value of the task into the design of the reward function, and build a virtual environment for pretraining to accelerate the convergence of anti-jamming decisions. In addition, the algorithm proposed is completely based on observable data, which is more realistic than those studies that assume the position or the channel strategy of the jammer. The simulation shows that the proposed algorithm can outperform the benchmarks of model-free RL algorithm in terms of converge speed and averaged reward.

Output event-triggered tracking synchronization of heterogeneous systems on directed digraph via model-free reinforcement learning

In this study, an event-triggered controller for the output tracking synchronization problem in heterogeneous multi-agent systems on a directed digraph is developed. Initially, the design of the distributed observer is proposed to estimate the information of the leader. Owing to the communication among agents, there exists a coupling behavior in the design of controllers for followers. The separation theorem is exploited to realize decoupling between agents. Then, an optimal conventional controller and event-triggered controller are designed to achieve the synchronization service of the output tracking problem for agents; the latter also improves the efficiency precisely by deliberately aperiodic sampling. Additionally, we prove that the inter-event time interval is strictly positive in the event-triggered controller; that is, the Zeno behavior can be avoided. Furthermore, the optimal control gain is obtained in a model-free manner. More specifically, a model-free reinforcement learning algorithm is investigated to learn the control gain, which does not require the followers’ dynamics. Finally, two simulation examples show the availability of the algorithm.

Efficient Robot Skills Learning with Weighted Near-Optimal Experiences Policy Optimization

Autonomous learning of robotic skills seems to be more natural and more practical than engineered skills, analogous to the learning process of human individuals. Policy gradient methods are a type of reinforcement learning technique which have great potential in solving robot skills learning problems. However, policy gradient methods require too many instances of robot online interaction with the environment in order to learn a good policy, which means lower efficiency of the learning process and a higher likelihood of damage to both the robot and the environment. In this paper, we propose a two-phase (imitation phase and practice phase) framework for efficient learning of robot walking skills, in which we pay more attention to the quality of skill learning and sample efficiency at the same time. The training starts with what we call the first stage or the imitation phase of learning, updating the parameters of the policy network in a supervised learning manner. The training set used in the policy network learning is composed of the experienced trajectories output by the iterative linear Gaussian controller. This paper also refers to these trajectories as near-optimal experiences. In the second stage, or the practice phase, the experiences for policy network learning are collected directly from online interactions, and the policy network parameters are updated with model-free reinforcement learning. The experiences from both stages are stored in the weighted replay buffer, and they are arranged in order according to the experience scoring algorithm proposed in this paper. The proposed framework is tested on a biped robot walking task in a MATLAB simulation environment. The results show that the sample efficiency of the proposed framework is much higher than ordinary policy gradient algorithms. The algorithm proposed in this paper achieved the highest cumulative reward, and the robot learned better walking skills autonomously. In addition, the weighted replay buffer method can be made as a general module for other model-free reinforcement learning algorithms. Our framework provides a new way to combine model-based reinforcement learning with model-free reinforcement learning to efficiently update the policy network parameters in the process of robot skills learning.

Research on sports action training method based on generative confrontation network model and artificial intelligence

In order to improve the effect of sports movement training, this paper builds a sports movement training model based on artificial intelligence technology based on the generation of confrontation network model. Moreover, in order to achieve the combination of model and model-free deep reinforcement learning algorithm, this paper implements the model’s guidance and constraints on deep reinforcement learning algorithm from the perspective of reward value and behavior strategy and divides the model into two situations. In one case, the existing or manually established expert rules are used as model constraints, which is equivalent to online learning by experts. In another case, expert samples are used as model constraints, and an imitation learning method based on generative adversarial networks is introduced. Moreover, using expert samples as training data, the mechanism that the model is guided by the reward value is combined with the model-free algorithm by generating a confrontation network structure. Finally, this paper studies the performance of the model through experimental research. The research results show that the model constructed in this paper has a certain effect.

Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator

In recent years, reinforcement learning (RL) has been widely used to solve multi-agent navigation tasks, and a high-fidelity level for the simulator is critical to narrow the gap between simulation and real-world tasks. However, high-fidelity simulators have high sampling costs and bottleneck the training model-free RL algorithms. Hence, we propose a Multi-Fidelity Simulator framework to train Multi-Agent Reinforcement Learning (MFS-MARL), reducing the total data cost with samples generated by a low-fidelity simulator. We apply the depth-first search to obtain local feasible policies on the low-fidelity simulator as expert policies to help the original reinforcement learning algorithm explore. We built a multi-vehicle simulator with variable fidelity levels to test the proposed method and compared it with the vanilla Soft Actor-Critic (SAC) and expert actor methods. The results show that our method can effectively obtain local feasible policies and can achieve a 23% cost reduction in multi-agent navigation tasks.

Perception-Action Coupling Target Tracking Control for a Snake Robot via Reinforcement Learning.

Visual-guided locomotion for snake-like robots is a challenging task, since it involves not only the complex body undulation with many joints, but also a joint pipeline that connects the vision and the locomotion. Meanwhile, it is usually difficult to jointly coordinate these two separate sub-tasks as this requires time-consuming and trial-and-error tuning. In this paper, we introduce a novel approach for solving target tracking tasks for a snake-like robot as a whole using a model-free reinforcement learning (RL) algorithm. This RL-based controller directly maps the visual observations to the joint positions of the snake-like robot in an end-to-end fashion instead of dividing the process into a series of sub-tasks. With a novel customized reward function, our RL controller is trained in a dynamically changing track scenario. The controller is evaluated in four different tracking scenarios and the results show excellent adaptive locomotion ability to the unpredictable behavior of the target. Meanwhile, the results also prove that the RL-based controller outperforms the traditional model-based controller in terms of tracking accuracy.

Priority-Aware Reinforcement-Learning-Based Integrated Design of Networking and Control for Industrial Internet of Things

Industrial Internet of Things (IIoT) envisions the tight coupling of numerous critical industrial manufacturing subsystems, such as control, networking, and computing through the ubiquitous Internet of Things technologies. Nonetheless, such interconnectivity poses significant challenges to the successful management and operation of massively distributed industrial manufacturing systems. Without carefully integrated system design, the nonoptimal management and operation of highly intertwined subsystems can lead to the loss of productivity and ultimately the value of factories and plants. To address this issue, in this article, we conduct the integrated design that is capable of simultaneously configuring both control and networking subsystems in IIoT with consideration for their inherent interdependencies. We first analyze the performance of the dynamic backoff exponential (BE) in IEEE 802.15.4 carrier-sense multiple access (CSMA) and show the performance improvement of dynamic BE. We then design a model-free reinforcement learning algorithm to configure the control and networking subsystems automatically via systematic trial and error, as it is impractical to build a model for a highly intertwined complex IIoT system. Considering the time-sensitive characteristics of IIoT systems, we design priority-aware policies based on importance among networking traffic (i.e., sensing traffic and actuation traffic) to improve the convergence speed. The experimental results demonstrate that our priority-aware reinforcement-learning-based integrated design can successfully reconfigure the complex and highly intertwined IIoT system at runtime with a minimal convergence time. Besides, our approach reduces convergence time by 37.5%, and energy consumption by 9.2%, compared to the standard reinforcement learning approach.

An optimal policy for joint compression and transmission control in delay-constrained energy harvesting IoT devices

Energy-efficient communication remains one of the key requirements of the Internet of Things (IoT) platforms. The concern on energy consumption can be mitigated by exploiting technical ploys to reduce the volume of data for transmission (e.g., via sensing data compression) as well as by resorting to technological advancements (e.g., energy harvesting). However, these mitigating measures carry their own cost, which is the additional complexity of control and optimization in the digital communication chain. In particular, compression ratio is another control knob that needs adjusting besides the usual transmission parameters. Also, with the random and sporadic nature of the harvested energy, the goal shifts from mere energy conservation to judicious consumption of the renewable energy in a foresighted manner. In this paper, we assume an energy-harvesting IoT device that is tasked with (loss-lessly) compressing and reporting delay-constrained sensing events to an IoT gateway over a time-varying wireless channel. We are interested in computing an optimal policy for joint compression and transmission control adaptive to the node’s energy availability, transmission buffer length, as well as its wireless channel conditions. We cast the problem as a Constrained Markov Decision Process (CMDP), and propose a two-timescale model-free reinforcement learning (RL) algorithm that is able to shape the optimal control policy in the absence of the statistical knowledge of the underlying system dynamics. Exhaustive simulation experiments are conducted to investigate the convergence of the learning algorithm, to explore the impacts of different system parameters (such as: the rate of sensing events, the energy arrival rate, and battery capacity) on the performance of the proposed policy, as well as to compare against some baseline schemes.

Reinforcement Learning Based Decision Making of Operational Indices in Process Industry Under Changing Environment

The plant-wide production process is composed of multiple unit processes, in which the operational indices of each unit process are assigned and adjusted according to product quality, yield, and actual operating modes. Due to the changing operational conditions of the production process, the operational indices cannot be effectively adjusted by most of the model-based methods or evolutionary computation. In this article, the decision making of operational indices is formulated as a continuous state, continuous action reinforcement learning (RL) problem and a model-free RL algorithm is proposed, which learns a decision policy to determine the operational indices according to the actual operational conditions. Different from the existing methods, this article presents a multiactor networks ensemble algorithm and an actor-critic framework with stochastic policy to avoid falling into local optimums. The relatively overall optimal policy is obtained by extracting the results of parallel training of multiactor networks, which guarantees the optimality of the obtained policy. In addition, by using the experience replay, it is particularly valuable to effectively deal with the problem that lacking of sampling data in the model-free RL. Simulation studies are conducted on actual data of a mineral processing plant and the results demonstrate the effectiveness of the proposed algorithm.

PMA-DRL: A parallel model-augmented framework for deep reinforcement learning algorithms

In recent years, aided by deep neural networks (DNNs), reinforcement learning (RL) algorithms have been achieving great success in more and more tasks. In general, model-free RL algorithms are widely applicable, but sometimes suffer from low sample efficiency. Although the environment dynamics can be incorporated into model-free RL algorithms to enhance sample efficiency, which can be transformed into the joint algorithm of model-free and model-based RL algorithms, the model bias of dynamics model may still hurt the performance. Another attractive study direction for RL algorithms, is using parallel strategy. Meanwhile, parallel RL algorithms can achieve outstanding results under less time cost, but are still with low sample efficiency, because most of such algorithms are vanilla model-free ones. Therefore, aiming to enhance the model performance for RL algorithms with DNNs, in this paper, we propose a novel parallel model-augmented framework, called PMA-DRL, to combine the dynamics model and parallel RL algorithms together. First, we introduce a local dynamics model (LDM) for each local agent model (LAM) in parallel RL algorithms. Next, we propose to build a better LDM by using an ensemble of LDMs, and such ensemble can help to reduce model bias, because the experienced observations under different LAMs are always diverse. Then, the observation diversity can be further increased by employing LDMs to explore. Finally, we implement experiments on classic control tasks (CCTs) and Atari games, which demonstrates the efficiency of our proposed framework.

Controller Optimization for Multirate Systems Based on Reinforcement Learning

The goal of this paper is to design a model-free optimal controller for the multirate system based on reinforcement learning. Sampled-data control systems are widely used in the industrial production process and multirate sampling has attracted much attention in the study of the sampled-data control theory. In this paper, we assume the sampling periods for state variables are different from periods for system inputs. Under this condition, we can obtain an equivalent discrete-time system using the lifting technique. Then, we provide an algorithm to solve the linear quadratic regulator (LQR) control problem of multirate systems with the utilization of matrix substitutions. Based on a reinforcement learning method, we use online policy iteration and off-policy algorithms to optimize the controller for multirate systems. By using the least squares method, we convert the off-policy algorithm into a model-free reinforcement learning algorithm, which only requires the input and output data of the system. Finally, we use an example to illustrate the applicability and efficiency of the model-free algorithm above mentioned.

Theory-Based Causal Transfer:Integrating Instance-Level Induction and Abstract-Level Structure Learning

Learning transferable knowledge across similar but different settings is a fundamental component of generalized intelligence. In this paper, we approach the transfer learning challenge from a causal theory perspective. Our agent is endowed with two basic yet general theories for transfer learning: (i) a task shares a common abstract structure that is invariant across domains, and (ii) the behavior of specific features of the environment remain constant across domains. We adopt a Bayesian perspective of causal theory induction and use these theories to transfer knowledge between environments. Given these general theories, the goal is to train an agent by interactively exploring the problem space to (i) discover, form, and transfer useful abstract and structural knowledge, and (ii) induce useful knowledge from the instance-level attributes observed in the environment. A hierarchy of Bayesian structures is used to model abstract-level structural causal knowledge, and an instance-level associative learning scheme learns which specific objects can be used to induce state changes through interaction. This model-learning scheme is then integrated with a model-based planner to achieve a task in the OpenLock environment, a virtual “escape room” with a complex hierarchy that requires agents to reason about an abstract, generalized causal structure. We compare performances against a set of predominate model-free reinforcement learning (RL) algorithms. RL agents showed poor ability transferring learned knowledge across different trials. Whereas the proposed model revealed similar performance trends as human learners, and more importantly, demonstrated transfer behavior across trials and learning situations.1

Safety Augmented Value Estimation From Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks

Reinforcement learning (RL) for robotics is challenging due to the difficulty in hand-engineering a dense cost function, which can lead to unintended behavior, and dynamical uncertainty, which makes exploration and constraint satisfaction challenging. We address these issues with a new model-based reinforcement learning algorithm, Safety Augmented Value Estimation from Demonstrations (SAVED), which uses supervision that only identifies task completion and a modest set of suboptimal demonstrations to constrain exploration and learn efficiently while handling complex constraints. We then compare SAVED with 3 state-of-the-art model-based and model-free RL algorithms on 6 standard simulation benchmarks involving navigation and manipulation and a physical knot-tying task on the da Vinci surgical robot. Results suggest that SAVED outperforms prior methods in terms of success rate, constraint satisfaction, and sample efficiency, making it feasible to safely learn a control policy directly on a real robot in less than an hour. For tasks on the robot, baselines succeed less than 5% of the time while SAVED has a success rate of over 75% in the first 50 training iterations. Code and supplementary material is available at https://tinyurl.com/saved-rl.

Automated clash resolution for reinforcement steel design in concrete frames via Q-learning and Building Information Modeling

The design of reinforcing steel bars (rebars) is critical to reinforced concrete (RC) structures. Generally, a good number of rebars are required by a design code, particularly at member connections. As such, rebar clashes (i.e., collisions and congestions) would be inevitable. It would be impractical, labor-intensive, and error-prone to avoid all possible clashes manually or even using standard design software. The building information modeling (BIM) technology has been utilized by the present architecture, engineering, and construction (ACE) industry for clash-free rebar designs. However, most existing BIM-based approaches offer the clash resolution strategy for moving components with an optimization algorithm, and are only applicable to the RC structures with regular shapes. In particular, the optimized path of rebars cannot be adjusted to avoid the obstacles, thus limiting the practical applications. Furthermore, most existing studies lack the learning from design code and constructibility constraints to realize automatic and intelligent arrangement and adjustment of rebars for avoiding the obstacles encountered in complex RC joints and frame structures. Considering these shortcomings, the authors have recently proposed an immediate reward-based multi-agent reinforcement learning (MARL) system with BIM, towards automatic clash-free rebar designs of RC joints without clashes. However, as the immediate reward is required in the MARL system for guiding the learning of a rebar design, it will not succeed in clash-free rebar designs of complex RC structures where immediate reward is often unavailable. In this study, this study further extends the previous work with Q-learning (a model-free reinforcement learning algorithm) for more realistic path planning considering both immediate and delayed rewards in clash-free rebar designs for real-world RC structures. In particular, the rebar design problem is treated as a path-planning problem of multi-agent system, where each rebar is deemed as an intelligence reinforcement learning agent. Next, by employing the Q-learning as the reinforcement learning engine, the particular form of state, action, and immediate and delayed rewards for the reinforcement MARL for automatic rebar designs considering more actual constructible constraints and design codes can be developed. Comprehensive experiments on three typical beam-column joints and a two-story RC building frame were conducted to evaluate the efficiency of the proposed method. The study results of paths of rebar designs, success rates, and average time confirm that the proposed framework with MARL and BIM is effective and efficient.

Dopamine transients do not act as model-free prediction errors during associative learning

Dopamine neurons are proposed to signal the reward prediction error in model-free reinforcement learning algorithms. This term represents the unpredicted or ‘excess’ value of the rewarding event, value that is then added to the intrinsic value of any antecedent cues, contexts or events. To support this proposal, proponents cite evidence that artificially-induced dopamine transients cause lasting changes in behavior. Yet these studies do not generally assess learning under conditions where an endogenous prediction error would occur. Here, to address this, we conducted three experiments where we optogenetically activated dopamine neurons while rats were learning associative relationships, both with and without reward. In each experiment, the antecedent cues failed to acquire value and instead entered into associations with the later events, whether valueless cues or valued rewards. These results show that in learning situations appropriate for the appearance of a prediction error, dopamine transients support associative, rather than model-free, learning.

An Enhanced Model-Free Reinforcement Learning Algorithm to Solve Nash Equilibrium for Multi-Agent Cooperative Game Systems

Solving the Nash equilibrium is important for multi-agent game systems, and the speed of reaching Nash equilibrium is critical for the agent to quickly make real-time decisions. A typical scheme is the model-free reinforcement learning algorithm based on policy iteration, which is slow because each iteration will be calculated from the start state to the end state. In this paper, we propose a faster scheme based on value iteration, using Q-function in an online manner to solve the Nash equilibrium of the system. Since the calculation is based on the value from the last iteration, the convergence speed of the proposed scheme is much faster than the policy iteration. The rationality and convergence of this scheme are analyzed and proved theoretically. An actor-critic network structure is used to implement this scheme through simulation. The simulation results show that the convergence speed of our proposed scheme is about 10 times faster than that of the policy iteration algorithm.