Model-based Reinforcement Learning Method Research Articles

Optimization of pressure management strategies for geological CO2 storage using surrogate model-based reinforcement learning

Injecting greenhouse gas (e.g. CO2) into deep underground reservoirs for permanent storage can inadvertently lead to fault reactivation, caprock fracturing and greenhouse gas leakage when the injection-induced stress exceeds the critical threshold. It is essential to monitor the evolution of pressure and the movement of the CO2 plume closely during the injection to allow for timely remediation actions or rapid adjustments to the storage design. Extraction of pre-existing fluids at various stages of the injection process, referred as pressure management, can mitigate associated risks and lessen environmental impact. However, identifying optimal pressure management strategies typically requires thousands of simulations, making the process computationally prohibitive. This paper introduces a novel surrogate model-based reinforcement learning method for devising optimal pressure management strategies for geological CO2 sequestration efficiently. Our approach comprises of two steps. The first step involves developing a surrogate model using the embed to control method, which employs an encoder-transition-decoder structure to learn dynamics in a latent or reduced space. The second step, leveraging this proxy model, utilizes reinforcement learning to find an optimal strategy that maximizes economic benefits while satisfying various control constraints. The reinforcement learning agent receives the latent state representation and immediate reward tailored for CO2 sequestration and choose real-time controls which are subject to predefined engineering constraints in order to maximize the long-term cumulative rewards. To demonstrate its effectiveness, this framework is applied to a compositional simulation model where CO2 is injected into saline aquifer. The results reveal that our surrogate model-based reinforcement learning approach significantly optimizes CO2 sequestration strategies, leading to notable economic gains compared to baseline scenarios.

Online model adaptation in Monte Carlo tree search planning

AbstractWe propose a model-based reinforcement learning method using Monte Carlo Tree Search planning. The approach assumes a black-box approximated model of the environment developed by an expert using any kind of modeling framework and it improves the model as new information from the environment is collected. This is crucial in real-world applications, since having a complete knowledge of complex environments is impractical. The expert’s model is first translated into a neural network and then it is updated periodically using data, i.e., state-action-next-state triplets, collected from the real environment. We propose three different methods to integrate data acquired from the environment with prior knowledge provided by the expert and we evaluate our approach on a domain concerning air quality and thermal comfort control in smart buildings. We compare the three proposed versions with standard Monte Carlo Tree Search planning using the expert’s model (without adaptation), Proximal Policy Optimization (a popular model-free DRL approach) and Stochastic Lower Bounds Optimization (a popular model-based DRL approach). Results show that our approach achieves the best results, outperforming all analyzed competitors.

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

Model-based reinforcement learning can effectively improve the sample efficiency of reinforcement learning, but the environment model in this method has errors. The model errors can mislead the policy optimization, leading to suboptimal policy. To improve the generalization ability of the environment model, existing methods often use ensemble models or Bayesian models to build the environment model. However, these methods are computationally intensive and complex to update. Since the generated model can describe the stochastic nature of the environment, this paper proposes a model-based reinforcement learning method based on a conditional variational auto-encoder. In this paper, we use a conditional variational auto-encoder to learn task-related representations and apply the generative model to predict environmental changes. Considering the problem of multi-step error accumulation, model adaptation is utilized to minimize the difference between simulated and real data distributions. Furthermore, the experiments verified that the proposed method can learn task-relevant representations and accelerate policy learning.

Unified Human–Robot–Environment Interaction Control in Contact-Rich Collaborative Manipulation Tasks via Model-Based Reinforcement Learning

Many safety-critical or performance-demanding systems are human-in-the-loop, i.e., the robot interacts with human being and environment, in which human-in-the-loop control becomes a key research topic. In this paper, a unified optimal interaction control in joint space is presented for multi-point human-robot-environment interaction (HREI) problems, which are very common in human-robot collaborative manipulation tasks. Specifically, model-based reinforcement learning method is leveraged for obtaining optimal interaction control. For multi-point human-robot-environment interaction, the interaction forces exerted on each link are isolated and estimated via the backward generalized momentum observer method. In human-robot-environment interaction problems, the environmental as well as the human arm's dynamics parameters are usually unknown, stochastic, and time-varying. To obviate the dependence on these parameters, the GMM/GMR method is employed to learn the unknown external dynamics. It is noteworthy the interaction forces are considered as system states, the time derivatives of which are computed based on the GMM/GMR learning results via chain-rule. Then the iterative linear quadratic Gaussian with learned external dynamics (ILQG-LED) method is utilized to realize optimal multi-point human-robot-environment interaction control. The validity of the proposed method is verified through experimental studies.

Dyna-Validator: A Model-based Reinforcement Learning Method with Validated Simulated Experiences

Dyna is a planning paradigm that naturally weaves learning and planning together through environmental models. Dyna-style reinforcement learning improves the sample efficiency using the simulation experience generated by the environment model to update the value function. However, the existing Dyna-style planning methods are usually based on tabular methods, only suitable for tasks with low-dimensional and small-scale space. In addition, the quality of the simulation experience generated by the existing methods cannot be guaranteed, which significantly limits its application in tasks such as continuous control of high-dimensional robots and autonomous driving. To this end, we propose a model-based approach that controls planning through a validator. The validator filters high-quality experiences for policy learning and decides whether to stop planning. To deal with the exploration and exploitation dilemma in reinforcement learning, a combination of ϵ-greedy strategy and simulated annealing (SA) cooling schedule control is designed as an action selection strategy. The excellent performance of the proposed method is demonstrated in a set of classical Atari games. Experimental results show that learning dynamic models in some games can improve sample efficiency. This benefit is maximized by choosing the proper planning steps. In the optimization planning phase, our method maintains a smaller gap with the current state-of-the-art model-based reinforcement learning (MuZero). In order to achieve a good compromise between model accuracy and optimal programming step size, it is necessary to control the programming reasonably. The practical application of this method in a physical robot system helps reduce the influence of an imprecise depth prediction model on the task. Without human supervision, it is easier to collect training data and learn complex skills (such as grabbing and carrying items) while being more effective at scaling tasks that have never been seen before.

Robust safe reinforcement learning control of unknown continuous-time nonlinear systems with state constraints and disturbances

Industrial processes must operate safely and optimally in practice, which typically entails solving a constrained optimal control (COC) problem. Finding a control policy for such a problem, however, is challenging especially when the system dynamics are unknown. In this paper, a novel safe reinforcement learning (RL) algorithm is proposed to deal with the COC problem for the continuous-time nonlinear system with unknown dynamics and disturbances. The presented method can ensure that the system operates within a predefined safe region in the presence of system model uncertainties and external disturbances, which goes beyond the results of typical RL methods. To handle the system state constraints, the problem is first transformed into an unconstrained one via the proposed data-driven slack function approach. Then an improved model-based RL method is devised to learn a near-optimal and safe control policy in real-time. Finally, a robust compensator and the composite neural-network updating rule are designed to eliminate the influence of disturbances and uncertainties. Theoretical analysis based on the Lyapunov approach is conducted to prove the closed-loop system stability and the algorithm convergence, which further guarantees the satisfaction of safety constraints. The effectiveness of the proposed RL algorithm is verified through simulations.

Model-Based Reinforcement Learning Method for Microgrid Optimization Scheduling

Due to the uncertainty and randomness of clean energy, microgrid operation is often prone to instability, which requires the implementation of a robust and adaptive optimization scheduling method. In this paper, a model-based reinforcement learning algorithm is applied to the optimal scheduling problem of microgrids. During the training process, the current learned networks are used to assist Monte Carlo Tree Search (MCTS) in completing game history accumulation, and updating the learning network parameters to obtain optimal microgrid scheduling strategies and a simulated environmental dynamics model. We establish a microgrid environment simulator that includes Heating Ventilation Air Conditioning (HVAC) systems, Photovoltaic (PV) systems, and Energy Storage (ES) systems for simulation. The simulation results show that the operation of microgrids in both islanded and connected modes does not affect the training effectiveness of the algorithm. After 200 training steps, the algorithm can avoid the punishment of exceeding the red line of the bus voltage, and after 800 training steps, the training result converges and the loss values of the value and reward network converge to 0, showing good effectiveness. This proves that the algorithm proposed in this paper can be applied to the optimization scheduling problem of microgrids.

Tensor Implementation of Monte-Carlo Tree Search for Model-Based Reinforcement Learning

Monte-Carlo tree search (MCTS) is a widely used heuristic search algorithm. In model-based reinforcement learning, MCTS is often utilized to improve action selection process. However, model-based reinforcement learning methods need to process large number of observations during the training. If MCTS is involved, it is necessary to run one instance of MCTS for each observation in every iteration of training. Therefore, there is a need for efficient method to process multiple instances of MCTS. We propose a MCTS implementation that can process batch of observations in fully parallel fashion on a single GPU using tensor operations. We demonstrate efficiency of the proposed approach on a MuZero reinforcement learning algorithm. Empirical results have shown that our method outperforms other approaches and scale well with increasing number of observations and simulations.

Deep Reinforcement Learning Evolution Algorithm for Dynamic Antenna Control in Multi-Cell Configuration HAPS System

In this paper, we propose a novel Deep Reinforcement Learning Evolution Algorithm (DRLEA) method to control the antenna parameters of the High-Altitude Platform Station (HAPS) mobile to reduce the number of low-throughput users. Considering the random movement of the HAPS caused by the winds, the throughput of the users might decrease. Therefore, we propose a method that can dynamically adjust the antenna parameters based on the throughput of the users in the coverage area to reduce the number of low-throughput users by improving the users’ throughput. Different from other model-based reinforcement learning methods, such as the Deep Q Network (DQN), the proposed method combines the Evolution Algorithm (EA) with Reinforcement Learning (RL) to avoid the sub-optimal solutions in each state. Moreover, we consider non-uniform user distribution scenarios, which are common in the real world, rather than ideal uniform user distribution scenarios. To evaluate the proposed method, we do the simulations under four different real user distribution scenarios and compare the proposed method with the conventional EA and RL methods. The simulation results show that the proposed method effectively reduces the number of low throughput users after the HAPS moves.

Uncertainty-Aware Model-Based Reinforcement Learning: Methodology and Application in Autonomous Driving

To further improve learning efficiency and performance of reinforcement learning (RL), a novel uncertainty-aware model-based RL method is proposed and validated in autonomous driving scenarios in this paper. First, an action-conditioned ensemble model with the capability of uncertainty assessment is established as the environment model. Then, a novel uncertainty-aware model-based RL method is developed based on the adaptive truncation approach, providing virtual interactions between the agent and environment model, and improving RL’s learning efficiency and performance. The proposed method is then implemented in end-to-end autonomous vehicle control tasks, validated and compared with state-of-the-art methods under various driving scenarios. Validation results suggest that the proposed method outperforms the model-free RL approach with respect to learning efficiency, and model-based approach with respect to both efficiency and performance, demonstrating its feasibility and effectiveness.

Linearization of nonlinear frequency modulated continuous wave generation using model-based reinforcement learning.

The prevalence of machine learning (ML) opens up new directions for plenty of scientific fields. The development of optics technologies also benefits from it. However, due to the complex properties of nonlinear and dynamic optical systems, optical system control with ML is still in its infancy. In this manuscript, to demonstrate the feasibility of optical system control using reinforcement learning (RL), i.e., a branch of ML, we solve the linearization problem in the frequency modulated continuous wave (FMCW) generation with the model-based RL method. The experiment results indicate an excellent improvement in the linearity of the generated FMCW, showing a sharp peak in the frequency spectrum. We confirm that the RL method learns the implicit physical characteristics very well and accomplishes the goal of the linear FMCW generation effectively, indicating that the marriage of ML and optics systems could have the potential to open a new era for the development of optical system control.

RL-QN: A Reinforcement Learning Framework for Optimal Control of Queueing Systems

With the rapid advance of information technology, network systems have become increasingly complex and hence the underlying system dynamics are often unknown or difficult to characterize. Finding a good network control policy is of significant importance to achieve desirable network performance (e.g., high throughput or low delay). In this work, we consider using model-based reinforcement learning (RL) to learn the optimal control policy for queueing networks so that the average job delay (or equivalently the average queue backlog) is minimized. Traditional approaches in RL, however, cannot handle the unbounded state spaces of the network control problem. To overcome this difficulty, we propose a new algorithm, called RL for Queueing Networks (RL-QN), which applies model-based RL methods over a finite subset of the state space while applying a known stabilizing policy for the rest of the states. We establish that the average queue backlog under RL-QN with an appropriately constructed subset can be arbitrarily close to the optimal result. We evaluate RL-QN in dynamic server allocation, routing, and switching problems. Simulation results show that RL-QN minimizes the average queue backlog effectively.

Coordination of distributed unmanned surface vehicles via model-based reinforcement learning methods

This article presents a coordination algorithm for organizing a fleet of unmanned surface vehicles (USVs) to search multiple moving object targets in the ocean environment. During the fleet maneuver, USVs can exchange local sensing information through a wireless communication network. Based on both received and self-perceived information, a USV constructs a grid confidence map, which reflects how well the USV fleet perceives on every region of the search area. Then, the USV coordination is modeled as a reinforcement learning (RL) problem, where reward functions are defined based on the information obtained from the grid confidence map. Therefore, USVs are encouraged to explore new regions and prevented visiting the already searched areas. Search routes of USVs are calculated via a policy-iteration based path planning algorithm, while inter-vehicle collisions are avoided by applying policy constraints. Real-world experiments were conducted in the ocean environment to evaluate the validity of the proposed method. Compared to the conventional formation control strategy and the uncoordinated algorithm, experiment results show that the proposed method is more intelligent and efficient for searching object targets.

Latent Representation Prediction Networks

Modern model-based reinforcement learning methods for high-dimensional inputs often incorporate an unsupervised learning step for dimensionality reduction. The training objective of these unsupervised learning methods often leverages only static inputs such as reconstructing observations. These representations are combined with predictor functions for simulating rollouts to navigate the environment. We advance this idea by taking advantage of the fact that we navigate dynamic environments with visual stimulus and create a representation that is specifically designed with control and actions in mind. We propose to learn a feature map that is maximally predictable for a predictor function. This results in representations that are well suited for the task of planning, where the predictor is used as a forward model. To this end, we introduce a new way of learning this representation along with the prediction function, a system we dub Latent Representation Prediction Network (LARP). The prediction function is used as a forward model for a search on a graph in a viewpoint-matching task, and the representation learned to maximize predictability is found to outperform other representations. The sample efficiency and overall performance of our approach are shown to rival standard reinforcement learning methods, and our learned representation transfers successfully to unseen environments.

Safe Model-Based Reinforcement Learning for Systems With Parametric Uncertainties.

Reinforcement learning has been established over the past decade as an effective tool to find optimal control policies for dynamical systems, with recent focus on approaches that guarantee safety during the learning and/or execution phases. In general, safety guarantees are critical in reinforcement learning when the system is safety-critical and/or task restarts are not practically feasible. In optimal control theory, safety requirements are often expressed in terms of state and/or control constraints. In recent years, reinforcement learning approaches that rely on persistent excitation have been combined with a barrier transformation to learn the optimal control policies under state constraints. To soften the excitation requirements, model-based reinforcement learning methods that rely on exact model knowledge have also been integrated with the barrier transformation framework. The objective of this paper is to develop safe reinforcement learning method for deterministic nonlinear systems, with parametric uncertainties in the model, to learn approximate constrained optimal policies without relying on stringent excitation conditions. To that end, a model-based reinforcement learning technique that utilizes a novel filtered concurrent learning method, along with a barrier transformation, is developed in this paper to realize simultaneous learning of unknown model parameters and approximate optimal state-constrained control policies for safety-critical systems.

Deep reinforcement learning with credit assignment for combinatorial optimization

Recent advances in Deep Reinforcement Learning (DRL) demonstrates the potential for solving Combinatorial Optimization (CO) problems. DRL shows advantages over traditional methods both on scalability and computation efficiency. However, the DRL problems transformed from CO problems usually have a huge state space, and the main challenge of solving them has changed from high computation complexity to high sample complexity.Credit assignment determines the contribution of each internal decision to the final success or failure, and it has been shown to be effective in reducing the sample complexity of the training process. In this paper, we resort to a model-based reinforcement learning method to assign credits for model-free DRL methods. Since heuristic methods plays an important role on state-of-the-art solutions for CO problems, we propose using a model to represent those heuristic knowledge and derive the credit assignment from the model. This model-based credit assignment can facilitate the model-free DRL to perform a more effective exploration, and the data collected by the model-free DRL refines the model continuously as the training progresses. Extensive experiments on various CO problems with different settings show that our framework outperforms previous state-of-the-art methods on performance and training efficiency.

A model-based reinforcement learning method based on conditional generative adversarial networks

Deep reinforcement learning (DRL) integrates the advantages of the perception of deep learning and enables reinforcement learning scale to problems with high dimensional state and action spaces that were previously intractable. The success of DRL primarily relies on the high level representation ability of deep learning. To obtain a good performed representation model, excessive training samples and training time are necessary. However, collecting a large number of samples in real world is extremely expensive and time consuming. To mitigate the sample inefficiency problem, we propose a novel model-based reinforcement learning method by combining conditional generative adversarial networks (CGAN-MbRL) with the state-of-the-art policy learning method. The proposed CGAN-MbRL can directly deal with the high dimensional state, and mitigate the problem of sample inefficiency to some extent. Finally, the effectiveness of the proposed method is demonstrated through the illustrative data and the RL benchmark.

Risk-Aware Model-Based Control.

Model-Based Reinforcement Learning (MBRL) algorithms have been shown to have an advantage on data-efficiency, but often overshadowed by state-of-the-art model-free methods in performance, especially when facing high-dimensional and complex problems. In this work, a novel MBRL method is proposed, called Risk-Aware Model-Based Control (RAMCO). It combines uncertainty-aware deep dynamics models and the risk assessment technique Conditional Value at Risk (CVaR). This mechanism is appropriate for real-world application since it takes epistemic risk into consideration. In addition, we use a model-free solver to produce warm-up training data, and this setting improves the performance in low-dimensional environments and covers the shortage of MBRL’s nature in the high-dimensional scenarios. In comparison with other state-of-the-art reinforcement learning algorithms, we show that it produces superior results on a walking robot model. We also evaluate the method with an Eidos environment, which is a novel experimental method with multi-dimensional randomly initialized deep neural networks to measure the performance of any reinforcement learning algorithm, and the advantages of RAMCO are highlighted.

Deep reinforcement learning with a particle dynamics environment applied to emergency evacuation of a room with obstacles

Efficient emergency evacuation is crucial for survival. A very successful model for simulating emergency evacuation is the social-force model. At the heart of the model is the self-driven force that is applied to an agent and is directed towards the exit. However, it is not clear if the application of this force results in optimal evacuation, especially in complex environments with obstacles. In this paper, we develop a deep reinforcement learning algorithm in association with the social force model to train agents to find the fastest evacuation path. During training, we penalize every step of an agent in the room and give zero reward at the exit. We adopt the Dyna-Q learning approach, which incorporates both the model-free Q-learning algorithm and the model-based reinforcement learning method, to update a deep neural network used to approximate the action value functions. We first show that in the case of a room without obstacles the resulting self-driven force points directly towards the exit as in the social force model. To quantitatively validate our method, we compare the total time elapsed when agents escape a room with one door and without obstacles employing the Dyna-Q model with the result obtained using the social-force model. We find that the median exit time intervals calculated using the two methods are not significantly different. We confirm that the proposed method obtains trajectories that minimize the travel time by comparing our results to results generated by geodesics-based adaptive pedestrian dynamics. Then, we investigate evacuation of a room with one obstacle and one exit. We show that our method produces similar results with the social force model when the obstacle is convex. However, in the case of concave obstacles, which sometimes can act as traps for agents governed purely by the social force model and prohibit complete room evacuation, our approach is clearly advantageous since it derives a policy that results in object avoidance and complete room evacuation without additional assumptions. We also study evacuation of a room with multiple exits. We show that agents are able to evacuate efficiently from the nearest exit through a shared network trained for a single agent. Finally, we test the robustness of the Dyna-Q learning approach in a complex environment with multiple exits and obstacles. Overall, we show that our model, based on the Dyna-Q reinforcement learning approach, can efficiently handle modeling of emergency evacuation in complex environments with multiple room exits and obstacles where it is difficult to obtain an intuitive rule for fast evacuation.

Reinforcement Learning Based on Equivalent Consumption Minimization Strategy for Optimal Control of Hybrid Electric Vehicles

Hybrid electric vehicles, operated by engines and motors, require an energy management strategy to achieve competitive fuel economy performance. The equivalent consumption minimization strategy is a well-known algorithm that can be employed for the energy management of hybrid electric vehicles, based on the concept of the equivalent cost of fossil fuels and electric battery energy. However, in the equivalent consumption minimization strategy approach, a parameter called the equivalent factor should be determined to obtain the optimal control policy. In this study, reinforcement learning based approaches are proposed to determine the equivalent factor. First, we show that the equivalent factor can be indirectly extracted from the reinforcement learning results, using the control action from reinforcement learning for the specific driving cycle. In addition, a novel approach that combines reinforcement learning and the equivalent consumption minimization strategy is proposed, where the equivalent factor is determined based on the interaction between the reinforcement learning agent and driving environment, while the control input is decided by the equivalent consumption minimization strategy based on the determined equivalent factor. A model-based reinforcement learning method is used, and the proposed method is validated for vehicle simulation using a parallel hybrid electric vehicle. The simulation results show that the proposed method can achieve a near-optimal solution, which is close to the global solution obtained with the dynamic programming approach (96.7% compared to dynamic programming result in average), and improved performance of 4.3% in average compared with the existing adaptive equivalent consumption minimization strategy.