Model-based Reinforcement Learning Research Articles

Curiosity model policy optimization for robotic manipulator tracking control with input saturation in uncertain environment

In uncertain environments with robot input saturation, both model-based reinforcement learning (MBRL) and traditional controllers struggle to perform control tasks optimally. In this study, an algorithmic framework of Curiosity Model Policy Optimization (CMPO) is proposed by combining curiosity and model-based approach, where tracking errors are reduced via training agents on control gains for traditional model-free controllers. To begin with, a metric for judging positive and negative curiosity is proposed. Constrained optimization is employed to update the curiosity ratio, which improves the efficiency of agent training. Next, the novelty distance buffer ratio is defined to reduce bias between the environment and the model. Finally, CMPO is simulated with traditional controllers and baseline MBRL algorithms in the robotic environment designed with non-linear rewards. The experimental results illustrate that the algorithm achieves superior tracking performance and generalization capabilities.

Model-Based Reinforcement Learning With Isolated Imaginations.

World models learn the consequences of actions in vision-based interactive systems. However, in practical scenarios like autonomous driving, noncontrollable dynamics that are independent or sparsely dependent on action signals often exist, making it challenging to learn effective world models. To address this issue, we propose Iso-Dream++, a model-based reinforcement learning approach that has two main contributions. First, we optimize the inverse dynamics to encourage the world model to isolate controllable state transitions from the mixed spatiotemporal variations of the environment. Second, we perform policy optimization based on the decoupled latent imaginations, where we roll out noncontrollable states into the future and adaptively associate them with the current controllable state. This enables long-horizon visuomotor control tasks to benefit from isolating mixed dynamics sources in the wild, such as self-driving cars that can anticipate the movement of other vehicles, thereby avoiding potential risks. On top of our previous work (Pan et al. 2022), we further consider the sparse dependencies between controllable and noncontrollable states, address the training collapse problem of state decoupling, and validate our approach in transfer learning setups. Our empirical study demonstrates that Iso-Dream++ outperforms existing reinforcement learning models significantly on CARLA and DeepMind Control.

Data-driven adaptive consensus control for heterogeneous nonlinear Multi-Agent Systems using online reinforcement learning

In this paper, a distributed control algorithm based on a data-driven approach is developed to solve the consensus control of heterogeneous nonlinear Multi-Agent Systems (MAS). The consensus obtained from the solution of the Hamilton–Jacobi–Bellman (HJB) equation is challenging for unknown nonlinear systems. To address this issue, improved online reinforcement learning (RL) is employed to generate an approximate solution for each agent to achieve consensus. Unlike model-based RL and traditional algorithms, this method leverages I/O data to guide the learning of policies without any prior knowledge of agent dynamics. Furthermore, the adaptability of algorithm to heterogeneous nonlinear agents is enhanced by implementing online updates to the control strategy and dynamic linearization (DL). The convergence analysis of the algorithm is provided, along with the impact of the learning rateparameters on the consensus of MAS. By comparing with other data-driven methods, simulations are conducted to verify the stability and adaptability of the proposed algorithm.

Reinforcement learning in building controls: A comparative study of algorithms considering model availability and policy representation

Reinforcement Learning (RL) has presented considerable potential as an advanced control technique in building controls to enable buildings operating more energy-efficient. As various types of RL algorithms have been studied on their performance of building controls, benchmarking these algorithms across the entire spectrum of features is essential to provide an overview and deepen the understanding of RL applications. Therefore, this study aims to compare and analyze the effectiveness of various RL algorithms, encompassing the entire RL categories featured by value-based, policy gradient, actor-critic and model-based RL considering model availability and policy representation. To provide a comprehensive analysis, in addition to the control performance quantified by the cumulative rewards based on the cost function of RL, data demand and robustness of hyperparameter tuning were investigated. The open-source Gym-Eplus framework was selected as the virtual environment to train and test different RL agents. The results showed that both model-free and model-based RL agents outperformed the baseline rule-based control in terms of energy consumption and thermal comfort, and RL agents were capable of evaluating both short-term and long-term rewards to achieve adaptive control optimization continuously along with the online control process. Model-based RL agent improved the data sampling efficiency but presented a relatively sacrificed control performance during the tested summer days.

M 3 Rec: A Context-aware Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation

Reinforcement learning (RL) has shown great promise in optimizing long-term user interest in recommender systems. However, existing RL-based recommendation methods need a large number of interactions for each user to learn the recommendation policy. The challenge becomes more critical when recommending to new users who have a limited number of interactions. To that end, in this paper, we address the cold-start challenge in the RL-based recommender systems by proposing a novel context-aware offline meta-level model-based reinforcement learning approach for user adaptation. Our proposed approach learns to infer each user's preference with a user context variable that enables recommendation systems to better adapt to new users with limited contextual information. To improve adaptation efficiency, our approach learns to recover the user choice function and reward from limited contextual information through an inverse reinforcement learning method, which is used to assist the training of a meta-level recommendation agent. To avoid the need for online interaction, the proposed method is trained using historically collected offline data. Moreover, to tackle the challenge of offline policy training, we introduce a mutual information constraint between the user model and recommendation agent. Evaluation results show the superiority of our developed offline policy learning method when adapting to new users with limited contextual information. In addition, we provide a theoretical analysis of the recommendation performance bound.

An Overview of the Free Energy Principle and Related Research.

The free energy principle and its corollary, the active inference framework, serve as theoretical foundations in the domain of neuroscience, explaining the genesis of intelligent behavior. This principle states that the processes of perception, learning, and decision making-within an agent-are all driven by the objective of "minimizing free energy," evincing the following behaviors: learning and employing a generative model of the environment to interpret observations, thereby achieving perception, and selecting actions to maintain a stable preferred state and minimize the uncertainty about the environment, thereby achieving decision making. This fundamental principle can be used to explain how the brain processes perceptual information, learns about the environment, and selects actions. Two pivotal tenets are that the agent employs a generative model for perception and planning and that interaction with the world (and other agents) enhances the performance of the generative model and augments perception. With the evolution of control theory and deep learning tools, agents based on the FEP have been instantiated in various ways across different domains, guiding the design of a multitude of generative models and decision-making algorithms. This letter first introduces the basic concepts of the FEP, followed by its historical development and connections with other theories of intelligence, and then delves into the specific application of the FEP to perception and decision making, encompassing both low-dimensional simple situations and high-dimensional complex situations. It compares the FEP with model-based reinforcement learning to show that the FEP provides a better objective function. We illustrate this using numerical studies of Dreamer3 by adding expected information gain into the standard objective function. In a complementary fashion, existing reinforcement learning, and deep learning algorithms can also help implement the FEP-based agents. Finally, we discuss the various capabilities that agents need to possess in complex environments and state that the FEP can aid agents in acquiring these capabilities.

DynaGlaucoDetect: Leveraging Dyna-Q learning for glaucoma detection

Glaucoma is an eye disease that requires early detection and proper diagnosis for timely intervention and treatment which can help slow down further progression and to manage intraocular pressure. This paper aims to address the problem by proposing a novel approach that combines a model-based Reinforcement Learning (RL) approach, called DynaGlaucoDetect, with ocular gaze data. By leveraging the RL algorithms to simulate and predict the dynamics of glaucoma, a model-based approach can improve the accuracy and efficiency of glaucoma detection by enabling better preservation of visual health. The RL agent is trained using real experiences and synthetic experiences which are generated using the model-based algorithm Dyna-Q. Two different Q-table generation methods have been discussed: the Direct Synthesis Method (DSM) and the Indirect Synthesis Method (IdSM). The presence of glaucoma has been detected by comparing the reward score a patient obtains with the threshold values obtained through the performed experimentation. The scores obtained using DSM and IdSM have been compared to understand the learning of the agent in both cases. Finally, hyperparameter tuning has been performed to identify the best set of hyperparameters.

Decentralized multi-agent cooperation via adaptive partner modeling

AbstractMulti-agent reinforcement learning encounters a non-stationary challenge, where agents concurrently update their policies, leading to changes in the environment. Existing approaches have tackled this challenge through communication among agents to obtain their partners’ actions, but this introduces computational complexity known as partner sample complexity. An alternative approach is to develop partner models that generate samples instead of direct communication to mitigate this complexity. However, a discrepancy arises between the real policies distribution and the policy of partner models, termed as model bias, which can significantly impact performance when heavily relying on partner models. In order to achieve a trade-off between sample complexity and performance, a novel multi-agent model-based reinforcement learning algorithm called decentralized adaptive partner modeling (DAPM) is proposed, which utilizes fictitious self play (FSP) to construct partner models and update policies. Model bias is addressed by establishing an upper bound to restrict the usage of partner models. Coupled with that, an adaptive rollout approach is introduced, enabling real agents to dynamically communicate with partner models based on their quality, ensuring that agent performance can progressively improve with partner model samples. The effectiveness of DAPM is exhibited in two multi-agent tasks, showing that DAPM outperforms existing model-free algorithms in terms of partner sample complexity and training stability. Specifically, DAPM requires 28.5% fewer communications compared to the best baseline and exhibits reduced fluctuations in the learning curve, indicating superior performance.

A variable speed limit control approach for freeway tunnels based on the model-based reinforcement learning framework with safety perception

To improve the traffic safety and efficiency of freeway tunnels, this study proposes a novel variable speed limit (VSL) control strategy based on the model-based reinforcement learning framework (MBRL) with safety perception. The MBRL framework is designed by developing a multi-lane cell transmission model for freeway tunnels as an environment model, which is built so that agents can interact with the environment model while interacting with the real environment to improve the sampling efficiency of reinforcement learning. Based on a real-time crash risk prediction model for freeway tunnels that uses random deep and cross networks, the safety perception function inside the MBRL framework is developed. The reinforcement learning components fully account for most current tunnels' application conditions, and the VSL control agent is trained using a deep dyna-Q method. The control process uses a safety trigger mechanism to reduce the likelihood of crashes caused by frequent changes in speed. The efficacy of the proposed VSL strategies is validated through simulation experiments. The results show that the proposed VSL strategies significantly increase traffic safety performance by between 16.00% and 20.00% and traffic efficiency by between 3.00% and 6.50% compared to a fixed speed limit approach. Notably, the proposed strategies outperform traditional VSL strategy based on the traffic flow prediction model in terms of traffic safety and efficiency improvement, and they also outperform the VSL strategy based on model-free reinforcement learning framework when sampling efficiency is considered together. In addition, the proposed strategies with safety triggers are safer than those without safety triggers. These findings demonstrate the potential for MBRL-based VSL strategies to improve traffic safety and efficiency within freeway tunnels.

Enhancing military medical evacuation dispatching with armed escort management: comparing model-based reinforcement learning approaches

The military medical evacuation (MEDEVAC) dispatching problem involves determining optimal policies for evacuating combat casualties to maximize patient survivability during military operations. This study explores a variation of the MEDEVAC dispatching problem, focusing on controlling armed escorts using a Markov decision process (MDP) model and model-based reinforcement learning (RL) approaches. A discounted, continuous-time MDP model over an infinite horizon is developed to maximize the expected total discounted reward of the system. Two model-based RL solution approaches are proposed: one utilizing semi-gradient descent Q-learning and another employing semi-gradient descent SARSA. A computational example, set in western and central Africa during contingency operations, assesses the performance of the RL-generated policies against the myopic policy, which military medical planners currently employ. Solution quality is derived from expected response time, a crucial determinant of life-saving potential in MEDEVAC operations. The research also explores sensitivity analysis and excursion scenarios to evaluate the RL-generated policies further. By explicitly controlling armed escort assets, dispatching authorities can better manage the location and allocation of these resources throughout combat operations. The findings of this study have the potential to inform military medical planning, operations, and tactics, ultimately leading to improved MEDEVAC system performance and higher patient survivability rates.

Ultra-fast tuning of neural network controllers with application in path tracking of autonomous vehicle

Neural network (NN) controllers have shown great potential in solving complex control or decision-making tasks. However, most of the NN controllers either rely on the availability of large datasets or require dense interactions with the environment, which hinders their application in real systems. In this paper, we introduce a model-based reinforcement learning (MBRL) algorithm, aimed at realizing ultra-fast tuning of deep NN controller from a small sample set of real-world data. The algorithm uses Gaussian processes (GPs) to model the unknown dynamics of real system and updates controller parameters through stochastic gradient descent. By using particle-based method for long-term predictions, the algorithm can easily incorporate online state estimators and filters into controller learning, which is conductive to learning from systems with partially measurable states and stochastic control delay. We apply the algorithm to calibrate a deep NN controller for the path tracking of a full-size autonomous vehicle (AV). Simulation and field test results show that the deep NN controller can be well calibrated after only one interaction with the environment and can achieve similar tracking performance to optimization-based methods such as nonlinear model prediction control (NMPC) in various test scenarios by combining with a feed-forward pure pursuit (PP) controller.

Probabilistic Reach-Avoid for Bayesian Neural Networks

Model-based reinforcement learning seeks to simultaneously learn the dynamics of an unknown stochastic environment and synthesise an optimal policy for acting in it. Ensuring the safety and robustness of sequential decisions made through a policy in such an environment is a key challenge for policies intended for safety-critical scenarios. In this work, we investigate two complementary problems: first, computing reach-avoid probabilities for iterative predictions made with dynamical models, with dynamics described by Bayesian neural network (BNN); second, synthesising control policies that are optimal with respect to a given reach-avoid specification (reaching a “target” state, while avoiding a set of “unsafe” states) and a learned BNN model. Our solution leverages interval propagation and backward recursion techniques to compute lower bounds for the probability that a policy's sequence of actions leads to satisfying the reach-avoid specification. Such computed lower bounds provide safety certification for the given policy and BNN model. We then introduce control synthesis algorithms to derive policies maximizing said lower bounds on the safety probability. We demonstrate the effectiveness of our method on a series of control benchmarks characterized by learned BNN dynamics models. On our most challenging benchmark, compared to purely data-driven policies the optimal synthesis algorithm is able to provide more than a four-fold increase in the number of certifiable states and more than a three-fold increase in the average guaranteed reach-avoid probability.

TEAMSTER: Model-Based Reinforcement Learning for Ad Hoc Teamwork (Abstract Reprint)

This paper investigates the use of model-based reinforcement learning in the context of ad hoc teamwork. We introduce a novel approach, named TEAMSTER, where we propose learning both the environment's model and the model of the teammates' behavior separately. Compared to the state-of-the-art PLASTIC algorithms, our results in four different domains from the multi-agent systems literature show that TEAMSTER is more flexible than the PLASTIC-Model, by learning the environment's model instead of assuming a perfect hand-coded model, and more robust/efficient than PLASTIC-Policy, by being able to continuously adapt to newly encountered teams, without implicitly learning a new environment model from scratch.

OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning

Model-based offline reinforcement learning (RL) algorithms have emerged as a promising paradigm for offline RL. These algorithms usually learn a dynamics model from a static dataset of transitions, use the model to generate synthetic trajectories, and perform conservative policy optimization within these trajectories. However, our observations indicate that policy optimization methods used in these model-based offline RL algorithms are not effective at exploring the learned model and induce biased exploration, which ultimately impairs the performance of the algorithm. To address this issue, we propose Offline Conservative ExplorAtioN (OCEAN), a novel rollout approach to model-based offline RL. In our method, we incorporate additional exploration techniques and introduce three conservative constraints based on uncertainty estimation to mitigate the potential impact of significant dynamic errors resulting from exploratory transitions. Our work is a plug-in method and can be combined with classical model-based RL algorithms, such as MOPO, COMBO, and RAMBO. Experiment results of our method on the D4RL MuJoCo benchmark show that OCEAN significantly improves the performance of existing algorithms.

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

Model-based reinforcement learning can effectively improve the sample efficiency of reinforcement learning, but the environment model in this method has errors. The model errors can mislead the policy optimization, leading to suboptimal policy. To improve the generalization ability of the environment model, existing methods often use ensemble models or Bayesian models to build the environment model. However, these methods are computationally intensive and complex to update. Since the generated model can describe the stochastic nature of the environment, this paper proposes a model-based reinforcement learning method based on a conditional variational auto-encoder. In this paper, we use a conditional variational auto-encoder to learn task-related representations and apply the generative model to predict environmental changes. Considering the problem of multi-step error accumulation, model adaptation is utilized to minimize the difference between simulated and real data distributions. Furthermore, the experiments verified that the proposed method can learn task-relevant representations and accelerate policy learning.

A probabilistic successor representation for context-dependent learning.

Two of the main impediments to learning complex tasks are that relationships between different stimuli, including rewards, can be uncertain and context-dependent. Reinforcement learning (RL) provides a framework for learning, by predicting total future reward directly (model-free RL), or via predictions of future states (model-based RL). Within this framework, "successor representation" (SR) predicts total future occupancy of all states. A recent theoretical proposal suggests that the hippocampus encodes the SR in order to facilitate prediction of future reward. However, this proposal does not take into account how learning should adapt under uncertainty and switches of context. Here, we introduce a theory of learning SRs using prediction errors which includes optimally balancing uncertainty in new observations versus existing knowledge. We then generalize that approach to a multicontext setting, allowing the model to learn and maintain multiple task-specific SRs and infer which one to use at any moment based on the accuracy of its predictions. Thus, the context used for predictions can be determined by both the contents of the states themselves and the distribution of transitions between them. This probabilistic SR model captures animal behavior in tasks which require contextual memory and generalization, and unifies previous SR theory with hippocampal-dependent contextual decision-making. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Symmetry-Informed Reinforcement Learning and its Application to Low-Level Attitude Control of Quadrotors

Symmetry is ubiquitous in nature, physics, and mathematics. However, a classical symmetry-agnostic Reinforcement Learning (RL) approach cannot guarantee to respect symmetry. Researchers have shown that if the symmetry of a system cannot be respected, the performance of a symmetry-agnostic RL approach can be inhibited. To this end, this paper develops a generally applicable Neural Network (NN) module with symmetry that can enforce the symmetry of a system to be respected. Based on the NN module with symmetry, this paper proposes a symmetry-informed Model-Based RL (MBRL) approach that respects symmetry and improves data efficiency. The symmetry-informed MBRL approach is applied to the attitude control of a quadrotor in simulation to evaluate the effectiveness of the approach. The simulation results show that the data efficiency of the symmetry-informed MBRL approach is much superior to that of a symmetry-agnostic MBRL approach. An NN module with symmetry can respect the symmetry of a quadrotor while a naive NN cannot enforce the symmetry of a quadrotor to be respected.

Multiphase Autonomous Docking via Model-Based and Hierarchical Reinforcement Learning

With the rise of traffic around Earth’s orbit, spacecraft mission designs have placed an unprecedented demand on the capabilities of autonomous systems. In the early 2000s, the state-of-the-art autonomous spacecraft controllers were designed for static and uncluttered environments. A little over a decade later, the challenges facing spacecraft autonomy now include cluttered, dynamic environments with time-varying constraints, logical modes, fault tolerances, uncertain dynamics, and complex maneuvers. With this rise in complexity, many areas of research have been investigating more experimental control strategies, such as reinforcement learning (RL), as a potential solution to this problem. The research presented herein aims to expand on efforts to quantify the use of RL in autonomous rendezvous, proximity operations, and docking (ARPOD) environments, with consideration to the inherent drawbacks of the more common algorithms present in the field. We present hierarchical model-based RL as a solution to an autonomous docking problem. This algorithm can learn satellite parameters, extrapolate trajectory information, and learn uncertain dynamics via data collection. By using gradient-free model predictive control logic, the algorithm can handle nondifferentiable objectives and complex constraints. Lastly, the hierarchical structure demonstrates an ability to generate feasible trajectories in the presence of integrated third-party subcontrollers commonly found in spacecraft. This study highlights the ability of the hierarchical algorithm to combine and manipulate third-party subpolicies to achieve trajectories not previously trained on.

Data-Efficient Reinforcement Learning for Complex Nonlinear Systems.

This article proposes a data-efficient model-free reinforcement learning (RL) algorithm using Koopman operators for complex nonlinear systems. A high-dimensional data-driven optimal control of the nonlinear system is developed by lifting it into the linear system model. We use a data-driven model-based RL framework to derive an off-policy Bellman equation. Building upon this equation, we deduce the data-efficient RL algorithm, which does not need a Koopman-built linear system model. This algorithm preserves dynamic information while reducing the required data for optimal control learning. Numerical and theoretical analyses of the Koopman eigenfunctions for dataset truncation are discussed in the proposed model-free data-efficient RL algorithm. We validate our framework on the excitation control of the power system.

Model-based reinforcement learning for service mesh fault resiliency in a web application-level

Microservice-based architectures enable different aspects of applications to be created and updated independently, even after deployment. Associated technologies such as service mesh provide fault resiliency through attribute configurations that govern self-adaptive application-level behavior in response to failures, in a manner transparent to the application and constituent microservices. While this provides tremendous flexibility, the configured values of these attributes and the relationships among them can significantly affect the performance and fault resilience of the overall application. It is thus important to perform fault injection and load testing on the application, prior to full deployment. However, given a large number of possible attribute combinations and the complexities of the distributed system underlying microservices and service mesh architectures, it is virtually impossible to determine through traditional software development practices the worst combinations of attribute values and load settings with respect to self-adaptive application-level fault resiliency. To this end, we present a model-based reinforcement learning approach that determines the combinations of attribute and load settings that result in the most significant fault resilience behaviors at an application level. We validate our approach through a case study on a simple request-response service using the Istio service mesh. Our analysis shows that, even for a simple service, our model-based reinforcement learning approach outperforms a baseline selection of action parameters. Further, we show that communicative multi-agent reinforcement learning improves the performance of both the non-communicative single and multi-agent learning paradigms.