Model-based Deep Reinforcement Learning Research Articles

Model-based deep reinforcement learning for active control of flow around a circular cylinder using action-informed episode-based neural ordinary differential equations

Model-based deep reinforcement learning for active control of flow around a circular cylinder using action-informed episode-based neural ordinary differential equations

Dyna algorithm-based reinforcement learning energy management for fuel cell hybrid engineering vehicles

Fuel cell hybrid engineering vehicles utilize green energy with high energy conversion efficiency. Optimizing the utilization of hybrid energy storage systems is crucial for improving energy efficiency and promoting energy conservation. This study introduces an innovative energy management framework that employs reinforcement learning to regulate a fuel cell/battery hybrid engineering vehicle. This novel framework facilitates the system achieve optimal energy management, even when confronted with demanding and ever-changing operational scenarios. The proposed strategy integrates the concepts of the Dyna algorithm and the deep Q-network (DQN) algorithm. A working conditions predictive model is developed using the gated recurrent unit (GRU) neural network. The aim of this predictive model is to generate a virtual depiction of the environment and accelerate the process of model updating. The results from simulations and hardware-in-the-loop test illustrate that the predictive model exhibits effective prediction capabilities, enabling the development of an accurate environmental model even when limited sample data is available. The proposed strategy exhibits superior performance compared to the DQN and Q-Learning algorithms regarding adaptability, real-time optimization, training efficiency, convergence speed, and energy conservation. When interacting with the environment, our proposed learning algorithm exhibits a faster convergence rate. Moreover, this strategy achieves a fuel economy of 95 %, as determined by the dynamic programming-based method. The model-based deep reinforcement learning approach is practical and does not require extensive training. An added benefit of this energy management architecture is its use of a more accurate world model.

Model-based deep reinforcement learning for accelerated learning from flow simulations

Model-based deep reinforcement learning for accelerated learning from flow simulations

Model-Based Deep Reinforcement Learning Framework for Channel Access in Wireless Networks

Model-Based Deep Reinforcement Learning Framework for Channel Access in Wireless Networks

Fuzzy-based predictive deep reinforcement learning for robust and constrained optimal control of industrial solar thermal plants

Integrating distributed solar fields (DSFs) into conventional heat and power plants (CHPs) of industries is mostly constrained by the availability of a real-time capable control scheme. Safe and efficient operation of industrial DSF requires the supply of a fluctuating and periodically available energy at the required temperature while reducing losses and ensuring operational constraint. Reinforcement learning (RL), specifically Q-learning methods, are being recently applied for such control tasks. However, existing RL approaches cannot be directly applied to continuous domain control tasks or tasks with operation constraints. While using deep learning in model-free RL can remove these limitations, tractability and scalability boundaries on performing the deep learning are becoming significant, with scarce data and multi-objective concerns. In contrast, model-based deep RL schemes can be applied for sample efficiency and satisfying operational constraints, but their modeling framework is not general and accurate. To address these challenges, the work here develops a hybrid fuzzy convolution model (HFCM) that takes full advantages of data, models (dynamic and steady-state), and prior knowledge on industrial DSF. The HFCM is then extended for use in deep deterministic policy gradient (DDPG) algorithm to learn the DSF control task. This is done so by solving a multi-objective optimization problem, which is formulated as a constrained Markov decision process (CMDP) with continuous state and actions. Some practical and relevant findings were made with this HFCM. Firstly, it allowed the DDPG agent to learn independently a temperature state and a disturbance state, using only temperature measurement and single realizations of state vector. It also permitted the testing of an operational state violation strategy on a DDPG actions, and at the same time, the correction of the effects at a safety layer, if needed. Furthermore, the proposed control strategy simultaneously reduced mean temperature tracking error by 24%–51% and energy gain by 13.9%–17.35% when compared respectively to model-free DDPG and MPC baselines.

Advancing Process Control in Fluidized Bed Biomass Gasification Using Model-Based Deep Reinforcement Learning

This study presents a model-based deep reinforcement learning (MB-DRL) controller for the fluidized bed biomass gasification (FBG) process. The MB-DRL controller integrates a deep neural network (DNN) model and a reinforcement learning-based optimizer. The DNN model is trained with operational data from a pilot-scale FBG plant to approximate FBG process dynamics. The reinforcement learning-based optimizer employs a specially designed reward function, determining optimal control policies for FBG. Moreover, the controller includes an online learning component, ensuring periodic updates to the DNN model training. The performance of the controller is evaluated by testing its control accuracy for regulating synthetic gas composition, flow rate, and CO concentration in the FBG. The evaluation also includes a comparison with a model predictive controller. The results demonstrate the superior control performance of MB-DRL, surpassing MPC by over 15% in regulating synthetic gas composition and flow rate, with similar effectiveness observed in synthetic gas temperature control. Additionally, this study also includes systematic investigations into factors like DNN layer count and learning update intervals to provide insights for the practical implementation of the controller. The results, presenting a 50% reduction in control error with the addition of a single layer to the DNN model, highlight the significance of optimizing MB-DRL for effective implementation.

Learning to see via epiretinal implant stimulation in silico with model-based deep reinforcement learning

Objective: Diseases such as age-related macular degeneration and retinitis pigmentosa cause the degradation of the photoreceptor layer. One approach to restore vision is to electrically stimulate the surviving retinal ganglion cells with a microelectrode array such as epiretinal implants. Epiretinal implants are known to generate visible anisotropic shapes elongated along the axon fascicles of neighboring retinal ganglion cells. Recent work has demonstrated that to obtain isotropic pixel-like shapes, it is possible to map axon fascicles and avoid stimulating them by inactivating electrodes or lowering stimulation current levels. Avoiding axon fascicule stimulation aims to remove brushstroke-like shapes in favor of a more reduced set of pixel-like shapes. Approach: In this study, we propose the use of isotropic and anisotropic shapes to render intelligible images on the retina of a virtual patient in a reinforcement learning environment named rlretina. The environment formalizes the task as using brushstrokes in a stroke-based rendering task. Main Results: We train a deep reinforcement learning agent that learns to assemble isotropic and anisotropic shapes to form an image. We investigate which error-based or perception-based metrics are adequate to reward the agent. The agent is trained in a model-based data generation fashion using the psychophysically validated axon map model to render images as perceived by different virtual patients. We show that the agent can generate more intelligible images compared to the naive method in different virtual patients. Significance: This work shares a new way to address epiretinal stimulation that constitutes a first step towards improving visual acuity in artificially-restored vision using anisotropic phosphenes.

Optimization of Peer-to-Peer Energy Trading with a Model-Based Deep Reinforcement Learning in a Non-Sharing Information Scenario

Optimization of Peer-to-Peer Energy Trading with a Model-Based Deep Reinforcement Learning in a Non-Sharing Information Scenario

Multi-Zone HVAC Control With Model-Based Deep Reinforcement Learning

Multi-Zone HVAC Control With Model-Based Deep Reinforcement Learning

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Inventory management of new products in retailers using model-based deep reinforcement learning

This study addresses the optimal inventory management problem for new smartphone products as an effective example of a supply chain with a short product life cycle. The determination of the optimal inventory level leads to a reduction of lost opportunities and defective inventory, which is an important issue from a profit improvement perspective. Mathematical optimization and reinforcement learning approaches have been proposed for inventory management; however, most of these approaches focus on products that are regularly sold over a long period. Thus, when the target is a new product, it is difficult to optimize inventory control from its day of release due to a lack of sufficient data for learning. To solve this problem, we focus on model-based deep reinforcement learning with high sample efficiency and propose an inventory management method for new products that combines model learning in an offline environment and planning in an online environment. Simulations using real-world historical sales datasets demonstrate that the proposed method outperforms existing methods in terms of profitability, efficiency, and customer satisfaction. In particular, the proposed method improves total rewards and inventory turnover by ¿5% each than the heuristic method while maintaining the same stock-out rate. In addition, the results demonstrate that the proposed method can maintain stable inventory control for multiproduct and multistore supply chains.

Surrogate model-based deep reinforcement learning for experimental study of active flow control of circular cylinder

The flow around a circular cylinder is a classical problem in fluid mechanics, and the reduction of drag and lift has been a long-standing research focus in flow control. In this study, we apply deep reinforcement learning (DRL) to intelligently determine suction flow rate on a circular cylinder model in wind tunnel, aiming to minimize aerodynamic forces while considering energy dissipation efficiency. However, DRL has been criticized for its low data utilization rate and long training period, leading to high experimental training cost. To address these issues, this study employs a surrogate model to optimize the reward function and hyperparameters, and this method is called SM-DRL. This SM-DRL method efficiently expedites the DRL training process, significantly reducing the experimental training cost. In addition, DRL training was conducted in a variable flow field, and the robustness of the obtained DRL model was tested. The results indicate that the DRL agent can determine the optimal control strategy, i.e., automatically select the optimal suction flow rate in terms of the incoming wind velocity, resulting in a significant reduction in lift fluctuations. For Reynolds number of 1.65×104, the reduction in lift fluctuations of the circular cylinder exceeds 50%.

Application of Model-based Deep Reinforcement Learning Framework to Thermal Power Plant Operation Considering Performance Change

In recent years, there have been increasing expectations for the development of advanced plant operational support systems that can automate complex tasks and autonomouslyoptimize operational procedures in thermal power plants. The performance of the equipment changes during operation and maintenance; hence, it is necessary to adjust the operating process to satisfy the operational constraints. In this study, we investigated a framework based on model-based deep reinforcement learning for acquiring control methods that are robust to changes in equipment performance using a digital twin model. A case study of the operational planning of a thermal power plant was presented and it was demonstrated that a stable control system can be constructed even when plant characteristics are changing.

Modeling and energy dynamic control for a ZEH via hybrid model-based deep reinforcement learning

Modeling and energy dynamic control for a ZEH via hybrid model-based deep reinforcement learning

Autonomous Navigation System in Pedestrian Scenarios Using a Dreamer-Based Motion Planner

Navigation among pedestrians is a crucial capability of service robots; however, it is a challenge to manage time-varying environments stably. Recent deep reinforcement learning (DRL)-based approaches to crowd navigation have yielded numerous promising applications. However, they rely heavily on initial imitation learning and colossal positive datasets. Moreover, the difficulties in accurately localizing robots, detecting and tracking humans, representing and generalizing reciprocal human relationships restrict their deployment in real-world problems. We propose a Dreamer-based motion planner for collision-free navigation in diverse pedestrian scenarios. Our RL framework can completely learn from zero experience via a model-based DRL. The robot and humans are first projected onto a map, which is subsequently decoded into low-dimensional latent state. A predictive dynamic model in the latent space is jointly created to efficiently optimize the navigation policy. Additionally, we leverage the techniques of system identification, domain randomization, clustering and LiDAR SLAM for practical deployment. Simulation ablations and real implementations demonstrate that our motion planner outperforms state-of-the-art methods, and that the navigation system can be physically implemented in the real world.

Model-Based Deep Reinforcement Learning with Traffic Inference for Traffic Signal Control

In the modern world, the extremely rapid growth of traffic demand has become a major problem for urban traffic development. Continuous optimization of signal control systems is an important way to relieve traffic pressure in cities. In recent years, with the impressive development of deep reinforcement learning (DRL), some DRL approaches have started to be applied to traffic signal control. Unlike traditional signal control methods, agents trained using DRL approaches continuously receive feedback from the environment to continuously improve the policy. Since current research in the field is more focused on the performance of the agent, data efficiency during training is ignored to some extent. However, in traffic signal control tasks, the cost of trial and error is very expensive. In this paper, we propose a DRL approach based on a traffic inference model. The proposed traffic inference model is based on the future information given based on upstream intersections and data from the environment to continuously learn the changing patterns of the traffic environment in order to make inferences about changes in the traffic environment. In the proposed algorithm, the inference model interacts with the agent instead of the environment. Through comprehensive experiments based on realistic datasets, we demonstrate that our proposed algorithm is superior to other algorithms in terms of its data efficiency and stronger performance.

Aero-Engine Modeling and Control Method with Model-Based Deep Reinforcement Learning

Due to the strong representation ability and capability of learning from data measurements, deep reinforcement learning has emerged as a powerful control method, especially for nonlinear systems, such as the aero-engine control system. In this paper, a novel application of deep reinforcement learning (DRL) is presented for aero-engine control. In addition, transition dynamic characteristic information of the aero-engine is extracted from the replay buffer of deep reinforcement learning to train a neural-network dynamic prediction model for the aero-engine. In turn, the dynamic prediction model is used to improve the learning efficiency of reinforcement learning. The practical applicability of the proposed control system is demonstrated by the numerical simulations. Compared with the traditional control system, this novel aero-engine control system has faster response speed, stronger self-learning ability, and avoids the complicated manual parameter adjustment without sacrificing the control performance. Moreover, the dynamic prediction model has satisfactory prediction accuracy, and the model-based method can achieve higher learning efficiency than the model-free method.

A model-based deep reinforcement learning approach to the nonblocking coordination of modular supervisors of discrete event systems

Modular supervisory control may lead to conflicts among the modular supervisors for large-scale discrete event systems. The existing methods for ensuring nonblocking control of modular supervisors either exploit favorable structures in the system model to guarantee the nonblocking property of modular supervisors or employ hierarchical model abstraction methods for reducing the computational complexity of designing a nonblocking coordinator. The nonblocking modular control problem is, in general, NP-hard. This study integrates supervisory control theory and a model-based deep reinforcement learning method to synthesize a nonblocking coordinator for the modular supervisors. The deep reinforcement learning method significantly reduces the computational complexity by avoiding the computation of synchronization of multiple modular supervisors and the plant models. The supervisory control function is approximated by the deep neural network instead of a large-sized finite automaton. Furthermore, the proposed model-based deep reinforcement learning method is more efficient than the standard deep Q network algorithm.

World Model-based Deep Reinforcement Learning Considering Timings of Touchdown and Liftoff

World Model-based Deep Reinforcement Learning Considering Timings of Touchdown and Liftoff

Data-driven distributed formation control of under-actuated unmanned surface vehicles with collision avoidance via model-based deep reinforcement learning

This paper addresses the distributed formation control with collision avoidance for multiple under-actuated unmanned surface vehicles (USVs) subject to fully unknown models. A fully data-driven distributed control approach is proposed for multiple USVs to achieve a desired formation based on model-based deep reinforcement learning. Specifically, a deep neural network is firstly trained to approximate the dynamic model of each USV by utilizing recorded input and output data. Then, by taking collision avoidance requirements into account, the model predictive formation controllers are proposed for USVs to achieve the safe formation control task based on the learned vehicle dynamics. It is shown that after learning with offline and online data, the proposed fully data-driven distributed controllers are able to achieve a safe formation. Simulations results are given to substantiate the feasibility and efficacy of the proposed model-based deep reinforcement learning method for distributed formation control of under-actuated USVs with fully unknown models.