Deep Reinforcement Learning Control Research Articles

Optimizing Irrigation Efficiency using Deep Reinforcement Learning in the Field

Agricultural irrigation is a significant contributor to freshwater consumption. However, the current irrigation systems used in the field are not efficient. They rely mainly on soil moisture sensors and the experience of growers, but do not account for future soil moisture loss. Predicting soil moisture loss is challenging because it is influenced by numerous factors, including soil texture, weather conditions, and plant characteristics. This paper proposes a solution to improve irrigation efficiency, which is called DRLIC . DRLIC is a sophisticated irrigation system that uses deep reinforcement learning (DRL) to optimize its performance. The system employs a neural network, known as the DRL control agent, which learns an optimal control policy that considers both the current soil moisture measurement and the future soil moisture loss. We introduce an irrigation reward function that enables our control agent to learn from previous experiences. However, there may be instances where the output of our DRL control agent is unsafe, such as irrigating too much or too little water. To avoid damaging the health of the plants, we implement a safety mechanism that employs a soil moisture predictor to estimate the performance of each action. If the predicted outcome is deemed unsafe, we perform a relatively conservative action instead. To demonstrate the real-world application of our approach, we develop an irrigation system that comprises sprinklers, sensing and control nodes, and a wireless network. We evaluate the performance of DRLIC by deploying it in a testbed consisting of six almond trees. During a 15-day in-field experiment, we compare the water consumption of DRLIC with a widely-used irrigation scheme. Our results indicate that DRLIC outperforms the traditional irrigation method by achieving water savings of up to 9.52%.

A frequency cooperative control strategy for multimicrogrids with EVs based on improved evolutionary-deep reinforcement learning

The construction of microgrid has fully promoted the large-scale access of distributed power and the rapid development of electric vehicles (EVs). And microgrid system faces operational issues such as strong random disturbances from distributed power sources and loads, as well as unexpected events during operation. These issues can lead to unstable microgrid frequency, excessive discharge of EVs, and increased control costs. To solve the issues, a frequency cooperative control strategy for multimicrogrids with EVs based on improved evolutionary-deep reinforcement learning (EDRL) is proposed in this paper. First, consider the effect of the vehicle-to-grid (V2G) process on the minimum time required to fully charge an EV, and the impact of the output distribution of micro turbines (MTs) on the regulation cost, and based on the coupling relationship between generator terminal voltage regulation and system frequency control, a multimicrogrid comprehensive control model is constructed. Second, in order to deal with engineering tasks with deceptive rewards and sparse rewards such as integrated frequency control, evolutionary algorithms and deep reinforcement learning algorithms are combined to effectively assist the training process to jump out of local optimal solutions and approach optimal control strategies. Meanwhile, the algorithm is improved based on the novelty search and intelligent partition strategy, so that the strategy has better convergence characteristics and can reduce the cost of information transmission and computational complexity under the premise of ensuring the control effect. Furthermore, the state space, action space and reward function of the controller are defined. Finally, the simulation results show that the proposed controller has the ability of coordinated control, can effectively reduce the adjustment cost of MT unit and the unnecessary discharging of EVs while ensuring the frequency regulation requirements of each submicrogrid, which is far superior to PID control, fuzzy control, and traditional deep reinforcement learning control.

Environmental-Sensing and adaptive optimization of wave energy converter based on deep reinforcement learning and computational fluid dynamics

This paper introduces a novel coupled model for real-time control of the point absorber wave energy converter (WEC) using parallelized deep reinforcement learning (DRL), where the WEC is situated within a numerical wave tank (NWT) built with the method of computational fluid dynamics (CFD). An in-house solver is developed to couple with the DRL and CFD to solve the interaction between WEC and the fluid environment. Validations on wave generation, wave-floater interaction, and power take-off (PTO) unit are carried out. Then, neglecting the detailed model for the PTO technologies, the DRL-based strategy dynamically adjusts the PTO force as a function of the wave features and floater motion. Based on the interaction data, the model-free DRL is outstanding in adaptability and robustness. Simulation results reveal that DRL control improves the wave energy absorption in irregular wave environments, resulting in improvement of 107.5 % compared to the resistive control, with better device protection performance than the model predictive control (MPC). An additional analysis of model-free characteristics of DRL demonstrates the optimization ability independent of floater modeling. This work is the first in-depth study of DRL control of WECs in CFD simulation, providing a more accurate simulation and an optimization process closer to the reality.

Soft-constrained Deep Reinforcement Learning Controller using Non-linear Model Predictive Filter for Real-time Safe Autonomous Driving

Soft-constrained Deep Reinforcement Learning Controller using Non-linear Model Predictive Filter for Real-time Safe Autonomous Driving

AI-based optimal control of fed-batch biopharmaceutical process leveraging deep reinforcement learning

Optimal control of the fed-batch biopharmaceutical process remains an open research and industrial challenge. The fed-batch process is characterized by non-steady operation, partially observable states, and batch uncertainties. Recently, deep reinforcement learning (DRL) has emerged as a powerful tool for chemical process systems. However, directly applying DRL to the simulation-based biopharmaceutical process has been proven a failure due to the mismatch between reinforcement learning algorithms and simulation-based environments. To fill this research gap, we proposed a novel DRL-based control framework that innovatively incorporates human knowledge. A case study of open-source simulator (IndPenSim), is used to verify the effectiveness of the proposed framework. The DRL controller demonstrates advantages in batch yields, computational loads, and measurement requirements compared to the existing control methods. An improvement of 14 % average penicillin yield is achieved with only 0.01∼0.03 s online computation time per step, showing its potential in the control applications for fed-batch biopharmaceutical process.

An innovative heterogeneous transfer learning framework to enhance the scalability of deep reinforcement learning controllers in buildings with integrated energy systems

AbstractDeep Reinforcement Learning (DRL)-based control shows enhanced performance in the management of integrated energy systems when compared with Rule-Based Controllers (RBCs), but it still lacks scalability and generalisation due to the necessity of using tailored models for the training process. Transfer Learning (TL) is a potential solution to address this limitation. However, existing TL applications in building control have been mostly tested among buildings with similar features, not addressing the need to scale up advanced control in real-world scenarios with diverse energy systems. This paper assesses the performance of an online heterogeneous TL strategy, comparing it with RBC and offline and online DRL controllers in a simulation setup using EnergyPlus and Python. The study tests the transfer in both transductive and inductive settings of a DRL policy designed to manage a chiller coupled with a Thermal Energy Storage (TES). The control policy is pre-trained on a source building and transferred to various target buildings characterised by an integrated energy system including photovoltaic and battery energy storage systems, different building envelope features, occupancy schedule and boundary conditions (e.g., weather and price signal). The TL approach incorporates model slicing, imitation learning and fine-tuning to handle diverse state spaces and reward functions between source and target buildings. Results show that the proposed methodology leads to a reduction of 10% in electricity cost and between 10% and 40% in the mean value of the daily average temperature violation rate compared to RBC and online DRL controllers. Moreover, online TL maximises self-sufficiency and self-consumption by 9% and 11% with respect to RBC. Conversely, online TL achieves worse performance compared to offline DRL in either transductive or inductive settings. However, offline Deep Reinforcement Learning (DRL) agents should be trained at least for 15 episodes to reach the same level of performance as the online TL. Therefore, the proposed online TL methodology is effective, completely model-free and it can be directly implemented in real buildings with satisfying performance.

Successful application of predictive information in deep reinforcement learning control: A case study based on an office building HVAC system

Reinforcement Learning (RL), a promising algorithm for the operational control of Heating, Ventilation, and Air Conditioning (HVAC) systems, has garnered considerable attention and applications. However, traditional RL algorithms typically do not incorporate predictive information for future scenarios, and only a limited number of studies have examined the enhancement and impact of predictive information on RL algorithms. To address the issue of coupling RL and predictive information in HVAC system operation optimization, we employed an open-source framework to examine the impact of various predictive information strategies on RL outcomes. We propose a joint gated recurrent unit (GRU)-RL algorithm to handle situations where a time-series exists in state space. The results from four classic test cases demonstrate that the proposed GRU-RL method can reduce operating costs by approximately 14.5% and increase comfort performance by 88.4% in indoor comfort control and cost-management tasks. Moreover, the GRU-RL method outperformed the conventional DRL method and was merely augmented with prediction information. In indoor temperature regulation, the GRU-RL algorithm improves control efficacy by 14.2% compared to models without predictive information and offers an approximately 5% improvement over traditional network models. Finally, all models were made open source for easy replication and further research.

A reinforcement learning-based temperature control of fluidized bed reactor in gas-phase polyethylene process

This study investigates using deep reinforcement learning (DRL) with proportional-integral-derivative (PID) control for temperature cascade control in a fluidized bed reactor within a commercial gas-phase polyethylene process. The heat exchange system's nonlinearity and frequent disturbances pose challenges for PID controllers, particularly under varying conditions. To address this, a PID-DRL cascade control scheme is developed, where a DRL controller is used in the secondary loop. The DRL controller, designed using the actor-critic framework, is trained using the Deep Deterministic Policy Gradient algorithm. The DRL controller is evaluated in three stand-alone secondary loop experiments, as well as three cascade control experiments. Results reveal the DRL controller surpass the traditional PID controller in both scenarios. The DRL controller shows better set point tracking and interference suppression, indicated by lower integral absolute error (IAE) values. The proposed cascade control structure can be used to enhance reactor stability and product quality in gas-phase polyethylene processes.

Empowering Dynamic Active and Reactive Power Control: A Deep Reinforcement Learning Controller for Three-Phase Grid-Connected Electric Vehicles

Empowering Dynamic Active and Reactive Power Control: A Deep Reinforcement Learning Controller for Three-Phase Grid-Connected Electric Vehicles

Effective Multi-Agent Deep Reinforcement Learning Control With Relative Entropy Regularization

Effective Multi-Agent Deep Reinforcement Learning Control With Relative Entropy Regularization

Comparing actor-critic deep reinforcement learning controllers for enhanced performance on a ball-and-plate system

The optimization of controller parameters remains an ongoing challenge in the field of control system applications. This study introduces a novel approach involving the creation of custom actor-critic deep reinforcement learning (DRL) based PID controllers. These controllers are designed with the goal of achieving adaptive tuning, precise trajectory tracking, and stability in a ball-and-plate system. To achieve this objective, multiple actor-critic reinforcement learning agents were developed using different learning algorithms: Soft actor critic (SAC), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3). These agents incorporate multilayer-perceptron (MLP) policy learning algorithms in both the actor and critic network architectures, employing non-linear activation functions. This enables them to fine-tune PID control parameters within an infinite search space. Additionally, a custom reward function derived from the system’s environment was integrated into the learning process. The performance of the proposed methods was compared against a benchmark method, specifically, an existing deep reinforcement learning controllers reported in the literature. The evaluation of these controllers and other approaches was based on error metrics and time response analysis. Results demonstrate that the proposed controller denoted as SAC-PID(5) excelled in trajectory tracking and outperformed other methods. It exhibited minimal predictive errors and the shortest time responses in the majority of experiments. This highlights the significance of designing a customized SAC agents with appropriate network architecture, which positively impacts the learning process for intelligent tuning of controllers for classical control systems.

Safety-oriented dynamic speed harmonization of mixed traffic flow in nonrecurrent congestion

Freeway nonrecurrent congestion adversely affects collision risks, travel time, emissions, and fuel consumption. Optimizing mixed flow involving Human-Driven Vehicles (HDVs) and Connected Automated vehicles (CAVs) is expected to exist as a challenging issue for a long time prior to full adoption of CAVs. To address this, a safety-oriented Dynamic Speed Harmonization (DSH) strategy for mixed traffic flow in nonrecurrent congestion is proposed in this study. This research utilizes a Deep Reinforcement Learning (DRL) framework to investigate the effects of DSH on the measures of effectiveness in microscopic simulation. A holistic evaluation of HDVs and CAVs integrated with DSH highlights the synergies and trade-offs across different metrics. The results reveal that the implementation of DSH can further improve safety and enhance mobility with increasing CAV penetration rates. While special events may exacerbate congestion, their impact can be mitigated to some extent through speed controls. Spatiotemporal patterns of speed variations at the bottleneck demonstrate that the DRL controller has the capability to dampen oscillations and smoothen traffic flow. A series of sensitivity analyses also indicate the adaptability of the agent under adverse weather scenarios, and the differences of surrogate safety measurements in response to various parametric thresholds.

Surrogate empowered Sim2Real transfer of deep reinforcement learning for ORC superheat control

The Organic Rankine Cycle (ORC) is widely used in industrial waste heat recovery due to its simple structure and easy maintenance. However, in the context of smart manufacturing in the process industry, traditional model-based optimization control methods are unable to adapt to the varying operating conditions of the ORC system or sudden changes in operating modes. Deep reinforcement learning (DRL) has significant advantages in situations with uncertainty as it directly achieves control objectives by interacting with the environment without requiring an explicit model of the controlled plant. Nevertheless, direct application of DRL to physical ORC systems presents unacceptable safety risks, and its generalization performance under model-plant mismatch is insufficient to support ORC control requirements. Therefore, this paper proposes a Sim2Real transfer learning-based DRL control method for ORC superheat control, which aims to provide a new simple, feasible, and user-friendly solution for energy system optimization control. Experimental results show that the proposed method greatly improves the training speed of DRL in ORC control problems and solves the generalization performance issue of the agent under multiple operating conditions through Sim2Real transfer.

Priori knowledge-based deep reinforcement learning control for fan coil unit system

Controlling the water pump and fan in the heating, ventilation and air-conditioning (HVAC) system is crucial for achieving optimal performance of the fan coil unit (FCU) systems. In this paper, the researchers investigate the control problem of the FCU system under the model-free assumption. Specifically, the researchers innovatively model the control problem of the FCU system as a Markov decision process (MDP) and optimize the FCU control using reinforcement learning (RL) methods. However, exploration strategies in the early stage of control using RL methods can lead to poor control performance. To address this, a novel deep deterministic policy gradient control method based on the priori knowledge (DDPG-PK) for the FCU system is proposed, which combines priori knowledge and deep reinforcement learning (DRL) algorithms, following human experience to avoid bad control behavior and achieve better policy control. To validate the performance of the proposed algorithm, the researchers build a variable operating conditions simulation model based on the actual parameters of the real FCU system and related historical data on a general thermodynamic basis. Experimental results show that, compared to other benchmark methods, the DDPG-PK method improves the FCU system's 27.5% satisfaction rate and 23.0% energy efficiency rate in the first three years, and can still keep a state-of-the-art performance in following years.

Dual regularized policy updating and shiftpoint detection for automated deployment of reinforcement learning controllers on industrial mechatronic systems

We propose an algorithmic pipeline enabling deep reinforcement learning controllers to detect when a significant change in system characteristics has occurred and update the control policy accordingly to reattain performance. Reinforcement learning algorithms can learn a policy directly from input–output data and thus optimize for system-specific properties. Yet they face difficulties to adapt, after deployment, to varying operating conditions. Real-world industrial mechatronic systems however demand further levels of performance through adaptation while remaining safe. So far, methods that detect changes in environments exist but have never been studied and applied as a means to update control policies for time-varying systems. We benchmark several methods that detect significant changes in these systems, i.e. shiftpoint detection methods, and present a novel algorithm with a dual regularization architecture. This architecture exploits the prior policy while allowing sufficient flexibility to update for the safety-critical and time-varying system. We validate the method’s performance through benchhmarking and study the effect of its different components and targeted ablation studies on mechatronic systems, both in simulations and experimentally. Results show that our algorithmic pipeline allows for rapid shiftpoint detection, followed by a policy update that reaches expert performance after convergence.

Optimal control of a solar-driven seasonal sorption storage system through deep reinforcement learning

Deep reinforcement learning (DRL) has demonstrated its effectiveness in the control of energy systems, although it has not yet been applied to sorption thermal energy storage (TES) systems. The operation of sorption TES systems is notably more complicated compared to other TES variants. The discharge of a sorption TES occurs at a particular desorption and evaporation temperature. Achieving a continuous and efficient discharge of a sorption TES is a challenging control task if heat required at the evaporator is obtained from the sun or the environment. Its operation is especially complicated during winter, because of the limited availability of solar irradiation and low ambient temperatures. Thus, this study analyzes for first time in the literature the competitiveness of deep reinforcement learning to control a solar-driven seasonal sorption TES system and compares it against traditional optimized rule-based control strategy. The system, located in Central Europe, supplied domestic hot water and space heating to a single-family house. Two DRL models were developed and trained to operate the system under two different sets of data: 120 winter consecutive days and 60 winter non-consecutive days. The results showed that the DRL control strategy reduced the system operational costs by 28% in a 60 winter days scenario. For a 120 winter days scenario, the operational cost savings decreased to 13% because the smart control performed worst once the sorption TES was fully discharged. These results were derived from a four-year validation data set, bolstering their robustness. The study demonstrates the successful application of DRL in controlling a solar-driven seasonal sorption TES system, yielding considerable economic savings compared to an RBC strategy. Subsequent work will consist of implementing the smart control strategy at prototype level to assess its performance.

Design and implementation of an indoor environment management system using a deep reinforcement learning approach

The Indoor Household Environment (IHE) has caught the attention of industry and academia due to the substantial amount of time that families spend indoors. Domestic air pollution imposes significant burdens on communities, although both indoor and outdoor sources of air pollution contribute to outdoor pollution. This article presents a precise approach for an Intelligent Home Energy Management System (IHEMS) that integrates Deep Reinforcement Learning (DRL) and Demand Response (DR) techniques. IHEMS employs Markov decision processes to handle problems and concerns efficiently. A novel approach has been developed to enhance the efficiency of deep reinforcement learning control, utilizing a double deep Q network and a restart strategy during a priority event. Unlike the HEMS technique, which uses a predictive control model, the optimal strategy resulted in a 20% decrease in monthly power consumption.

LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle

LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle

Research on Deep Reinforcement Learning Control Algorithm for Active Suspension Considering Uncertain Time Delay.

The uncertain delay characteristic of actuators is a critical factor that affects the control effectiveness of the active suspension system. Therefore, it is crucial to develop a control algorithm that takes into account this uncertain delay in order to ensure stable control performance. This study presents a novel active suspension control algorithm based on deep reinforcement learning (DRL) that specifically addresses the issue of uncertain delay. In this approach, a twin-delayed deep deterministic policy gradient (TD3) algorithm with system delay is employed to obtain the optimal control policy by iteratively solving the dynamic model of the active suspension system, considering the delay. Furthermore, three different operating conditions were designed for simulation to evaluate the control performance: deterministic delay, semi-regular delay, and uncertain delay. The experimental results demonstrate that the proposed algorithm achieves excellent control performance under various operating conditions. Compared to passive suspension, the optimization of body vertical acceleration is improved by more than 30%, and the proposed algorithm effectively mitigates body vibration in the low frequency range. It consistently maintains a more than 30% improvement in ride comfort optimization even under the most severe operating conditions and at different speeds, demonstrating the algorithm's potential for practical application.

Delta wing rocking motion suppression by deep reinforcement learning

The wing rock phenomenon is a self-excited roll motion occurring on the aircraft at high angles of attack (AoAs), which negatively impact safety and maneuverability. The limit cycle oscillation is a typical characteristic of this self-excited roll motion. A controller employing a model-free training approach is constructed on the basis of deep reinforcement learning (DRL) to address this severe nonlinear control problem. For an 80° swept delta wing model, the analytical model describing the nonlinear behavior of wing rock is presented and implemented in the simulation environment. The DRL-based controller is trained with the proximal policy optimization algorithm. A reward function is carefully designed for training to achieve excellent convergence stability. Various simulations were conducted at a series of unlearned initial conditions (roll angles, AoAs) to demonstrate the effectiveness and generalization capability of the proposed method and verify and evaluate the performance of the trained DRL controller. Finally, the scenario that undergoes disturbance is added to confirm the effectiveness and robustness of the proposed DRL-based controller. Results show that the DRL-based controller could effectively regulate the oscillation and retain the capability to suppress the un-wanted behavior with favorable generalization capability and robustness. Therefore, delta wing rocking motion suppression based on DRL has powerful intelligent control performance and provides a new idea for predicting the evolution of aerodynamics in rocking motion suppression.