Reinforcement Learning Control Research Articles

Asymmetric coding of reward prediction errors in human insula and dorsomedial prefrontal cortex

The signed value and unsigned salience of reward prediction errors (RPEs) are critical to understanding reinforcement learning (RL) and cognitive control. Dorsomedial prefrontal cortex (dMPFC) and insula (INS) are key regions for integrating reward and surprise information, but conflicting evidence for both signed and unsigned activity has led to multiple proposals for the nature of RPE representations in these brain areas. Recently developed RL models allow neurons to respond differently to positive and negative RPEs. Here, we use intracranially recorded high frequency activity (HFA) to test whether this flexible asymmetric coding strategy captures RPE coding diversity in human INS and dMPFC. At the region level, we found a bias towards positive RPEs in both areas which paralleled behavioral adaptation. At the local level, we found spatially interleaved neural populations responding to unsigned RPE salience and valence-specific positive and negative RPEs. Furthermore, directional connectivity estimates revealed a leading role of INS in communicating positive and unsigned RPEs to dMPFC. These findings support asymmetric coding across distinct but intermingled neural populations as a core principle of RPE processing and inform theories of the role of dMPFC and INS in RL and cognitive control.

Efficient model-free control of chiller plants via cluster-based deep reinforcement learning

Optimizing the operation of chiller plants can reduce energy consumption in heating, ventilation and air conditioning (HVAC) systems and is one of the effective ways to achieve “carbon neutrality” in buildings. Reinforcement learning (RL) demonstrates great potential in this field. However, in real applications, RL continues to face challenges in dealing with extensive state spaces and achieving stable convergence. The main challenge lies in the high coupling and complexity of chiller plant control, making it difficult for RL agents to finely optimize each state during training, resulting in suboptimal control effectiveness. Additionally, as the state space expands, the computation cost and training duration also rise greatly. To address these issues, this paper proposes a clustering-based RL control method. Firstly, the state is divided into more homogeneous subsets by the K-means algorithm. Then, different RL agents are trained based on these subsets, and a public experience pool is established between neighboring agents to enhance the experience exchange. This design makes the training process more context-oriented, enabling agents to learn from similar feature states and reducing the issue of decreased boundary state handling due to state dividing. The experimental results demonstrate that the proposed method achieves 10.1% increase in energy-saving compared to rule-based control, and is close to that of the model-based control. (1.2% difference). Through state clustering, both the learning speed and control effectiveness of the agent are significantly improved. Compared to non-clustered RL, the proposed method reduces the training time by 66.7% and improves the energy saving by 0.9%.

Quantitative comparison of reinforcement learning and data-driven model predictive control for chemical and biological processes

As manufacturing processes transition towards digitalization, data-driven process control is emerging as a key area of interest in future artificial intelligence technology. A crucial aspect in implementing data-driven process control is “What should we learn from the data?”. In general, the data-driven control method can be categorized into two main approaches: Learning the model and learning the value. To assist in selecting the more suitable approach, this paper applies six different control methods, with three falling under each approach, to three distinct manufacturing process systems. The simulation results indicate that the model-learning approaches display higher data efficiency and exhibit lower variance in total cost. These methods prove to be particularly advantageous for addressing the regulation problems. Conversely, value-learning approaches show competitive potential in closed-loop identification and in managing economic cost problems. The remaining challenges associated with each technique are discussed, along with practical considerations for their implementation.

A Deep Reinforcement Learning Strategy for Surrounding Vehicles-Based Lane-Keeping Control.

As autonomous vehicles (AVs) are advancing to higher levels of autonomy and performance, the associated technologies are becoming increasingly diverse. Lane-keeping systems (LKS), corresponding to a key functionality of AVs, considerably enhance driver convenience. With drivers increasingly relying on autonomous driving technologies, the importance of safety features, such as fail-safe mechanisms in the event of sensor failures, has gained prominence. Therefore, this paper proposes a reinforcement learning (RL) control method for lane-keeping, which uses surrounding object information derived through LiDAR sensors instead of camera sensors for LKS. This approach uses surrounding vehicle and object information as observations for the RL framework to maintain the vehicle's current lane. The learning environment is established by integrating simulation tools, such as IPG CarMaker, which incorporates vehicle dynamics, and MATLAB Simulink for data analysis and RL model creation. To further validate the applicability of the LiDAR sensor data in real-world settings, Gaussian noise is introduced in the virtual simulation environment to mimic sensor noise in actual operational conditions.

Robust Control of An Inverted Pendulum System Based on Policy Iteration in Reinforcement Learning

This paper is primarily focused on the robust control of an inverted pendulum system based on policy iteration in reinforcement learning. First, a mathematical model of the single inverted pendulum system is established through a force analysis of the pendulum and trolley. Second, based on the theory of robust optimal control, the robust control of the uncertain linear inverted pendulum system is transformed into an optimal control problem with an appropriate performance index. Moreover, for the uncertain linear and nonlinear systems, two reinforcement-learning control algorithms are proposed using the policy iteration method. Finally, two numerical examples are provided to validate the reinforcement learning algorithms for the robust control of the inverted pendulum systems.

AdaptNet: Policy Adaptation for Physics-Based Character Control

Motivated by humans' ability to adapt skills in the learning of new ones, this paper presents AdaptNet, an approach for modifying the latent space of existing policies to allow new behaviors to be quickly learned from like tasks in comparison to learning from scratch. Building on top of a given reinforcement learning controller, AdaptNet uses a two-tier hierarchy that augments the original state embedding to support modest changes in a behavior and further modifies the policy network layers to make more substantive changes. The technique is shown to be effective for adapting existing physics-based controllers to a wide range of new styles for locomotion, new task targets, changes in character morphology and extensive changes in environment. Furthermore, it exhibits significant increase in learning efficiency, as indicated by greatly reduced training times when compared to training from scratch or using other approaches that modify existing policies. Code is available at https://motion-lab.github.io/AdaptNet .

Human-Guided Reinforcement Learning With Sim-to-Real Transfer for Autonomous Navigation.

Reinforcement learning (RL) is a promising approach in unmanned ground vehicles (UGVs) applications, but limited computing resource makes it challenging to deploy a well-behaved RL strategy with sophisticated neural networks. Meanwhile, the training of RL on navigation tasks is difficult, which requires a carefully-designed reward function and a large number of interactions, yet RL navigation can still fail due to many corner cases. This shows the limited intelligence of current RL methods, thereby prompting us to rethink combining RL with human intelligence. In this paper, a human-guided RL framework is proposed to improve RL performance both during learning in the simulator and deployment in the real world. The framework allows humans to intervene in RL's control progress and provide demonstrations as needed, thereby improving RL's capabilities. An innovative human-guided RL algorithm is proposed that utilizes a series of mechanisms to improve the effectiveness of human guidance, including human-guided learning objective, prioritized human experience replay, and human intervention-based reward shaping. Our RL method is trained in simulation and then transferred to the real world, and we develop a denoised representation for domain adaptation to mitigate the simulation-to-real gap. Our method is validated through simulations and real-world experiments to navigate UGVs in diverse and dynamic environments based only on tiny neural networks and image inputs. Our method performs better in goal-reaching and safety than existing learning- and model-based navigation approaches and is robust to changes in input features and ego kinetics. Furthermore, our method allows small-scale human demonstrations to be used to improve the trained RL agent and learn expected behaviors online.

Reinforcement Learning-Based Event-Triggered FCS-MPC for Power Converters

This article aims to first focus on an improvement of finite control-set model predictive control strategy for power converters that is based on reinforcement learning event-triggered predictive control architecture with the help of adaptive dynamic programming technique and event-triggered mechanism subject to system uncertainties. Our development, endowed with the merits of reinforcement learning and event-triggered control as well as predictive control solution, is able to alleviate the issues of parametric uncertainties and high switching frequency inherent in the existing scheme, while retaining the merits of the finite control-set model predictive control. Finally, this proposal is experimentally evaluated, where robust performance tests confirm the interest and applicability of the proposed control methodology.

Two‐layer leader‐follower optimal affine formation maneuver control for networked unmanned surface vessels with input saturations

AbstractThis article studies the two‐layer leader‐follower optimal affine formation maneuver (AFM) control problem for multiple unmanned surface vessels (USV). The basic idea of the developed two‐layer control framework is to realize the time‐varying formation for the leader group in advance, and then let the follower USVs agilely respond to the target formation affinely localized by leaders. To realize this goal, a time‐varying formation reference system is first synthesized to describe leaders' target formation in the affine map of the nominal configuration. The reference system is set up by virtual vessel velocity command and unrotated geometric command, which make it possesses an intuitive application structure for operators to plan AFM commands in real‐time sensed surrounding environments. Then, by means of optimized backstepping design and the adaptive dynamic programming technique, actor‐only reinforcement learning controllers are established for USVs in both two layers to realize the optimal target formation tracking objective. For the proposed optimal controllers, the persistent excitation condition that is required by common optimal control schemes is also removed, which endows the presented scheme with lower computational complexity. Finally, theoretical analysis and simulation results illustrate the closed‐loop stability and the effectiveness of the presented scheme.

Control Schemes for Quadrotor UAV: Taxonomy and Survey

Quadrotor Unmanned Aerial Vehicle (UAV) is an unstable system, so it needs to be controlled efficiently and intelligently. Moreover, due to its non-linear, coupled, and under-actuated nature, the quadrotor has become an important research platform to study and validate various control theories. Different control approaches have been used to control the quadrotor UAV. In this context, a comprehensive study of different control schemes is presented in this research. First, an overview of the working and different applications of quadrotor UAVs is presented. Second, a mathematical model of the quadrotor is discussed. Later, the experimental results of various existing control techniques are discussed and compared. The various control schemes discussed and described for quadrotors are; Proportional Integral and Derivative (PID), Linear Quadratic Regulator (LQR), H-infinity ( H ∞ ), Sliding Mode Control (SMC), Feedback Linearization (FBL), Model Predictive Control (MPC), Fuzzy Logic Control (FLC), Artificial Neural Network (ANN), Iterative Learning Control (ILC), Reinforcement Learning Control (RLC), Brain Emotional Learning Control (BELC), Memory Based Control (MBC), Nested Saturation Control (NSC), and Hybrid Controllers (HC). Comparison is done among all the control techniques and it is concluded that the hybrid control method gives improved results. This survey presents a broad overview of the state-of-the-art in UAV design, control, and implementation for real-life applications.

A practically implementable reinforcement learning control approach by leveraging offset-free model predictive control

This work addresses the problem of designing an offset-free implementable reinforcement learning (RL) controller for nonlinear processes. RL-based controllers can update the control policy using observed data obtained online based on the controller-process interactions. This allows to alleviate the regular model maintenance step that is essential in advanced control techniques such as model predictive control (MPC). However, random explorations are required for an RL agent to find optimal state–action regions so as to finally reach the optimal policy. This is not implementable in practical situations due to safety concerns and economic objectives. To address this issue, a pre-training strategy is proposed to provide a secure platform for online implementations of the RL controllers. To this end, an offset-free MPC (representative industrial MPC) optimization problem is leveraged to train the RL agent offline. Having obtained similar performance to the offset-free MPC, the RL agent is utilized for online control to interact with the actual process. The efficacy of the proposed approach to handle nonlinearity and changes in plant operating conditions (due to unmeasured disturbances) are demonstrated through simulations on a chemical reactor example for a pH neutralization process. The results show that the proposed RL controller can significantly improve the oscillatory closed-loop responses, obtained by running the offset-free MPC due to the plant-model mismatch and unmeasured disturbances.

Energy management for demand response in networked greenhouses with multi-agent deep reinforcement learning

Greenhouses are key to ensuring food security and realizing a sustainable future for agriculture. However, to ensure crop growth efficiency, greenhouses consume a significant amount of energy, primarily through climate control and artificial lighting systems. Owing to this high energy consumption, a network of greenhouses exhibits immense potential to participate in demand response programs for power grid stability. In this work, a multi-agent deep reinforcement learning (MADRL) control framework utilizing an actor-critic algorithm with a shared attention mechanism is proposed for energy management in networked greenhouses. A network of renewable energy integrated greenhouses is constructed to interact with the power grid, when necessary, to address the fluctuations associated with renewable energy generation and dynamic electricity prices. The viability and scalability of this multi-agent approach is demonstrated by evaluating its capabilities for a network of five greenhouses of varying capacities. The proposed MADRL-based control approach for demand-side energy management in networked greenhouses demonstrates efficiency in maintaining indoor climate in all greenhouses while ensuring a 28% reduction in net load demand as compared to well-known algorithms.

Priori knowledge-based deep reinforcement learning control for fan coil unit system

Controlling the water pump and fan in the heating, ventilation and air-conditioning (HVAC) system is crucial for achieving optimal performance of the fan coil unit (FCU) systems. In this paper, the researchers investigate the control problem of the FCU system under the model-free assumption. Specifically, the researchers innovatively model the control problem of the FCU system as a Markov decision process (MDP) and optimize the FCU control using reinforcement learning (RL) methods. However, exploration strategies in the early stage of control using RL methods can lead to poor control performance. To address this, a novel deep deterministic policy gradient control method based on the priori knowledge (DDPG-PK) for the FCU system is proposed, which combines priori knowledge and deep reinforcement learning (DRL) algorithms, following human experience to avoid bad control behavior and achieve better policy control. To validate the performance of the proposed algorithm, the researchers build a variable operating conditions simulation model based on the actual parameters of the real FCU system and related historical data on a general thermodynamic basis. Experimental results show that, compared to other benchmark methods, the DDPG-PK method improves the FCU system's 27.5% satisfaction rate and 23.0% energy efficiency rate in the first three years, and can still keep a state-of-the-art performance in following years.

Dual regularized policy updating and shiftpoint detection for automated deployment of reinforcement learning controllers on industrial mechatronic systems

We propose an algorithmic pipeline enabling deep reinforcement learning controllers to detect when a significant change in system characteristics has occurred and update the control policy accordingly to reattain performance. Reinforcement learning algorithms can learn a policy directly from input–output data and thus optimize for system-specific properties. Yet they face difficulties to adapt, after deployment, to varying operating conditions. Real-world industrial mechatronic systems however demand further levels of performance through adaptation while remaining safe. So far, methods that detect changes in environments exist but have never been studied and applied as a means to update control policies for time-varying systems. We benchmark several methods that detect significant changes in these systems, i.e. shiftpoint detection methods, and present a novel algorithm with a dual regularization architecture. This architecture exploits the prior policy while allowing sufficient flexibility to update for the safety-critical and time-varying system. We validate the method’s performance through benchhmarking and study the effect of its different components and targeted ablation studies on mechatronic systems, both in simulations and experimentally. Results show that our algorithmic pipeline allows for rapid shiftpoint detection, followed by a policy update that reaches expert performance after convergence.

Reinforcement learning swarm of self-organizing unmanned surface vehicles with unavailable dynamics

In this paper, self-organizing a swarm of unmanned surface vehicles (USVs) is creatively solved by optimal reinforcement learning control approach without acquiring USV dynamics. Self-organizing swarm mechanism for flexible decision-making is established by combing control theory with repulsive potentials, thereby lifting orientation constraints on USV swarm coordination. Distributed finite-time extended state observers are devised to sufficiently accommodate multiple unknowns covering internal dynamics and external uncertainties/disturbances. Swarming coordination of individual USV is optimally synthesized by reinforcement learning-based distributed control architecture. As a consequence, semi-global asymptotic stability of the entire self-organizing swarm can be rigorously ensured by Lyapunov analysis. Virtual-reality experiments show that reinforcement learning swarm is skilled in coordinating a herd of USVs tracking a trajectory.

Basal-bolus advisor for type 1 diabetes (T1D) patients using multi-agent reinforcement learning (RL) methodology

This study presents in-silico design and verification of an advanced multi-agent reinforcement learning (RL) strategy for personalized glucose regulation in individuals diagnosed with type 1 diabetes (T1D). The proposed framework realizes a closed-loop system, encompassing a blood glucose (BG) metabolic model acting as virtual patient, and a sophisticated multi-agent soft actor–critic RL controller serving as the basal-bolus advisor. The efficacy of the RL agents in meeting the recommended glycemic targets is benchmarked against traditional therapeutic approaches and rigorously assessed across three distinct scenarios: (A) nominal, three meals per day, (B) robustness against meal disturbances, and (C) robustness against insulin sensitivity disturbances. Key evaluation metrics are minimum, maximum and mean values of glucose concentrations, the temporal distribution across various BG intervals, and the daily averages of both bolus and basal insulin administrations. Empirical results underscore the superior performance of the RL-driven basal-bolus advisor, manifesting in diminished glycemic fluctuations and an augmented duration within the optimal range of 70–180 mg/dL. Specifically, in scenarios A, B, and C, the time within this desired range witnessed a surge from 66.66±34.97% to 92.55±4.05%, 64.13±33.84% to 93.91±6.03%, and 58.85±34.67% to 78.34±13.28%, respectively. The RL paradigm also exhibited a marked proficiency in curtailing severe hyperglycemia incidents (p ≤0.05) and diminishing hypoglycemia occurrences. In scenarios A and B, hypoglycemic incidents were curtailed from 14.2% ±32.27% to 3.77% ±4.01% and 16.59% ±32.42% to 2.63% ±4.09%, respectively. Intriguingly, scenario C recorded zero hypoglycemic incidents for both methodologies, attributed to a decline in insulin sensitivity. Additionally, a statistically significant decrement in the mean daily basal insulin dosage was observed with the RL agent in comparison to traditional therapy (p ≤0.05). Collectively, the outcomes accentuate the potential of the multi-agent RL strategy in enhancing glucose regulation and minimizing the peril of severe hyperglycemia for T1D patients.

Optimal Control of District Cooling Energy Plant With Reinforcement Learning and MPC

Abstract We consider the problem of optimal control of district cooling energy plants (DCEPs) consisting of multiple chillers, a cooling tower, and a thermal energy storage (TES), in the presence of time-varying electricity price. A straightforward application of model predictive control (MPC) requires solving a challenging mixed-integer nonlinear program (MINLP) because of the on/off of chillers and the complexity of the DCEP model. Reinforcement learning (RL) is an attractive alternative since its real-time control computation is much simpler. But designing an RL controller is challenging due to myriad design choices and computationally intensive training. In this paper, we propose an RL controller and an MPC controller for minimizing the electricity cost of a DCEP and compare them via simulations. The two controllers are designed to be comparable in terms of objective and information requirements. The RL controller uses a novel Q-learning algorithm that is based on least-squares policy iteration. We describe the design choices for the RL controller, including the choice of state space and basis functions, that are found to be effective. The proposed MPC controller does not need a mixed integer solver for implementation, but only a nonlinear program (NLP) solver. A rule-based baseline controller is also proposed to aid in comparison. Simulation results show that the proposed RL and MPC controllers achieve similar savings over the baseline controller, about 17%.

An efficient conjugate gradient based Cholesky CMA‐ES estimation algorithm for nonlinear systems

AbstractThis article studies the parameter estimation problems of nonlinear systems with colored noise using the covariance matrix adaptation evolution strategy (CMA‐ES), which is one of the most competitive evolutionary algorithms available and has been applied in the area of reinforcement learning and process control. However, a major limitation that impedes the application of the CMA‐ES is the high computational complexity caused by matrix decomposition. To solve this problem, an efficient Cholesky CMA‐ES which uses the Cholesky factor instead of the covariance matrix to reduce the computational complexity, and updates the search direction and distribution mean based on the conjugate gradient method to improve the search accuracy is proposed. By using the auxiliary model identification idea, the Cholesky CMA‐ES can be applied to solve the parameter estimation problems of the Hammerstein nonlinear systems with colored noise. Two simulation examples are provided to demonstrate its effectiveness.

Reinforcement Learning versus Model Predictive Control on greenhouse climate control

The greenhouse system plays a crucial role to ensure an adequate supply of fresh food for the growing global population. However, maintaining an optimal growing climate within a greenhouse requires resources and operational costs. To achieve economical and sustainable crop growth, efficient climate control in greenhouse production is paramount. Model Predictive Control (MPC) and Reinforcement Learning (RL) are the two approaches representing model-based and learning-based control, respectively. Each one has its own way to formulate control problems, define control objectives, and seek for optimal control actions that provide sustainable crop growth. Although certain forms of MPC and RL have been applied to greenhouse climate control, limited research has comprehensively analyzed the connections, differences, advantages, and disadvantages between these two approaches, both mathematically and in terms of performance. Therefore, this paper aims to address this gap by: (1) introducing a novel RL approach that utilizes Deep Deterministic Policy Gradient (DDPG) for large and continuous state–action space environments; (2) formulating the MPC and RL approaches for greenhouse climate control within a unified framework; (3) exploring the mathematical connections and differences between MPC and RL; (4) conducting a simulation study to analyze and compare the performance of MPC and RL; (5) presenting and interpreting the comparative results to provide valuable insights for the application of these control approaches in different scenarios. By undertaking these objectives, this paper seeks to contribute to the understanding and advancement of both MPC and RL methods in greenhouse climate control, fostering more informed decision-making regarding their selection and implementation based on specific requirements and constraints.

Learning a World Model With Multitimescale Memory Augmentation.

Model-based reinforcement learning (RL) is regarded as a promising approach to tackle the challenges that hinder model-free RL. The success of model-based RL hinges critically on the quality of the predicted dynamic models. However, for many real-world tasks involving high-dimensional state spaces, current dynamics prediction models show poor performance in long-term prediction. To that end, we propose a novel two-branch neural network architecture with multi-timescale memory augmentation to handle long-term and short-term memory differently. Specifically, we follow previous works to introduce a recurrent neural network architecture to encode history observation sequences into latent space, characterizing the long-term memory of agents. Different from previous works, we view the most recent observations as the short-term memory of agents and employ them to directly reconstruct the next frame to avoid compounding error. This is achieved by introducing a self-supervised optical flow prediction structure to model the action-conditional feature transformation at pixel level. The reconstructed observation is finally augmented by the long-term memory to ensure semantic consistency. Experimental results show that our approach is able to generate visually-realistic long-term predictions in DeepMind maze navigation games, and outperforms the prevalent state-of-the-art methods in prediction accuracy by a large margin. Furthermore, we also evaluate the usefulness of our world model by using the predicted frames to drive an imagination-augmented exploration strategy to improve the model-free RL controller.