Model-free Reinforcement Learning Approach Research Articles

Water management scheme based on prioritized deep deterministic policy gradient for proton exchange membrane fuel cells

AbstractTo effectively tackle the intricate and dynamic challenges encountered in proton exchange membrane fuel cells (PEMFCs), this paper introduces a model-free reinforcement learning approach to address its water management issue. Recognizing the limitations of conventional reinforcement learning methods such as Q-learning in handling the continuous actions and nonlinearity inherent in PEMFCs water management, we propose a prioritized deep deterministic policy gradient (DDPG) method. This method, rooted in the Actor-Critic framework, leverages double neural networks and prioritized experience replay to enable adaptive water management and balance. Additionally, we establish a PEMFCs water management platform and implement the prioritized DDPG method using "Tianshou", a modularized Python library for deep reinforcement learning. Through experimentation, the effectiveness of our proposed method is verified. This study contributes to advancing the understanding and management of water dynamics in PEMFCs, offering a promising avenue for enhancing their performance and reliability.

Reinforcement learning to achieve real-time control of triple inverted pendulum

This work uses reinforcement learning (RL) to achieve the first-ever data-driven real-time control of an actual, not simulated, triple inverted pendulum (TIP) in a model-free way. A swing-up control task for the TIP is formulated as a Markov decision process with a dense reward function, then conducted in real time by using a model-free RL approach. To increase the sample efficiency of learning, a structure-aware virtual experience replay (VER) method is proposed; it works together with an off-policy actor-critic algorithm. The VER exploits the geometrically-symmetric property of TIPs to create virtual sample trajectories from measured ones, then uses the resulting multifold augmented dataset to effectively train actor and critic networks during the learning process. These structure-infused training data serve to obtain additional information and hence increase the convergence speed of network learning. We combine the proposed VER with a state-of-the-art actor-critic algorithm, and then validate its effectiveness through numerical simulations. Notably, the inclusion of VER amplifies computational efficiency, slashing the requisite trials, training steps, and overall duration by approximately 66.67%. Finally experiments demonstrate the real-time control capability of the proposed approach on an actual TIP system.

Data Center HVAC Control Harnessing Flexibility Potential via Real-Time Pricing Cost Optimization Using Reinforcement Learning

With increasing electricity prices, cost savings through load shifting are becoming increasingly important for energy end users. While dynamic pricing encourages customers to shift demand to low price periods, the non-stationary and highly volatile nature of electricity prices poses a significant challenge to energy management systems. In this paper, we investigate the flexibility potential of data centres by optimising heating, ventilation, and air conditioning systems with a general model-free reinforcement learning approach. Since the soft actor-critic algorithm with feed-forward networks did not work satisfactorily in this scenario, we propose instead a parameterisation with a recurrent neural network architecture to successfully handle spot-market price data. The past is encoded into a hidden state, which provides a way to learn the temporal dependencies in the observations and highly volatile rewards. The proposed method is then evaluated in experiments on a simulated data centre. Considering real temperature and price signals over multiple years, the results show a cost reduction compared to a proportional, integral and derivative controller while maintaining the temperature of the data centre within the desired operating ranges. In this context, this work demonstrates an innovative and applicable reinforcement learning approach that incorporates complex economic objectives into agent decision-making. The proposed control method can be integrated into various Internet of things-based smart building solutions for energy management.

Uncertainty-Aware Model-Based Reinforcement Learning: Methodology and Application in Autonomous Driving

To further improve learning efficiency and performance of reinforcement learning (RL), a novel uncertainty-aware model-based RL method is proposed and validated in autonomous driving scenarios in this paper. First, an action-conditioned ensemble model with the capability of uncertainty assessment is established as the environment model. Then, a novel uncertainty-aware model-based RL method is developed based on the adaptive truncation approach, providing virtual interactions between the agent and environment model, and improving RL’s learning efficiency and performance. The proposed method is then implemented in end-to-end autonomous vehicle control tasks, validated and compared with state-of-the-art methods under various driving scenarios. Validation results suggest that the proposed method outperforms the model-free RL approach with respect to learning efficiency, and model-based approach with respect to both efficiency and performance, demonstrating its feasibility and effectiveness.

Bayesian Actor-Critic Wave Energy Converter Control With Modeling Errors

This paper presents a comparison of a Reinforcement-Learning (RL) based wave energy conversion controller against standard reactive damping and model predictive control (MPC) approaches, in the presence of modeling errors. Wave energy converters (WECs) are under the influence of many non-linear hydrodynamic forces, yet for ease and expediency, it is common to formulate linear WEC models and control laws. Therefore it is expected that significant modeling errors may be present, which may degrade model-based control performance. Model-free RL approaches to control may offer a significant advantage in robustness to modeling errors, in that the model is learned by the controller by experience. It is shown that, for an annual average sea state, RL-based controllers can outperform model-based control – reactive control and MPC – by 19% and 16%, respectively, when significant modeling error is present. Furthermore, compared to similar studies of RL-based control, the proposed model can reduce the training time from 8.4 hr to 1.5 hr.

FedDOVe: A Federated Deep Q-learning-based Offloading for Vehicular fog computing

Connected Autonomous Vehicles (CAVs) aim to provide various smart transportation applications which have computation-intensive tasks. The vehicles having less availability of computational resources offload tasks to roadside units (RSUs) for processing. Renewable energy-powered RSUs have limited energy, storage, and computational capabilities. To reduce the computation load at RSUs, vehicular fog computing is used for computation-intensive tasks. Vehicular fog computing brings the computational resources nearer to the requesting devices for reducing the latency in the request fulfillment. RSU computes the tasks either locally or offloads computation-intensive tasks to the fog server for further processing. In vehicular fog computing, one of the most important problems is to find an optimal association of RSUs with fog servers for computation offloading. A significant energy consumption occurs across RSUs in offloading computation-intensive tasks to fog servers. On the other hand, fog servers may have uneven computation load due to varying offloading rate across the road segments. The non-uniform distribution of computation load across fog servers may increase the latency in the request fulfillment. In this paper, we propose an online Federated Deep Q-learning-based Offloading technique for Vehicular fog computing (FedDOVe) which jointly optimizes the energy consumption across RSUs and load balancing across fog servers. FedDOVe is a model-free reinforcement learning approach that uses the global information for finding an optimal association between RSUs and fog servers. Simulation results show that the proposed FedDOVe reduces the energy consumption by about 47%–49% and improves load balancing by about 60%–65% as compared to other existing offloading techniques.

Online model-based reinforcement learning for decision-making in long distance routes

In road transportation, long-distance routes require scheduled driving times, breaks, and rest periods, in compliance with the regulations on working conditions for truck drivers, while ensuring goods are delivered within the time windows of each customer. However, routes are subject to uncertain travel and service times, and incidents may cause additional delays, making predefined schedules ineffective in many real-life situations. This paper presents a reinforcement learning (RL) algorithm capable of making en-route decisions regarding driving times, breaks, and rest periods, under uncertain conditions. Our proposal aims at maximizing the likelihood of on-time delivery while complying with drivers’ work regulations. We use an online model-based RL strategy that needs no prior training and is more flexible than model-free RL approaches, where the agent must be trained offline before making online decisions. Our proposal combines model predictive control with a rollout strategy and Monte Carlo tree search. At each decision stage, our algorithm anticipates the consequences of all the possible decisions in a number of future stages (the lookahead horizon), and then uses a base policy to generate a sequence of decisions beyond the lookahead horizon. This base policy could be, for example, a set of decision rules based on the experience and expertise of the transportation company covering the routes. Our numerical results show that the policy obtained using our algorithm outperforms not only the base policy (up to 83%), but also a policy obtained offline using deep Q networks (DQN), a state-of-the-art, model-free RL algorithm.

Energy saving evaluation of an energy efficient data center using a model-free reinforcement learning approach

To reduce cooling energy consumption, data centers are recommended to raise temperature setpoints of server intake. However, in tropical climates, Data Center operators are still found to be operating at lower temperatures. In this paper, we demonstrate that using a floating setpoint with a lowered temperature value for tropical climates reduces the overall energy consumption of Data Centers as opposed to raising the temperature in a static manner. We achieve this by applying a deep reinforcement learning algorithm to a hybrid data center model that was built from data collected off a highly efficient data center. This generates an optimal control strategy which minimizes the costs of energy consumption while operating within the required set of operational constraints. Following which, we evaluate the behavior of the control strategy to account for the exact sources of energy savings. The deep reinforcement learning algorithm learns by continually interacting with the built Data Center model without any prior knowledge of the Data Center. The algorithm is trained under the full-load and the part-load configuration of the Data Center. Testing results show that further energy savings of up to 3% and 5.5% (under full load and part load respectively) can be achieved with targeted cooling provisioning while operating within constraints in an already cooling-efficient Data Center. We find that while building level optimization studies of Data Centers generally improve energy efficiency, the source of energy savings is not well accounted for. Consequently, our studies show that the reduction of server fan usage and not the reduction of cooling energy consumption is the main contributor of energy savings in a deep reinforcement learning-driven data center operating in the tropics.

Autonomous control of unmanned aerial vehicle for chemical detection using deep reinforcement learning

The unmanned aerial vehicle (UAV) is a promising platform for remote chemical sensing to minimize human contact with toxic chemicals. Most previous studies in this field used predefined paths to search for areas based on sensor measurements at fixed points. However, operations on a predefined flight path are inefficient because in real-life scenarios, gas dispersion is stochastic and unpredictable. Thus, a model-free reinforcement-learning approach using a deep Q-network for autonomous UAV control is proposed. The UAV automatically selects its trajectory based on sensor measurements and GPS location data to map the contaminated area in real time. Path planning is simulated using Gaussian model-based gas dispersion data and its feasibility in outdoor experiments using gas simulants is proved. This study provides guidelines for developing a model that can perform autonomous chemical detection and source localization using UAV.

Comparison of Optimal Control Techniques for Building Energy Management

Optimal controllers can enhance buildings’ energy efficiency by taking forecast and uncertainties into account (e.g., weather and occupancy). This practice results in energy savings by making better use of energy systems within the buildings. Even though the benefits of advanced optimal controllers have been demonstrated in several research studies and some demonstration cases, the adoption of these techniques in the built environment remains somewhat limited. One of the main reasons is that these novel control algorithms continue to be evaluated individually. This hampers the identification of best practices to deploy optimal control widely in the building sector. This paper implements and compares variations of model predictive control (MPC), reinforcement learning (RL), and reinforced model predictive control (RL-MPC) in the same optimal control problem for building energy management. Particularly, variations of the controllers’ hyperparameters like the control step, the prediction horizon, the state-action spaces, the learning algorithm, or the network architecture of the value function are investigated. The building optimization testing (BOPTEST) framework is used as the simulation benchmark to carry out the study as it offers standardized testing scenarios. The results reveal that, contrary to what is stated in previous literature, model-free RL approaches poorly perform when tested in building environments with realistic system dynamics. Even when a model is available and simulation-based RL can be implemented, MPC outperforms RL for an equivalent formulation of the optimal control problem. The performance gap between both controllers reduces when using the RL-MPC algorithm that merges elements from both families of methods.

A Model-Free Approach to Intrusion Response Systems

With the rising number of data breaches, denial of service attacks and general malicious activity facing modern computer networks, there is an increasing need to quickly and effectively respond to attacks. Intrusion Detection Systems provide an automated method of identifying malicious activity within a network however the development of an Intrusion Response System which can automatically respond to these alerts is non-trivial. Current research in IRS proposes model-based methods for identifying possible routes a malicious actor may take when attacking a network and use subjective performance values for the cost and benefit of a response, both of which can be invalidated by the increasingly dynamic nature of network topologies and system configurations. The IRS proposed in this work utilises a Model-free Reinforcement Learning approach and evaluates the Reinforcement Learning agent's performance in stopping two distinct multi-stage attack scenarios on a virtualised testbed. Experimentation demonstrates that the agent can successfully halt both attack scenarios and find responses which have minimal impact on normal network operation based on experience gained through training. A further contribution is the novel use of a virtualised environment that demonstrates Intrusion Response Reinforcement Learning performance in a more realistic environment than simulated tasks common to previous literature.

An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach

Occupants’ behavior is a major source of uncertainty for the optimal operation of building energy systems. The highly stochastic hot water use behavior of occupants has led to conservative operational strategies for hot water systems, that try to ensure occupants’ comfort by following energy-intensive operational approaches. Intending to integrate the occupants’ behavior into hot water systems control, this study proposes a control framework based on Reinforcement Learning, which can learn the stochastic occupants' behavior and make a balance between opposing objectives of water hygiene, comfort, and energy use. A model-free approach is implemented to ensure transferability. To achieve fast convergence on the target house while being model-free, this study proposes an offline training procedure integrating a stochastic hot water use model to mimic the hot water use behavior of occupants. The proposed framework is then evaluated using an actual hot water use and weather dataset collected over 29 weeks from a residential house in Switzerland. The performance of the proposed control framework is compared to the conventional rule-based controller as the common practice in hot water systems. While the hot water use dataset has been collected during the COVID-19 pandemic, with an abnormal schedule of occupants, results indicate that the proposed control framework could successfully learn and adapt to the occupants' behavior and achieve 23.8% of energy saving, while maintaining the occupants' comfort and water hygiene. The adaptive nature of the proposed control framework shows significant potential in reducing the discrepancy between supply and demand in hot water systems.

Smart closed-loop control of laser welding using reinforcement learning

The present work demonstrates an adaptive closed-loop control for laser welding processes. Based on feedback signals from a sensing system, the controller interacts with the laser to adapt the processing parameters to achieve or maintain the target welding quality. The controller is constructed based on a model-free reinforcement learning approach, namely Q-learning. This algorithm allows autonomous learning of the control law independently from the starting conditions as well as any prior knowledge of the process dynamics. The controller's performance is demonstrated in both a well-controlled lab environment and more unpredictable industrial situations. For the demonstration, the control system is allowed to vary the laser power, and the feedback signal is given by an industrial laser process control unit (Coherent SmartSense+) using an optical sensor. The time needed to train the control system is approximately five and twenty minutes for the well-controlled and the industrial situations, respectively.

Deductive Reinforcement Learning for Visual Autonomous Urban Driving Navigation

Existing deep reinforcement learning (RL) are devoted to research applications on video games, e.g., The Open Racing Car Simulator (TORCS) and Atari games. However, it remains under-explored for vision-based autonomous urban driving navigation (VB-AUDN). VB-AUDN requires a sophisticated agent working safely in structured, changing, and unpredictable environments; otherwise, inappropriate operations may lead to irreversible or catastrophic damages. In this work, we propose a deductive RL (DeRL) to address this challenge. A deduction reasoner (DR) is introduced to endow the agent with ability to foresee the future and to promote policy learning. Specifically, DR first predicts future transitions through a parameterized environment model. Then, DR conducts self-assessment at the predicted trajectory to perceive the consequences of current policy resulting in a more reliable decision-making process. Additionally, a semantic encoder module (SEM) is designed to extract compact driving representation from the raw images, which is robust to the changes of the environment. Extensive experimental results demonstrate that DeRL outperforms the state-of-the-art model-free RL approaches on the public CAR Learning to Act (CARLA) benchmark and presents a superior performance on success rate and driving safety for goal-directed navigation.

Routing of Electric Vehicles With Intermediary Charging Stations: A Reinforcement Learning Approach.

In the past few years, the importance of electric mobility has increased in response to growing concerns about climate change. However, limited cruising range and sparse charging infrastructure could restrain a massive deployment of electric vehicles (EVs). To mitigate the problem, the need for optimal route planning algorithms emerged. In this paper, we propose a mathematical formulation of the EV-specific routing problem in a graph-theoretical context, which incorporates the ability of EVs to recuperate energy. Furthermore, we consider a possibility to recharge on the way using intermediary charging stations. As a possible solution method, we present an off-policy model-free reinforcement learning approach that aims to generate energy feasible paths for EV from source to target. The algorithm was implemented and tested on a case study of a road network in Switzerland. The training procedure requires low computing and memory demands and is suitable for online applications. The results achieved demonstrate the algorithm’s capability to take recharging decisions and produce desired energy feasible paths.

Model-free reinforcement learning approach to optimal speed control of combustion engines in start-up mode

This paper presents a model-free reinforcement learning approach for optimal speed control of gasoline engines. First, the physics of the controlled internal combustion engines are discussed to show the uncertainty and the complexity in the model of the dynamics during start-up operation mode, which is the main motivation for challenging learning-based design. Then, a learning algorithm, particularly focused on the continuous time nonlinear dynamics, is constructed to avoid the use of the probing noise usually required in the existing learning algorithms. With the constructed learning algorithm, a learning-based control scheme is designed to solve the optimal speed control problem of a production gasoline engine. Finally, experiments are conducted on a full-scale test bench with a 4-cylinder gasoline engine used for the production of hybrid electric vehicles, and simulation and experimental validation are demonstrated.

Long-Term Energy Collection in Self-Sustainable Sensor Networks: A Deep Q-Learning Approach

This article reports on the development of a deep Q-learning approach to long-term energy collection in self-sustainable sensor networks, which consists of two static chargers (SCs) and one rechargeable mobile sensor (MS). In particular, we assume that the SCs can harvest energy from the ambient environment and charge the MS via electromagnetic (EM) radiation. As the energy harvesting (EH) process is random and the radiated energy fades over distance, the achievable energy by the MS at each slot demonstrates spatiotemporal dynamics in a certain area. Thus, we focus on the problem of trajectory optimization for an autonomous MS to maximize the long-term average achievable energy per slot from both chargers. Due to the inaccessible charger-side information, such as the EH profile and locations of SCs as well as the transmit power, we introduce deep Q-learning, a model-free reinforcement learning approach, based on which the MS can learn and optimize the moving trajectory by intelligently tracking the aggregated received EM signal without any other explicit external information. Simulation results show that the MS can identify the best energy collecting location and finally moves there along the learned trajectory. We also investigate the impact of system parameters, such as initial position and moving cost per unit distance on the performance of the proposed training algorithm, such as convergence rate and stability via extensive numerical evaluations.

Investigating exploration for deep reinforcement learning of concentric tube robot control

PurposeConcentric tube robots are composed of multiple concentric, pre-curved, super-elastic, telescopic tubes that are compliant and have a small diameter suitable for interventions that must be minimally invasive like fetal surgery. Combinations of rotation and extension of the tubes can alter the robot’s shape but the inverse kinematics are complex to model due to the challenge of incorporating friction and other tube interactions or manufacturing imperfections. We propose a model-free reinforcement learning approach to form the inverse kinematics solution and directly obtain a control policy.MethodThree exploration strategies are shown for deep deterministic policy gradient with hindsight experience replay for concentric tube robots in simulation environments. The aim is to overcome the joint to Cartesian sampling bias and be scalable with the number of robotic tubes. To compare strategies, evaluation of the trained policy network to selected Cartesian goals and associated errors are analyzed. The learned control policy is demonstrated with trajectory following tasks.ResultsSeparation of extension and rotation joints for Gaussian exploration is required to overcome Cartesian sampling bias. Parameter noise and Ornstein–Uhlenbeck were found to be optimal strategies with less than 1 mm error in all simulation environments. Various trajectories can be followed with the optimal exploration strategy learned policy at high joint extension values. Our inverse kinematics solver in evaluation has 0.44 mm extension and 0.3^{circ } rotation error.ConclusionWe demonstrate the feasibility of effective model-free control for concentric tube robots. Directly using the control policy, arbitrary trajectories can be followed and this is an important step towards overcoming the challenge of concentric tube robot control for clinical use in minimally invasive interventions.

Model-free reinforcement learning with model-based safe exploration: Optimizing adaptive recovery process of infrastructure systems

Extreme events represent not only some of the most damaging events in our society and environment, but also the most difficult to predict. Model-based predictions of the disruptions induced by extreme events on urban infrastructure systems are often unreliable, as these events are unlikely by their very definition. Specifically, characterizing the effect of such disruptions to the urban infrastructure using a parameterized model is a difficult task. On the other hand, model-free approaches based on recent advancements in reinforcement learning can model the complex dynamics of urban society and infrastructure under the risk of extreme events explicitly without relying on any specific physics-based mechanism. However, these approaches usually require performing random exploration of the effects of management actions on the system (typically in the post-event situation) to allow for an acceptable approximation to the optimal management policy. When dealing with costly infrastructure systems and important communities, this random exploration can be unacceptable and risky. In this paper, we propose a method called Safe Q-learning, which is a model-free reinforcement learning approach with addition of a model-based safe exploration for near-optimal management of infrastructure system pre-event and their recovery post-event. Our method requires the decision-maker to model the structure of the state space of the problem, and a suitable equilibrium of the system (optimum functionality pre-event). This information is usually available for urban systems, as they spend long time in optimum equilibrium before the occurrence of such events. We show on several examples of infrastructure management how the proposed approach is able to achieve near-optimal performance without the risk due to random exploration.

User Scheduling and Resource Allocation in HetNets With Hybrid Energy Supply: An Actor-Critic Reinforcement Learning Approach

Densely deployment of various small-cell base stations in cellular networks to increase capacity will lead to heterogeneous networks (HetNets), and meanwhile, embedding the energy harvesting capabilities in base stations as an alternative energy supply is becoming a reality. How to make efficient utilization of radio resource and renewable energy is a brand-new challenge. This paper investigates the optimal policy for user scheduling and resource allocation in HetNets powered by hybrid energy with the purpose of maximizing energy efficiency of the overall network. Since wireless channel conditions and renewable energy arrival rates have stochastic properties and the environment’s dynamics are unknown, the model-free reinforcement learning approach is used to learn the optimal policy through interactions with the environment. To solve our problem with continuous-valued state and action variables, a policy-gradient-based actor-critic algorithm is proposed. The actor part uses the Gaussian distribution as the parameterized policy to generate continuous stochastic actions, and the policy parameters are updated with the gradient ascent method. The critic part uses compatible function approximation to estimate the performance of the policy and helps the actor learn the gradient of the policy. The advantage function is used to further reduce the variance of the policy gradient. Using the numerical simulations, we demonstrate the convergence property of the proposed algorithm and analyze network energy efficiency.