Model-free Reinforcement Learning Method Research Articles

Learning optimal edge processing with offloading and energy harvesting

Modern portable devices can execute increasingly sophisticated AI models on sensed data. The complexity of such processing tasks is data-dependent and has relevant energy cost. This work develops an Age of Information markovian model for a system where multiple battery-operated devices perform data processing and energy harvesting in parallel. Part of their computational burden is offloaded to an edge server which polls devices at given rate. The structural properties of an optimal policy for a single device-server system are derived. They permit to define a new model-free reinforcement learning method specialized for monotone policies, namely Ordered Q-Learning, providing a fast procedure to learn the optimal policy. The method is oblivious to the devices’ battery capacities, the cost and the value of data batch processing and to the dynamics of the energy harvesting process. Finally, the polling strategy of the server is optimized by combining this policy improvement technique with stochastic approximation methods. Extensive numerical results provide insight into the system properties and demonstrate that the proposed learning algorithms outperform existing baselines.

Controlling estimation error in reinforcement learning via Reinforced Operation

In Value-based and Actor-Critic reinforcement learning (RL) methods, both inaccuracy and instability of value estimation will detrimentally affect their performance. Some typical RL methods, such as Maxmin Q-learning and QMD3, are plagued by the underestimation problem while failing to trade off the estimation bias and variance jointly. We propose the Reinforced Operation (RO) to address these shortcomings, which selects the closest median among multiple Q-function. RO is applicable to any model-free RL method. Theoretically, we introduce the Mean Square Error (MSE) to jointly analyze the estimation bias and variance of value estimation methods. We also demonstrate the superiority of RO in MSE reduction and give an upper bound for the estimation bias of value estimation methods with arbitrary distribution to guide the calculation of the estimation bias of value estimation methods. Based on RO, we propose the variants of Q-learning and TD3, Reinforced Q-learning (RQ) and Reinforced Delayed Deep Deterministic policy gradient (RD3), respectively, to tackle different tasks. We empirically demonstrate that our method can reduce estimation error and achieve superior performance on discrete and continuous benchmark tasks.

A homotopy-based reinforcement learning scheme to optimal control for Markov switched interconnected systems

ABSTRACT In this paper, a homotopy-based reinforcement learning optimal control method is developed for Markov switched interconnected systems with unknown system dynamics. By utilising the subsystem decomposition method and parallel learning control method, the solution of the game coupled algebraic Riccati equations with jumping parameters is approximated. To dispense with the requirement of initial stability, a homotopy-based policy iteration is introduced, which can place unstable poles into a stable plane. In this regard, a model-free reinforcement learning method is presented to design the optimal controller for Markov switched interconnected systems. Finally, the effectiveness of the proposed method is verified by a numerical example and a practical example.

Adaptive Model-Based Reinforcement Learning for Fast-Charging Optimization of Lithium-Ion Batteries

Fast charging problem of lithium-ion batteries with minimum-charging time while limiting battery degradation, has been receiving increasing attention and is a critical challenge to battery community. Difficulties in this optimization lie in that: (i) The parameter space of charging strategies is high dimensional while the budget of the experimental cost is often limited; (ii) The evaluation of charging strategies' performance is expensive, and (iii) the degradation process of battery is strongly nonlinear and multiple degradation mechanisms occur simultaneously leading to difficulties for establishing accurate first-principles models. Current methods to address these difficulties are mainly electrochemical model-based optimization and grid search, which are rarely adaptive to battery degradation and/or are of low sample efficiency. In this work, we propose an adaptive model-based reinforcement learning (RL) approach for fast charging optimization while limiting battery degradation, in which a probabilistic surrogate model of differential Gaussian process (GP) is adopted to adaptively describe the degradation of cells. The effectiveness of the proposed approach is demonstrated on PETLION, a high-performance PET-based battery simulator. The results show that (i) compared with the model-free RL method, the proposed adaptive GP-based RL approach possesses superior charging performance and high sample efficiency, and (ii) the proposed method performs well in the handling of degradation constraints on voltage and temperature for dynamically aging batteries with its adaptability to the variations of environment.

Pontryagin's Minimum Principle-Guided RL for Minimum-Time Exploration of Spatiotemporal Fields.

This article studies the trajectory planning problem of an autonomous vehicle for exploring a spatiotemporal field subject to a constraint on cumulative information. Since the resulting problem depends on the signal strength distribution of the field, which is unknown in practice, we advocate the use of a model-free reinforcement learning (RL) method to find the solution. Given the vehicle's dynamical model, a critical (and open) question is how to judiciously merge the model-based optimality conditions into the model-free RL framework for improved efficiency and generalization, for which this work provides some positive results. Specifically, we discretize the continuous action space by leveraging analytic optimality conditions for the minimum-time optimization problem via Pontryagin's minimum principle (PMP). This allows us to develop a novel discrete PMP-based RL trajectory planning algorithm, which learns a planning policy faster than those based on a continuous action space. Simulation results: 1) validate the effectiveness of the PMP-based RL algorithm and 2) demonstrate its advantages, in terms of both learning efficiency and the vehicle's exploration time, over two baseline methods for continuous control inputs.

Tracking control of AUV via novel soft actor-critic and suboptimal demonstrations

Tracking control for autonomous underwater vehicles (AUVs) faces multifaceted challenges, making the acquisition of optimal demonstrations a daunting task. The suboptimal demonstrations mean less tracking accuracy. To address the issue of learning from suboptimal demonstrations, this paper proposes a model-free reinforcement learning (RL) method. Our approach utilizes suboptimal demonstrations to obtain an initial controller, which is iteratively refined during training. Given the suboptimal characteristics, demonstrations will be removed from the replay buffer upon reaching capacity. Building upon the soft actor-critic (SAC), our approach integrates a Recurrent Neural Network (RNN) into the policy network to capture the relationship between states and actions. Moreover, we introduce logarithmic and cosine functions to the reward function for enhancing the training effectiveness. Finally, we validate the effectiveness of the proposed Initialize Controller from Demonstrations (ICfD) algorithm through simulations with two reference trajectories. We provide a definition for tracking success. The success rates of ICfD in the two reference trajectories are 95.60% and 94.05%, respectively, surpassing the state-of-the-art RL method SACfD (80.03% 90.55%). The average one-step distance errors of ICfD are 1.20 m and 0.76 m, respectively, significantly lower than the S-plane controller (9.725 m 8.325 m). Besides, we evaluate the generalization of the ICfD controller in different scenarios.

Improved reinforcement learning-based real-time energy scheduling for prosumer with elastic loads in smart grid

In this study, we investigate the dynamic energy-scheduling problem of a prosumer (producer/consumer) with an energy storage device (EDS) and elastic loads. The goal is to develop efficient real-time scheduling strategies for prosumers, and to minimise their total long-term costs (i.e. cost of energy purchased from the external grid and depreciation cost of the EDS). The challenges are twofold: the uncertainty of the energy output of the prosumer and the time-coupling constraints of the EDS and elastic loads. To address these challenges, we first describe the dynamic energy-scheduling problem as a Markov decision process. Then, an approximate state dual-agent Q-learning algorithm is proposed to solve the optimal dynamic scheduling problem by improving the model-free reinforcement learning(RL) method. Compared with the traditional RL method, the proposed algorithm reduces the system-state dimensions and exhibits improved performance. The proposed algorithm can only be assisted by mutual interactions between the environment with a reward feedback mechanism to dynamically respond to uncertain changes in the environments, without modelling or predicting the system environments. Finally, extensive empirical evaluations using real-world traces are conducted to study the effectiveness of the proposed algorithm. The results show that the proposed algorithm can reduce the total cost of the prosumer by up to 6.3%, 11.7% and 22.4% compared with the traditional RL method, Lyapunov optimisation and greedy algorithm, respectively.

Automated Adversary-in-the-Loop Cyber-Physical Defense Planning

Security of cyber-physical systems (CPS) continues to pose new challenges due to the tight integration and operational complexity of the cyber and physical components. To address these challenges, this article presents a domain-aware, optimization-based approach to determine an effective defense strategy for CPS in an automated fashion—by emulating a strategic adversary in the loop that exploits system vulnerabilities, interconnection of the CPS, and the dynamics of the physical components. Our approach builds on an adversarial decision-making model based on a Markov Decision Process (MDP) that determines the optimal cyber (discrete) and physical (continuous) attack actions over a CPS attack graph. The defense planning problem is modeled as a non-zero-sum game between the adversary and defender. We use a model-free reinforcement learning method to solve the adversary’s problem as a function of the defense strategy. We then employ Bayesian optimization (BO) to find an approximate best-response for the defender to harden the network against the resulting adversary policy. This process is iterated multiple times to improve the strategy for both players. We demonstrate the effectiveness of our approach on a ransomware-inspired graph with a smart building system as the physical process. Numerical studies show that our method converges to a Nash equilibrium for various defender-specific costs of network hardening.

Improving Workers' Musculoskeletal Health During Human-Robot Collaboration Through Reinforcement Learning.

This study aims to improve workers' postures and thus reduce the risk of musculoskeletal disorders in human-robot collaboration by developing a novel model-free reinforcement learning method. Human-robot collaboration has been a flourishing work configuration in recent years. Yet, it could lead to work-related musculoskeletal disorders if the collaborative tasks result in awkward postures for workers. The proposed approach follows two steps: first, a 3D human skeleton reconstruction method was adopted to calculate workers' continuous awkward posture (CAP) score; second, an online gradient-based reinforcement learning algorithm was designed to dynamically improve workers' CAP score by adjusting the positions and orientations of the robot end effector. In an empirical experiment, the proposed approach can significantly improve the CAP scores of the participants during a human-robot collaboration task when compared with the scenarios where robot and participants worked together at a fixed position or at the individual elbow height. The questionnaire outcomes also showed that the working posture resulted from the proposed approach was preferred by the participants. The proposed model-free reinforcement learning method can learn the optimal worker postures without the need for specific biomechanical models. The data-driven nature of this method can make it adaptive to provide personalized optimal work posture. The proposed method can be applied to improve the occupational safety in robot-implemented factories. Specifically, the personalized robot working positions and orientations can proactively reduce exposure to awkward postures that increase the risk of musculoskeletal disorders. The algorithm can also reactively protect workers by reducing the workload in specific joints.

Crowd-Aware Mobile Robot Navigation Based on Improved Decentralized Structured RNN via Deep Reinforcement Learning.

Efficient navigation in a socially compliant manner is an important and challenging task for robots working in dynamic dense crowd environments. With the development of artificial intelligence, deep reinforcement learning techniques have been widely used in the robot navigation. Previous model-free reinforcement learning methods only considered the interactions between robot and humans, not the interactions between humans and humans. To improve this, we propose a decentralized structured RNN network with coarse-grained local maps (LM-SRNN). It is capable of modeling not only Robot-Human interactions through spatio-temporal graphs, but also Human-Human interactions through coarse-grained local maps. Our model captures current crowd interactions and also records past interactions, which enables robots to plan safer paths. Experimental results show that our model is able to navigate efficiently in dense crowd environments, outperforming state-of-the-art methods.

Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach

The large-scale integration of distributed energy resources into the energy industry enables the fast transition to a decarbonized future but raises some potential challenges of insecure and unreliable operations. Multi-energy Microgrids (MEMGs), as localized small multi-energy systems, can effectively integrate a variety of energy components with multiple energy sectors, which have been recently recognized as a valid solution to improve the operational security and reliability. As a result, a massive amount of research has been conducted to investigate MEMG energy management problems, including both model-based optimization and model-free learning approaches. Compared to optimization approaches, reinforcement learning is being widely deployed in MEMG energy management problems owing to its ability to handle highly dynamic and stochastic processes without knowing any system knowledge. However, it is still difficult for conventional model-free reinforcement learning methods to capture the physical constraints of the MEMG model, which may therefore destroy its secure operation. To address this research challenge, this paper proposes a novel safe reinforcement learning method by learning a dynamic security assessment rule to abstract a physical-informed safety layer on top of the conventional model-free reinforcement learning energy management policy, which can respect all the physical constraints through mathematically solving an action correction formulation. In this setting, the secure energy management of the MEMG can be guaranteed for both training and test procedures. Extensive case studies based on two integrated systems (i.e., a small 6-bus power and 7-node gas network, and a large 33-bus power and 20-node gas network) are carried out to verify the superior performance of the proposed physical-informed reinforcement learning method in achieving a cost-effective MEMG energy management performance while respecting all the physical constraints, compared to conventional reinforcement learning and optimization approaches.

Negotiation agent based on Deep reinforcement learning for multi-agent cooperative distributed predictive control

In this work, deep neural network trained by the model-free reinforcement learning method is proposed as a negotiation agent in multi-agent distributive model predictive control. The negotiation agent is implemented in the upper layer of a hierarchical control architecture based on distributed model predictive controller with pairwise negotiation fuzzy logic. The proposed is data-driven in order to achieve the minimization of any dynamic behavior index and the specification of constraints. Specifically, the deep neural network provides consensus coefficients to address the final control action applied by each agent. The proposed is successfully applied to a hydraulic system composed of eight interconnected tanks that are very difficult to control due to non-linearities nature and high interaction between its subsystems.

Model-Free trajectory planning for a rotary-wing unmanned aerial vehicle with an uncertain suspended payload

Payload transport using rotary-wing unmanned aerial vehicles (RUAVs) has grown in popularity. Suspending the payload from the RUAV expands the range of use cases but does so at the expense of changing the dynamics that may result in instability. This paper explores how a model-free solution can be used to plan robust trajectories that account for the added oscillations if the payload is varied. The twin-delayed deep deterministic policy gradient (TD3) algorithm is a model-free reinforcement learning method and is used to train an agent to be implemented as a local planner to plan optimal trajectories that can complete two defined circuits whilst minimizing the swing of the payload. A non-linear model predictive controller (NMPC) is implemented as a model-based solution to evaluate the capabilities of the model-free results determining its viability, without choosing a superior approach. The results indicate that the model-free TD3 agent has comparable results to the model-based NMPC for two defined circuits, with increased robustness to payload uncertainty when trained with different payload parameters.

Model-Based Reinforcement Learning With Kernels for Resource Allocation in RAN Slices

Network slicing is a key feature of 5G and beyond networks, allowing the deployment of separate logical networks (network slices), sharing a common underlying physical infrastructure, and characterized by distinct descriptors and behaviors. The dynamic allocation of physical network resources among coexisting slices should address a challenging trade-off: to use resources efficiently while assigning each slice sufficient resources to meet its service level agreement (SLA). We consider the allocation of time-frequency resources from a new perspective: to design a control algorithm capable of learning over the operating network, while keeping the SLA violation rate under an acceptable level during the learning process. For this purpose, traditional model-free reinforcement learning (RL) methods present several drawbacks: low sample efficiency, extensive exploration of the policy space, and inability to discriminate between conflicting objectives, causing inefficient use of the resources and/or frequent SLA violations during the learning process. To overcome these limitations, we propose a model-based RL approach built upon a novel modeling strategy that comprises a kernel-based classifier and a self-assessment mechanism. In numerical experiments, our proposal, referred to as kernel-based RL, clearly outperforms state-of-the-art RL algorithms in terms of SLA fulfillment, resource efficiency, and computational overhead.

Machine learning for flow-informed aerodynamic control in turbulent wind conditions

Control of aerodynamic forces in gusty, turbulent conditions is critical for the safety and performance of technologies such as unmanned aerial vehicles and wind turbines. The presence and severity of extreme flow conditions are difficult to predict, and explicit modeling of fluid dynamics for control is not feasible in real time. Model-free reinforcement learning methods present an end-to-end control solution for nonlinear systems as they require no prior knowledge, can easily integrate different types of measurements, and can adapt to varying conditions through interaction. Here, we show that reinforcement learning methods can achieve effective aerodynamic control in a highly turbulent environment. Algorithms are trained with different neural network structures, and we find that reinforcement learning agents with recurrent neural networks can effectively learn the nonlinear dynamics involved in turbulent flows and strongly outperform conventional linear control techniques. We also find that augmenting state observations with measurements from a set of bioinspired flow sensors can improve learning stability and control performance in aerodynamic systems. These results can serve to inform future gust mitigation systems for unmanned aerial vehicles and wind turbines, enabling operation in previously prohibitively dangerous conditions.

Towards self-learning control of HVAC systems with the consideration of dynamic occupancy patterns: Application of model-free deep reinforcement learning

This study proposes a self-learning control system that aims to learn occupancy profiles, building energy consumption patterns, and lag-time of the heating, ventilation, and air-conditioning (HVAC) systems. The control system learns by interacting with the environment with no need to develop building models and occupancy prediction models. The controller is developed based on a double deep Q-networks (DDQN) algorithm, as a model-free reinforcement learning method. The system's performance is evaluated and compared with that of a model predictive control (MPC) system under two scenarios of perfect and actual occupancy predictions based on occupancy data collected from 20 residential units. The MPC is assisted by a genetic algorithm and supervised learning models for predicting future occupancy patterns, indoor operative temperature, and building energy consumption. The results show that in the case of using perfect occupancy prediction, the self-learning controller operates almost as well as the MPC while not requiring any models. When occupancy prediction uncertainty is added to the problem, the proposed method outperforms the MPC in terms of thermal comfort by increasing the average temperature deviation and deviation period by 0.24 °C and 7.87%, respectively. However, the DDQN agent causes significant thermal comfort violations during the initial training period. The system causes up to a 2.8% longer deviation period and a 0.32 °C higher average temperature deviation, compared with the performance of the fully-trained system.

Deep reinforcement learning with planning guardrails for building energy demand response

Building energy demand response is projected to be important in decarbonizing energy use. A demand response program that communicates “artificial” hourly price signals to workers as part of a social game has the potential to elicit energy consumption changes that simultaneously reduce energy costs and emissions. The efficacy of such a program depends on the pricing agent’s ability to learn how workers respond to prices and mitigate the risk of high energy costs during this learning process. We assess the value of deep reinforcement learning (RL) for mitigating this risk. Specifically, we explore the value of combining: (i) a model-free RL method that can learn by posting price signals to workers, (ii) a supervisory “planning model” that provides a synthetic learning environment, and (iii) a guardrail method that determines whether a price should be posted to real workers or the planning environment for feedback. In a simulated medium-sized office building, we compare our pricing agent against existing model-free and model-based deep RL agents, and the simpler strategy of passing on the time-of-use price signal to workers. We find that our controller eliminates 175,000 US Dollars in initial investment, decreases by 30% the energy cost, and curbs emissions by 32% compared to energy consumption under the time-of-use rate. In contrast, the model-free and model-based deep RL benchmarks are unable to overcome initial learning costs. Our results bode well for risk-aware deep RL facilitating the deployment of building demand response.

Residual Policy Learning Facilitates Efficient Model-Free Autonomous Racing

Motion planning for autonomous racing is a challenging task due to the safety requirement while driving aggressively. Most previous solutions utilize the prior information or depend on complex dynamics modeling. Classical model-free reinforcement learning methods are based on random sampling, which severely increases the training consumption and undermines the exploration efficiency. In this letter, we propose an efficient residual policy learning method for high-speed autonomous racing named ResRace, which leverages only the real-time raw observation of LiDAR and IMU for low-latency obstacle avoiding and navigation. We first design a controller based on the modified artificial potential field (MAPF) to generate a policy for navigation. Besides, we utilize the deep reinforcement learning (DRL) algorithm to generate a residual policy as a supplement to obtain the optimal policy. Concurrently, the MAPF policy effectively guides the exploration and increases the update efficiency. This complementary property contributes to the fast convergence and few required resources of our method. We also provide extensive experiments to illustrate our method outperforms the leading algorithms and reaches the comparable level of professional human players on the five F1Tenth tracks.

Mitigating the risk of musculoskeletal disorders during human robot collaboration: a reinforcement learning approach

Work-related musculoskeletal disorders (MSDs) are often observed in human-robot collaboration (HRC), a common work configuration in modern factories. In this study, we aim to reduce the risk of MSDs in HRC scenarios by developing a novel model-free reinforcement learning (RL) method to improve workers’ postures. Our approach follows two steps: first, we adopt a 3D human skeleton reconstruction method to calculate workers’ Rapid Upper Limb Assessment (RULA) scores; next, we devise an online gradient-based RL algorithm to dynamically improve the RULA score. Compared with previous model-based studies, the key appeals of the proposed RL algorithm are two-fold: (i) the model-free structure allows it to “learn” the optimal worker postures without need any specific biomechanical models of tasks or workers, and (ii) the data-driven nature makes it accustomed to arbitrary users by providing personalized work configurations. Results of our experiments confirm that the proposed method can significantly improve the workers’ postures.

Subcutaneous insulin administration by deep reinforcement learning for blood glucose level control of type-2 diabetic patients

BackgroundType-2 diabetes mellitus is characterized by insulin resistance and impaired insulin secretion in the human body. Many endeavors have been made in terms of controlling and reducing blood glucose via the medium of automated controlling tools to increase precision and efficiency and reduce human error. Recently, reinforcement learning algorithms are proved to be powerful in the field of intelligent control, which was the motivation for the current study. MethodsFor the first time, a reinforcement algorithm called normalized advantage function (NAF) algorithm has been applied as a model-free reinforcement learning method to regulate the blood glucose level of type-2 diabetic patients through subcutaneous injection. The algorithm has been designed and developed in a model-free approach to avoid additional inaccuracies and parameter uncertainty introduced by the mathematical models of the glucoregulatory system. Insulin doses constitute the control action that is designed to be stated directly in clinical language with the unit IU. In this regard, a new environment state is considered in addition to the glucose level to take into account the delayed effect of insulin elimination under the skin. Finally, a simple but practical reward function is developed to be used with the NAF algorithm to correct the glucose level and maintain it in the desired range. ResultsThe simulation environment was set up to imitate the basal-bolus process accurately. Results for 30 days of simulation of the designed controller on three different average virtual patients verify the feasibility and effectiveness of the method and reveal our proposed controller's learning ability. Moreover, as the insulin elimination dynamic was taken into account, a more complete and more realistic model than the previously studied models has emerged. ConclusionNAF has proved a promising control approach, able to successfully regulate and significantly reduce the fluctuation of the blood glucose without meal announcements, compared to standard optimized open-loop basal-bolus therapies. The method and its results, which are directly in the clinical language, are applicable in real-time clinical situations.