On-policy Reinforcement Learning Research Articles

Integrating human learning and reinforcement learning: A novel approach to agent training

Off-policy reinforcement learning (RL) algorithms are known for improving sample efficiency by employing prior experiences in experience replay memory. However, most existing off-policy RL algorithms are bottlenecked by the slow convergence speed and demand of a large number of interaction samples. In contrast, on-policy RL algorithms converge fast and continuously generate new samples. However, their success heavily depends on the accuracy of the generated samples. To address these challenges, a novel RL framework called HI-FER (Human-learning Inspired Frequent Experience Replay) is proposed by mimicking the process of human learning. HI-FER employs a parallelized experience replay and repetitive training framework to expedite the convergence rate of off-policy algorithms, which imitates the function of the human brain’s parallel information processing and repetitive learning. Additionally, a periodic network reset strategy and dynamic memory updating are leveraged by imitating the forgetting mechanism of humans to prevent overfitting triggered by repetitive updating on limited experiences. Extensive comparison experiments and ablation studies are performed on benchmark environments to evaluate the proposed method. The empirical results demonstrate that HI-FER outperforms the baselines in terms of sample efficiency on state-based (14% improvements) and image-based (51% improvements) tasks from DMControl. Project website and code: https://github.com/Arya87/HI-FER.

Reinforcement Learning Powered Station Parameter Optimized Radiation Therapy (SPORT): A Novel Treatment Planning and Beam Delivery Technique

Conventional intensity modulated radiation therapy (IMRT) with a typical 5-20 fixed beams often does not provide sufficient angular sampling required for conformal dose shaping, whereas current volumetric modulated arc therapy (VMAT) discretizes the angular space into equally spaced control points without considering the differential need for intensity modulation of different angles, leading to undersampling at some angles while oversampling at some other angles. Our goal is to develop a node or station parameter optimized radiation therapy (SPORT) strategy with simultaneously optimized angular sampling and beam modulation by leveraging state-of-the-art reinforcement learning and the unique capability of modern digital LINACs in dose delivery through programmable nodal points. We developed a SPORT optimization framework, in which, the process of programming control points (or station parameters) was formulated as a stochastic dynamic programming problem, which was solved by a reinforcement learning-based algorithm. On-policy reinforcement learning method, namely, state-action-reward-state-action (SARSA) was integrated with deep convolutional neural network to predict station parameters by utilizing the patient's anatomical structures meanwhile considering the delivery capability of a typical digital LINAC machine. Here, the deep convolutional neural network estimated the state-action value by using the quality of the plan with current station parameters when a next potential station parameter was selected. The state-action value was then updated by SARSA learning. The quality of the plan was quantified by dosimetry constraints. The model was assessed by a retrospective study on a cohort of patients underwent head-and-neck radiation therapy. Dosimetric analysis and delivery efficiency comparisons were used to evaluate the performance of the proposed framework. Our model was used to generate 16 plans unseen in the original training set. All the plans predicted by our model achieved better dose distributions without violating clinical planning constraints. Moreover, instead of using 4 full standard arcs in the original clinically used plans obtained via manual optimization, the predicted plans only used one full standard arc (about 178 control points) plus boost from a few sub-arcs (less than 30 degrees of gantry angles), which significantly improved the efficiency of the beam delivery. We are in the process of integrating the sub-arcs into the full arc by considering the programmable capability of modern LINACs. We demonstrated that a machine learning-based SPORT framework capable of optimizing the spatial sampling and beam modulation simultaneously for modern radiation therapy. The framework not only significantly improves the quality and efficiency of beam delivery, but also has the potential to be incorporated into current clinical workflow to improve the efficiency of dose planning and delivery.

A Multi-Scaling Reinforcement Learning Trading System Based on Multi-Scaling Convolutional Neural Networks

Advancements in machine learning have led to an increased interest in applying deep reinforcement learning techniques to investment decision-making problems. Despite this, existing approaches often rely solely on single-scaling daily data, neglecting the importance of multi-scaling information, such as weekly or monthly data, in decision-making processes. To address this limitation, a multi-scaling convolutional neural network for reinforcement learning-based stock trading, termed multi-scaling convolutional neural network SARSA (state, action, reward, state, action), is proposed. Our method utilizes a multi-scaling convolutional neural network to obtain multi-scaling features of daily and weekly financial data automatically. This involves using a convolutional neural network with several filter sizes to perform a multi-scaling extraction of temporal features. Multiple-scaling feature mining allows agents to operate over longer time scaling, identifying low stock positions on the weekly line and avoiding daily fluctuations during continuous declines. This mimics the human approach of considering information at varying temporal and spatial scaling during stock trading. We further enhance the network’s robustness by adding an average pooling layer to the backbone convolutional neural network, reducing overfitting. State, action, reward, state, action, as an on-policy reinforcement learning method, generates dynamic trading strategies that combine multi-scaling information across different time scaling, while avoiding dangerous strategies. We evaluate the effectiveness of our proposed method on four real-world datasets (Dow Jones, NASDAQ, General Electric, and AAPLE) spanning from 1 January 2007 to 31 December 2020, and demonstrate its superior profits compared to several baseline methods. In addition, we perform various comparative and ablation tests in order to demonstrate the superiority of the proposed network architecture. Through these experiments, our proposed multi-scaling module yields better results compared to the single-scaling module.

An Actor-Critic Algorithm for the Stochastic Cutting Stock Problem

The inventory level has a significant influence on the cost of process scheduling. The stochastic cutting stock problem (SCSP) is a complicated inventory-level scheduling problem due to the existence of random variables. In this study, we applied a model-free on-policy reinforcement learning (RL) approach based on a well-known RL method, called the Advantage Actor-Critic, to solve a SCSP example. To achieve the two goals of our RL model, namely, avoiding violating the constraints and minimizing cost, we proposed a two-stage discount factor algorithm to balance these goals during different training stages and adopted the game concept of an episode ending when an action violates any constraint. Experimental results demonstrate that our proposed method obtains solutions with low costs and is good at continuously generating actions that satisfy the constraints. Additionally, the two-stage discount factor algorithm trained the model faster while maintaining a good balance between the two aforementioned goals.

Optimizing Patient-Specific Medication Regimen Policies Using Wearable Sensors in Parkinson’s Disease

Effective treatment of Parkinson’s disease (PD) is a continual challenge for healthcare providers, and providers can benefit from leveraging emerging technologies to supplement traditional clinic care. We develop a data-driven reinforcement learning (RL) framework to optimize PD medication regimens through wearable sensors. We leverage a data set of n = 26 PD patients who wore wrist-mounted movement trackers for two separate six-day periods. Using these data, we first build and validate a simulation model of how individual patients’ movement symptoms respond to medication administration. We then pair this simulation model with an on-policy RL algorithm that recommends optimal medication types, timing, and dosages during the day while incorporating human-in-the-loop considerations on medication administration. The results show that the RL-prescribed medication regimens outperform physicians’ medication regimens, despite physicians having access to the same data as the RL agent. To validate our results, we assess our wearable-based RL medication regimens using n = 399 PD patients from the Parkinson’s Progression Markers Initiative data set. We show that the wearable-based RL medication regimens would lead to significant symptom improvement for these patients, even more so than training RL policies directly from this data set. In doing so, we show that RL models from even small data sets of wearable data can offer novel, generalizable clinical insights and medication strategies, which may outperform those derived from larger data sets without wearable data. This paper was accepted by Carri Chan, healthcare management. Funding: This research is partially supported by the Science Alliance, University of Tennessee and by the Laboratory Directed Research and Development Program, Oak Ridge National Laboratory managed by UT-Battelle, LLC for the U.S. Department of Energy. Data used in this article were obtained from the Parkinson Progression Markers Initiative (PPMI) database, which is sponsored by the Michael J. Fox Foundation for Parkinson’s Research (MJFF). Supplemental Material: The data files and online appendix are available at https://doi.org/10.1287/mnsc.2023.4747 .

Delay and energy aware task scheduling mechanism for fog-enabled IoT applications: A reinforcement learning approach

With the expansion of the internet of things (IoT) devices and their applications, the demand for executing complex and deadline-aware tasks is growing rapidly. Fog-enabled IoT architecture has evolved to accomplish these tasks at the fog layer. However, fog computing devices have limited power supply and computation resources compared to cloud devices. In delay-sensitive applications of fog-enabled IoT architecture, executing tasks with stringent deadlines while reducing the service latency and energy usage of fog resources is a difficult challenge. This paper presents an effective task scheduling strategy to allocate fog computing resources for IoT requests to meet the deadline of the requests and resource availability. Initially, the scheduling problem is formulated as mixed-integer nonlinear programming (MINLP) to reduce the energy consumption of the fog resources and service time of the tasks subject to the deadline and resource availability constraints. To address the high dimensionality issue of the tasks in a dynamic environment, a fuzzy-based reinforcement learning (FRL) mechanism is employed to reduce the service delay of the tasks and energy usage of the fog nodes. Initially, the tasks are prioritized using fuzzy logic. Then the prioritized tasks are scheduled using the on-policy reinforcement learning technique, which enhances the long-term reward compared to the Q-learning approach. Further, the evaluation outcomes reflect that the proposed task scheduling technique outperforms the existing algorithms with an improvement of up to 23% and 18% regarding service latency and energy consumption, respectively.

Off-policy and on-policy reinforcement learning with the Tsetlin machine

The Tsetlin Machine is a recent supervised learning algorithm that has obtained competitive accuracy- and resource usage results across several benchmarks. It has been used for convolution, classification, and regression, producing interpretable rules in propositional logic. In this paper, we introduce the first framework for reinforcement learning based on the Tsetlin Machine. Our framework integrates the value iteration algorithm with the regression Tsetlin Machine as the value function approximator. To obtain accurate off-policy state-value estimation, we propose a modified Tsetlin Machine feedback mechanism that adapts to the dynamic nature of value iteration. In particular, we show that the Tsetlin Machine is able to unlearn and recover from the misleading experiences that often occur at the beginning of training. A key challenge that we address is mapping the intrinsically continuous nature of state-value learning to the propositional Tsetlin Machine architecture, leveraging probabilistic updates. While accurate off-policy, this mechanism learns significantly slower than neural networks on-policy. However, by introducing multi-step temporal-difference learning in combination with high-frequency propositional logic patterns, we are able to close the performance gap. Several gridworld instances document that our framework can outperform comparable neural network models, despite being based on simple one-level AND-rules in propositional logic. Finally, we propose how the class of models learnt by our Tsetlin Machine for the gridworld problem can be translated into a more understandable graph structure. The graph structure captures the state-value function approximation and the corresponding policy found by the Tsetlin Machine.

Content-Adaptive Auto-Occlusion Network for Occluded Person Re-Identification.

The occluded person re-identification (ReID) aims to match person images captured in severely occluded environments. Current occluded ReID works mostly rely on auxiliary models or employ a part-to-part matching strategy. However, these methods may be sub-optimal since the auxiliary models are constrained by occlusion scenes and the matching strategy will deteriorate when both query and gallery set contain occlusion. Some methods attempt to solve this problem by applying image occlusion augmentation (OA) and have shown great superiority in their effectiveness and lightness. But there are two defects that existed in the previous OA-based method: 1) The occlusion policy is fixed throughout the entire training and cannot be dynamically adjusted based on the current training status of the ReID network. 2) The position and area of the applied OA are completely random, without reference to the image content to choose the most suitable policy. To address these challenges, we propose a novel Content-Adaptive Auto-Occlusion Network (CAAO), that is able to dynamically select the proper occlusion region of an image based on its content and the current training status. Specifically, CAAO consists of two parts: the ReID network and the Auto-Occlusion Controller (AOC) module. AOC automatically generates the optimal OA policy based on the feature map extracted from the ReID network and applies occlusion on the images for ReID network training. An on-policy reinforcement learning based alternating training paradigm is proposed to iteratively update the ReID network and AOC module. Comprehensive experiments on occluded and holistic person ReID benchmarks demonstrate the superiority of CAAO.

Continuous control actions learning and adaptation for robotic manipulation through reinforcement learning

This paper presents a learning-based method that uses simulation data to learn an object manipulation task using two model-free reinforcement learning (RL) algorithms. The learning performance is compared across on-policy and off-policy algorithms: Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC). In order to accelerate the learning process, the fine-tuning procedure is proposed that demonstrates the continuous adaptation of on-policy RL to new environments, allowing the learned policy to adapt and execute the (partially) modified task. A dense reward function is designed for the task to enable an efficient learning of the agent. A grasping task involving a Franka Emika Panda manipulator is considered as the reference task to be learned. The learned control policy is demonstrated to be generalizable across multiple object geometries and initial robot/parts configurations. The approach is finally tested on a real Franka Emika Panda robot, showing the possibility to transfer the learned behavior from simulation. Experimental results show 100% of successful grasping tasks, making the proposed approach applicable to real applications.

A Novel Adaptive Sampling Strategy for Deep Reinforcement Learning

Reinforcement learning, as an effective method to solve complex sequential decision-making problems, plays an important role in areas such as intelligent decision-making and behavioral cognition. It is well known that the sample experience replay mechanism contributes to the development of current deep reinforcement learning by reusing past samples to improve the efficiency of samples. However, the existing priority experience replay mechanism changes the sample distribution in the sample set due to the higher sampling frequency assigned to a specific transition, and it cannot be applied to actor-critic and other on-policy reinforcement learning algorithm. To address this, we propose an adaptive factor based on TD-error, which further increases sample utilization by giving more attention weight to samples of larger TD-error, and embeds it flexibly into the original Deep Q Network and Advantage Actor-Critic algorithm to improve their performance. Then we carried out the performance evaluation for the proposed architecture in the context of CartPole-V1 and 6 environments of Atari game experiments, respectively, and the obtained results either on the conditions of fixed temperature or annealing temperature, when compared to those produced by the vanilla DQN and original A2C, highlight the advantages in cumulative rewards and climb speed of the improved algorithms.

Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction

Value function is the central notion of Reinforcement Learning (RL). Value estimation, especially with function approximation, can be challenging since it involves the stochasticity of environmental dynamics and reward signals that can be sparse and delayed in some cases. A typical model-free RL algorithm usually estimates the values of a policy by Temporal Difference (TD) or Monte Carlo (MC) algorithms directly from rewards, without explicitly taking dynamics into consideration. In this paper, we propose Value Decomposition with Future Prediction (VDFP), providing an explicit two-step understanding of the value estimation process: 1) first foresee the latent future, 2) and then evaluate it. We analytically decompose the value function into a latent future dynamics part and a policy-independent trajectory return part, inducing a way to model latent dynamics and returns separately in value estimation. Further, we derive a practical deep RL algorithm, consisting of a convolutional model to learn compact trajectory representation from past experiences, a conditional variational auto-encoder to predict the latent future dynamics and a convex return model that evaluates trajectory representation. In experiments, we empirically demonstrate the effectiveness of our approach for both off-policy and on-policy RL in several OpenAI Gym continuous control tasks as well as a few challenging variants with delayed reward.

Few-Shot Model-Based Adaptation in Noisy Conditions

Few-shot adaptation is a challenging problem in the context of simulation-to-real transfer in robotics, requiring safe and informative data collection. In physical systems, additional challenge may be posed by domain noise, which is present in virtually all real-world applications. In this letter, we propose to perform few-shot adaptation of dynamics models in noisy conditions using an uncertainty-aware Kalman filter-based neural network architecture. We show that the proposed method, which explicitly addresses domain noise, improves few-shot adaptation error over a blackbox adaptation LSTM baseline, and over a model-free on-policy reinforcement learning approach, which tries to learn an adaptable and informative policy at the same time. The proposed method also allows for system analysis by analyzing hidden states of the model during and after adaptation.

A Relearning Approach to Reinforcement Learning for control of Smart Buildings

This paper demonstrates that continual relearning of control policies using incremental deep reinforcement learning (RL) can improve policy learning for non-stationary processes. This approach has been demonstrated in a data-driven “smart building environment” that we use as a test-bed for developing HVAC controllers for reducing energy consumption of large buildings on our university campus. The non-stationarity in building operations and weather patterns makes it imperative to develop control strategies that are adaptive to changing conditions. On-policy RL algorithms, such as Proximal Policy Optimization (PPO) represent an approach for addressing this non-stationarity, but they cannot be applied to safety-critical systems. As an alternative, we develop an incremental RL technique that simultaneously reduces building energy consumption without sacrificing overall comfort. We compare the performance of our incremental RL controller to that of a static RL controller that does not implement the relearning function. The performance of the static controller diminishes significantly over time, but the relearning controller adjusts to changing conditions while ensuring comfort and optimal energy performance.

Actor-critic learning for optimal building energy management with phase change materials

Energy management in buildings using phase change materials (PCM) to improve thermal performance is challenging due to the nonlinear thermal capacity of the PCM. To address this problem, this paper adopts a model-free actor-critic on-policy reinforcement learning method based on deep deterministic policy gradient (DDPG). The proposed approach overcomes the major weakness of model-based approaches, such as approximate dynamic programming (ADP), which require an explicit thermal model of the building under control. This requirement makes a plug-and-play implementation of the energy management algorithm in an existing smart meter difficult due to the wide variety of building design and construction types. To overcome this difficulty, we use a DDPG algorithm that can learn policies in continuous action spaces without access to the full dynamics of the building. We demonstrate the competitive performance of DDPG by benchmarking it against an ADP-based approach with access to the full thermal dynamics of the building.

Output Feedback H∞ Control for Linear Discrete-Time Multi-Player Systems With Multi-Source Disturbances Using Off-Policy Q-Learning

In this paper, a data-driven optimal control method based on adaptive dynamic programming and game theory is presented for solving the output feedback solutions of the $H_\infty $ control problem for linear discrete-time systems with multiple players subject to multi-source disturbances. We first transform the $H_\infty $ control problem into a multi-player game problem following the theoretical solutions according to game theory. Since the system state may not be measurable, we derive the output feedback based control policies and disturbances through mathematical operations. Considering the advantages of off-policy reinforcement learning (RL) over on-policy RL, a novel off-policy game Q-learning algorithm dealing with mixed competition and cooperation among players is developed, such that the $H_\infty $ control problem can be finally solved for linear multi-player systems without the knowledge of system dynamics. Moreover, rigorous proofs of algorithm convergence and unbiasedness of solutions are presented. Finally, simulation results demonstrated the effectiveness of the proposed method.

Fault-Tolerant Control of Degrading Systems with On-Policy Reinforcement Learning

We propose a novel adaptive reinforcement learning control approach for fault tolerant control of degrading systems that is not preceded by a fault detection and diagnosis step. Therefore, a priori knowledge of faults that may occur in the system is not required. The adaptive scheme combines online and offline learning of the on-policy control method to improve exploration and sample efficiency, while guaranteeing stable learning. The offline learning phase is performed using a data-driven model of the system, which is frequently updated to track the system’s operating conditions. We conduct experiments on an aircraft fuel transfer system to demonstrate the effectiveness of our approach.

Stock Market Trading Agent Using On-Policy Reinforcement Learning Algorithms

Stock market has been a complex system which has been difficult to predict for humans, thereby, making the trading decisions difficult to take. It will be useful for traders if there is a model agent which can learn the stock market trends and suggest trading decisions which in turn maximizes the profits. Inorder to develop this agent we have formulated the problem as a Markov Decision Process (MDP) and created a stock trading environment which serves as a platform for this agent to trade the stocks. In this paper, we introduce a Reinforcement Learning based approach to develop a trading agent which performs trading actions on the environment and learns according to the rewards in terms of profit or loss it receives. We have applied different On-policy Reinforcement Learning Algorithms such as Vanilla Policy Gradient (VPG), Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) on the environment to obtain the profits while trading stocks for 3 companies viz. Apple, Microsoft and Nike. The performance of these algorithms in order to maximize the profits have been evaluated and the results and conclusions have been elaborated.

Event-triggered resilient control for cyber-physical system under denial-of-service attacks

In this paper, we research the resilient control problem for cyber-physical system (CPS) under denial-of-service (DoS) attacks. These malicious DoS attacks aim to impede the communication of measurement data or control data in order to endanger the functionality of the closed-loop system. Meanwhile, in order to save network resources, event-triggered mechanism has been introduced into this CPS. By exploiting the relationship between cyber system and physical system, we aim to design the resilient controller and resilient control strategy to tolerate a class of DoS signals characterised by probability without serious hazard to the stability and performance of CPS. Furthermore, considering that the transition probability of cyber state is unknown, the on-policy reinforcement learning method – SARSA (State-Action-Reward-State-Action) – is used to solve this problem. Thus a resilient control algorithm that integrates game theory, robust control theory, event-triggered control method and SARSA learning method is presented to enhance the security and robustness of the CPS. At last, the numerical simulation and experimental results are given to demonstrate the validity and applicability of the proposed algorithm.

An adaptive obstacle avoidance algorithm for unmanned surface vehicle in complicated marine environments

Unmanned surface vehicles (USVs) are important autonomous marine robots that have been studied and gradually applied into practice. However, the autonomous navigation of USVs, especially the issue of obstacle avoidance in complicated marine environment, is still a fundamental problem. After studying the characteristics of the complicated marine environment, we propose a novel adaptive obstacle avoidance algorithm for USVs, based on the Sarsa on-policy reinforcement learning algorithm. The proposed algorithm is composed of local avoidance module and adaptive learning module, which are organized by the “divide and conquer” strategy-based architecture. The course angle compensation strategy is proposed to offset the disturbances from sea wind and currents. In the design of payoff value function of the learning strategy, the course deviation angle and its tendency are introduced into action rewards and penalty policies. The validity of the proposed algorithm is verified by comparative experiments of simulations and sea trials in three sea-state marine environments. The results show that the algorithm can enhance the autonomous navigation capacity of USVs in complicated marine environments.

On-policy concurrent reinforcement learning

When an agent learns in a multi-agent environment, the payoff it receives is dependent on the behaviour of the other agents. If the other agents are also learning, its reward distribution becomes non-stationary. This makes learning in multi-agent systems more difficult than single-agent learning. Prior attempts at value-function based learning in such domains have used off-policy Q-learning that do not scale well as the cornerstone, with restricted success. This paper studies on-policy modifications of such algorithms, with the promise of scalability and efficiency. In particular, it is proven that these hybrid techniques are guaranteed to converge to their desired fixed points under some restrictions. It is also shown, experimentally, that the new techniques can learn (from self-play) better policies than the previous algorithms (also in self-play) during some phases of the exploration.