Policy Gradient Reinforcement Learning Research Articles

Adaptive reinforcement learning-based control using proximal policy optimization and slime mould algorithm with experimental tower crane system validation

This paper presents a novel optimal reference tracking control approach resulted from the combination of a popular policy gradient Reinforcement Learning (RL) algorithm, namely Proximal Policy Optimization (PPO), and a metaheuristic Slime Mould Algorithm (SMA). One of the most important parameters in the PPO-based RL process is the learning rate, which has a big impact on how the parameters of the actor neural network (NN) are iteratively updated. In every episode of the RL process, the weights and the biases of the actor NN are multiplied with the learning rate, determining how much the learning agent will step into a certain direction computed based on previous experiences. The classical PPO algorithm usually relies on fixed values for the learning rates which rarely change, or not at all, during the learning process. However, its main drawback is that the learning agent cannot take advantage of positive momentum in the learning process by accelerating towards good learning experiences or slow down and quickly change the direction in the case of consecutive negative learning experiences. The main objective of the combination proposed in this paper is to create an adaptive SMA-based PPO approach applied to control systems, which instead of using fixed learning rate values, it uses the SMA to compute optimal values of the learning rates in each time step of the learning process based on the progress of the learning agent. This paper investigates if the adaptive SMA-based PPO control approach can be considered as an alternative to the classical PPO version, which employs fixed values of the learning rate. A comparison is carried out using control system performance indices gathered while performing an optimal reference tracking control task on tower crane system laboratory equipment.

Manipulating Camera Gimbal Positioning by Deep Deterministic Policy Gradient Reinforcement Learning for Drone Object Detection

The object recognition technology of unmanned aerial vehicles (UAVs) equipped with “You Only Look Once” (YOLO) has been validated in actual flights. However, here, the challenge lies in efficiently utilizing camera gimbal control technology to swiftly capture images of YOLO-identified target objects in aerial search missions. Enhancing the UAV’s energy efficiency and search effectiveness is imperative. This study aims to establish a simulation environment by employing the Unity simulation software for target tracking by controlling the gimbal. This approach involves the development of deep deterministic policy-gradient (DDPG) reinforcement-learning techniques to train the gimbal in executing effective tracking actions. The outcomes of the simulations indicate that when actions are appropriately rewarded or penalized in the form of scores, the reward value can be consistently converged within the range of 19–35. This convergence implies that a successful strategy leads to consistently high rewards. Consequently, a refined set of training procedures is devised, enabling the gimbal to accurately track the target. Moreover, this strategy minimizes unnecessary tracking actions, thus enhancing tracking efficiency. Numerous benefits arise from training in a simulated environment. For instance, the training in this simulated environment is facilitated through a dataset composed of actual flight photographs. Furthermore, offline operations can be conducted at any given time without any constraint of time and space. Thus, this approach effectively enables the training and enhancement of the gimbal’s action strategies. The findings of this study demonstrate that a coherent set of action strategies can be proficiently cultivated by employing DDPG reinforcement learning. Furthermore, these strategies empower the UAV’s gimbal to rapidly and precisely track designated targets. Therefore, this approach provides both convenience and opportunities to gather more flight-scenario training data in the future. This gathering of data will lead to immediate training opportunities and help improve the system’s energy consumption.

Deep Reinforcement Learning Lane-Changing Decision Algorithm for Intelligent Vehicles Combining LSTM Trajectory Prediction

Intelligent decisions for autonomous lane-changing in vehicles have consistently been a focal point of research in the industry. Traditional lane-changing algorithms, which rely on predefined rules, are ill-suited for the complexities and variabilities of real-world road conditions. In this study, we propose an algorithm that leverages the deep deterministic policy gradient (DDPG) reinforcement learning, integrated with a long short-term memory (LSTM) trajectory prediction model, termed as LSTM-DDPG. In the proposed LSTM-DDPG model, the LSTM state module transforms the observed values from the observation module into a state representation, which then serves as a direct input to the DDPG actor network. Meanwhile, the LSTM prediction module translates the historical trajectory coordinates of nearby vehicles into a word-embedding vector via a fully connected layer, thus providing predicted trajectory information for surrounding vehicles. This integrated LSTM approach considers the potential influence of nearby vehicles on the lane-changing decisions of the subject vehicle. Furthermore, our study emphasizes the safety, efficiency, and comfort of the lane-changing process. Accordingly, we designed a reward and penalty function for the LSTM-DDPG algorithm and determined the optimal network structure parameters. The algorithm was then tested on a simulation platform built with MATLAB/Simulink. Our findings indicate that the LSTM-DDPG model offers a more realistic representation of traffic scenarios involving vehicle interactions. When compared to the traditional DDPG algorithm, the LSTM-DDPG achieved a 7.4% increase in average single-step rewards after normalization, underscoring its superior performance in enhancing lane-changing safety and efficiency. This research provides new ideas for advanced lane-changing decisions in autonomous vehicles.

Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning

Large Language Models (LLM) exhibit zero-shot mathematical reasoning capacity as a behavior emergent with scale, commonly manifesting as chain-of-thoughts (CoT) reasoning. However, multiple empirical findings suggest that this prowess is exclusive to LLMs that have exorbitant sizes (beyond 50 billion parameters). Meanwhile, educational neuroscientists suggest that symbolic algebraic manipulation be introduced around the same time as arithmetic word problems so as to modularize language-to-formulation, symbolic manipulation of the formulation, and endgame arithmetic. In this paper, we start with the hypothesis that much smaller LMs, which are weak at multi-step reasoning, can achieve reasonable arithmetic reasoning if arithmetic word problems are posed as a formalize-then-solve task. In our architecture, which we call SyReLM, the LM serves the role of a translator to map natural language arithmetic questions into a formal language (FL) description. A symbolic solver then evaluates the FL expression to obtain the answer. A small frozen LM, equipped with an efficient low-rank adapter, is capable of generating FL expressions that incorporate natural language descriptions of the arithmetic problem (e.g., variable names and their purposes, formal expressions combining variables, etc.). We adopt policy-gradient reinforcement learning to train the adapted LM, informed by the non-differentiable symbolic solver. This marks a sharp departure from the recent development in tool-augmented LLMs, in which the external tools (e.g., calculator, Web search, etc.) are essentially detached from the learning phase of the LM. SyReLM shows massive improvements (e.g., +30.65 absolute point improvement in accuracy on the SVAMP dataset using GPT-J 6B model) over base LMs, while keeping our testbed easy to diagnose and interpret, and within the reach of most researchers.

RRIoT: Recurrent reinforcement learning for cyber threat detection on IoT devices

To address the recent worldwide proliferation of cybersecurity attacks across computing systems, especially internet-of-things devices, new robust and automated methods are needed to detect and mitigate the attacks in real time, ensuring the confidentiality, integrity, and availability of systems. Machine Learning (ML) techniques have shown promise in detecting some types of attacks. However, they are not universally successful in detecting sequential attacks. In this study, we propose RRIoT, to mitigate this issue. RRIoT applies a Deep Deterministic Policy Gradient reinforcement learning (RL) algorithm in conjunction with an LSTM layer within an adversarial environment to detect and identify attacks. We evaluate our method against novel and state-of-the-art ML/RL algorithms which build upon previous RL algorithms such as deep Q-networks (DQN), dueling deep Q-networks (DDQN), and deep deterministic policy gradient (DDPG). Our results indicate that our proposed RRIoT generally performs better than existing ML algorithms and performs as well as or better than novel RL algorithms with new network architectures. We leverage Shapley Additive Global Importance (SAGE) to provide additional insight into which features contribute most to a model's performance and verify feature importance through the implementation of an ablation study across three IoT telemetry datasets.

Data-Driven Optimal Bipartite Consensus Control for Second-Order Multiagent Systems via Policy Gradient Reinforcement Learning.

This article investigates the optimal bipartite consensus control (OBCC) problem for unknown second-order discrete-time multiagent systems (MASs). First, the coopetition network is constructed to describe the cooperative and competitive relationships between agents, and the OBCC problem is proposed by the tracking error and related performance index function. Based on the distributed policy gradient reinforcement learning (RL) theory, a data-driven distributed optimal control strategy is obtained to guarantee the bipartite consensus of all agents' position and velocity states. In addition, the offline data sets ensure the learning efficiency of the system. These data sets are generated by running the system in real time. Besides, the designed algorithm is an asynchronous version, which is essential to solve the challenge caused by the computational ability difference between nodes in MASs. Then, by means of the functional analysis and Lyapunov theory, the stability of the proposed MASs and the convergence of the learning process are analyzed. Furthermore, an actor-critic structure containing two neural networks is used to implement the proposed methods. Finally, a numerical simulation shows the effectiveness and validity of the results.

A perspective on the use of deep deterministic policy gradient reinforcement learning for retention time modeling in reversed-phase liquid chromatography

Artificial intelligence and machine learning techniques are increasingly used for different tasks related to method development in liquid chromatography. In this study, the possibilities of a reinforcement learning algorithm, more specifically a deep deterministic policy gradient algorithm, are evaluated for the selection of scouting runs for retention time modeling. As a theoretical exercise, it is investigated whether such an algorithm can be trained to select scouting runs for any compound of interest allowing to retrieve its correct retention parameters for the three-parameter Neue-Kuss retention model. It is observed that three scouting runs are generally sufficient to retrieve the retention parameters with an accuracy (mean relative percentage error MRPE) of 1 % or less. When given the opportunity to select additional scouting runs, this does not lead to a significantly improved accuracy. It is also observed that the agent tends to give preference to isocratic scouting runs for retention time modeling, and is only motivated towards selecting gradient scouting runs when penalized (strongly) for large analysis/gradient times. This seems to reinforce the general power and usefulness of isocratic scouting runs for retention time modeling. Finally, the best results (lowest MRPE) are obtained when the agent manages to retrieve retention time data for % ACN at elution of the compound under consideration that spread the entire relevant range of ACN (5 % ACN to 95 % ACN) as well as possible, i.e., resulting in retention data at a low, intermediate and high % ACN. Based on the obtained results, we believe reinforcement learning holds great potential to automate and rationalize method development in liquid chromatography in the future.

A DDPG-based energy efficient federated learning algorithm with SWIPT and MC-NOMA

Federated learning (FL) has emerged as a promising distributed machine learning technique. It has the potential to play a key role in future Internet of Things (IoT) networks by ensuring the security and privacy of user data combined with efficient utilization of communication resources. This paper addresses the challenge of maximizing energy efficiency in FL systems. We employed simultaneous wireless information and power transfer (SWIPT) and multi-carrier non-orthogonal multiple access (MC-NOMA) techniques. Also, we jointly optimized power allocation and central processing unit (CPU) resource allocation to minimize latency-constrained energy consumption. We formulated an optimization problem using a Markov decision process (MDP) and utilized a deep deterministic policy gradient (DDPG) reinforcement learning algorithm to solve our MDP problem. We tested the proposed algorithm through extensive simulations and confirmed it converges in a stable manner and provides enhanced energy efficiency compared to conventional schemes.

Robust interplanetary trajectory design under multiple uncertainties via meta-reinforcement learning

This paper focuses on the application of meta-reinforcement learning to the robust design of low-thrust interplanetary trajectories in the presence of multiple uncertainties. A closed-loop control policy is used to optimally steer the spacecraft to a final target state despite the considered perturbations. The control policy is approximated by a deep recurrent neural network, trained by policy-gradient reinforcement learning on a collection of environments featuring mixed sources of uncertainty, namely dynamic uncertainty and control execution errors. The recurrent network is able to build an internal representation of the distribution of environments, thus better adapting the control to the different stochastic scenarios. The results in terms of optimality, constraint handling, and robustness on a fuel-optimal low-thrust transfer between Earth and Mars are compared with those obtained via a traditional reinforcement learning approach based on a feed-forward neural network.

Cooperative control of velocity and heading for unmanned surface vessel based on twin delayed deep deterministic policy gradient with an integral compensator

This paper addresses cooperative control of velocity and heading for an unmanned surface vessel (USV) utilizing a twin delay deep deterministic policy gradient (TD3) reinforcement learning algorithm. The utilization of a deep neural network establishes a direct correlation between the USV’s state parameters and motor control quantities. A reward function is devised to update the network parameters and which acquires the trained model. The introducing of an integral compensator effectively eliminates the steady-state error of the system, thereby significantly enhancing the precision of both velocity control and heading control. Furthermore, a two-stage training algorithm comprising offline learning and online learning has been devised. Through offline learning, a deep neural network model for the USV controller is obtained. Subsequently, the optimization of the controller strategy is conducted during the online learning phase. Ultimately, the simulation results demonstrate the exceptional control performance attained by the proposed algorithm.

Machine Parameter Optimization of a Clinical Linear Accelerator Using Deep Reinforcement Learning for Automatic Generation of Deliverable Prostate VMAT Plans

Reinforcement learning (RL) is an artificial intelligence (AI) approach involving learning through trial and error which has demonstrated performance exceeding humans and existing algorithms in multiple domains. We hypothesized that machine parameter optimization using RL will produce deliverable treatment plans faster and with superior dosimetry compared to conventional inverse optimization, which was tested in the setting of VMAT for localized prostate cancer. We included 151 previously treated prostate cancer patients, separated into training (n = 116) and testing (n = 35) cohorts. We used a policy gradient RL approach, where contours, dose distribution and machine parameters for the current control point are taken as input by a 3D convolutional neural network (CNN) and continuous machine parameters for the next control point are predicted, allowing the CNN to directly control the linear accelerator (Linac) model. Training was initialized using a subset of 20 training cases with plans produced in our clinical treatment planning system (TPS) prescribed 60 Gy in 20 fractions (denoted Training-TPS), but continued for all 116 training cases through trial and error-based RL. Following training, RL VMAT was applied to the initialization subset of 20 training cases (Training-RL) and 35 test cases (Test-RL), comparing resultant dosimetry. Training was conducted for 10 days using 4 GPUs generating 22,000 plans for exploration, enabling dosimetric improvements compared to the TPS-based plans and generalization to all 116 training cases. Following training, mean ± SD RL execution time for automatic plan generation was 2.9 ± 0.6 seconds. Automatic gradient descent-based parameter rescaling was applied to enable quantitative comparison with the TPS-based plans, and final dose metrics are provided in Table 1. Overall dosimetry was comparable between the TPS and RL approaches in the training and test sets, although PTV coverage and maximum dose did not meet objectives in several RL-based plans. To our knowledge, this is the first demonstration of direct RL-based control of a 3D Linac model. Initial performance is promising, demonstrating model convergence in the training set with a small number of existing plans required for initialization. However, limitations exist in model generalizability which are under continued investigation through approaches made possible through RL including data augmentation.

Embedding active learning in batch-to-batch optimization using reinforcement learning

Batch-to-batch (B2B) or run-to-run (R2R) optimization refers to the strategy of updating the operating parameters of a batch run based on the results of previous runs and exploits the repetitive nature of batch process operation. Although B2B optimization uses feedback from previous batch runs to learn about model uncertainty and improve the operation of future runs, the standard techniques have the limitations of passive learning and being myopic in making adjustments. This work proposes a novel way to use the reinforcement learning approach to embed the active learning feature into B2B optimization. For this, the B2B optimization problem is formulated as a maximization of a long-term performance of repeated batch runs, which are modeled as a stochastic process with uncertain parameters. To solve the resulting Bayes-Adaptive Markov decision process (BAMDP) problem in a near-optimal manner, a policy gradient reinforcement learning algorithm is employed. Through case studies, the behavior and effectiveness of the proposed B2B optimization method are examined by comparing it with the traditional certainty equivalence based B2B optimization method with passive learning.

Management of Congestion in Distribution Networks Utilizing Demand Side Management and Reinforcement Learning

The high penetration of distributed energy resources and heavy flexible loads, such as electric vehicles, has changed the operating conditions of the distribution network. In particular, the adoption of green energy innovations has caused an increase in electricity consumption. Such an increase can result in thermal overloading due to power flow exceeding a network asset's transfer capability, possibly damaging devices such as distribution transformers and feeders. It is very challenging to design a congestion management scheme given the uncertainty of flexible loads consumption and electricity prices. Obtaining stochastic models for such loads may not be easily available in practice. In this article, a deep deterministic policy gradient (DDPG) reinforcement learning (RL) scheme is proposed to alleviate congestion. DDPG RL is a model free technique that does not require explicit probabilistic models of the controllable loads to determine the change in electricity prices needed, in the form of tariffs and/or subsidies. The DDPG RL technique is compared with the existing model-based congestion management scheme using an IEEE 33 bus system and the results obtained demonstrate the out performance of the proposed technique in terms of the electricity cost and peak-to-average ratio of the load profiles.

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

We present a mathematical framework and computational methods for optimally designing a finite sequence of experiments. This sequential optimal experimental design (sOED) problem is formulated as a finite-horizon partially observable Markov decision process (POMDP) under a Bayesian setting and with information-theoretic utilities. The formulation is general and may accommodate continuous random variables, non-Gaussian posteriors, and nonlinear forward models. The sOED design policy incorporates elements of feedback and lookahead simultaneously, and we show it to generalize the commonly-used batch and greedy design strategies. We solve for the sOED policy using the policy gradient (PG) method from reinforcement learning, and provide a derivation for the PG expression in the sOED context. Adopting an actor–critic approach, the policy and value functions are parameterized using deep neural networks and improved via PG estimates produced from simulated episodes of designs and observations. The new PG-sOED algorithm is first validated on a linear-Gaussian benchmark, and then compared against other design baselines on a sensor movement problem for contaminant source inversion in a convection–diffusion field. We provide explanation for the policy behaviors using knowledge of the underlying physical process.

Decentralized multi-agent control of a three-tank hybrid system based on twin delayed deep deterministic policy gradient reinforcement learning algorithm

Decentralized multi-agent control of a three-tank hybrid system based on twin delayed deep deterministic policy gradient reinforcement learning algorithm

Molecule generation using transformers and policy gradient reinforcement learning

Generating novel valid molecules is often a difficult task, because the vast chemical space relies on the intuition of experienced chemists. In recent years, deep learning models have helped accelerate this process. These advanced models can also help identify suitable molecules for disease treatment. In this paper, we propose Taiga, a transformer-based architecture for the generation of molecules with desired properties. Using a two-stage approach, we first treat the problem as a language modeling task of predicting the next token, using SMILES strings. Then, we use reinforcement learning to optimize molecular properties such as QED. This approach allows our model to learn the underlying rules of chemistry and more easily optimize for molecules with desired properties. Our evaluation of Taiga, which was performed with multiple datasets and tasks, shows that Taiga is comparable to, or even outperforms, state-of-the-art baselines for molecule optimization, with improvements in the QED ranging from 2 to over 20 percent. The improvement was demonstrated both on datasets containing lead molecules and random molecules. We also show that with its two stages, Taiga is capable of generating molecules with higher biological property scores than the same model without reinforcement learning.

A Hybrid Reinforcement Learning Approach With a Spiking Actor Network for Efficient Robotic Arm Target Reaching

The increasing demand for applications in competitive fields, such as assisted living and aerial robots, drives contemporary research into the development, implementation and integration of power-constrained solutions. Although, deep neural networks (DNNs) have achieved remarkable performances in many robotics applications, energy consumption remains a major limitation. The paper at hand proposes a hybrid variation of the well-established deep deterministic policy gradient (DDPG) reinforcement learning approach to train a 6 degree of freedom robotic arm in the target-reach task <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://www.youtube.com/watch?v=238693OpvD0.</uri> In particular, we introduce a spiking neural network (SNN) for the actor model and a DNN for the critic one, aiming to find an optimal set of actions for the robot. The deep critic network is employed only during training and discarded afterwards, allowing the deployment of the SNN in neuromorphic hardware for inference. The agent is supported by a combination of RGB and laser scan data exploited for collision avoidance and object detection. We compare the hybrid-DDPG model against a classic DDPG one, demonstrating the superiority of our approach.

Data-Based Predictive Control via Multistep Policy Gradient Reinforcement Learning.

In this article, a model-free predictive control algorithm for the real-time system is presented. The algorithm is data driven and is able to improve system performance based on multistep policy gradient reinforcement learning. By learning from the offline dataset and real-time data, the knowledge of system dynamics is avoided in algorithm design and application. Cooperative games of the multiplayer in time horizon are presented to model the predictive control as optimization problems of multiagent and guarantee the optimality of the predictive control policy. In order to implement the algorithm, neural networks are used to approximate the action-state value function and predictive control policy, respectively. The weights are determined by using the methods of weighted residual. Numerical results show the effectiveness of the proposed algorithm.

Combining Neural Networks with Logic Rules

How to utilize symbolic knowledge in deep learning is an important problem. Deep neural networks are flexible and powerful, while symbolic knowledge has the virtue of interpretability and intuitiveness. It is necessary to combine the two together to inject symbolic knowledge into neural networks. We propose a novel approach to combine neural networks with logic rules. In this approach, task-specific supervised learning and policy-based reinforcement learning are performed alternately to train a neural model until convergence. The basic idea is to use supervised learning to train a deep model and use reinforcement learning to propel the deep model to meet logic rules. In the process of the policy gradient reinforcement learning, if a predicted output of a deep model meets all logical rules, the deep model is given a positive reward, otherwise, it is given a negative reward. By maximizing the expected rewards, the deep model can be gradually adjusted to meet logical constraints. We conduct experiments on the tasks of named entity recognition. The experimental results demonstrate the effectiveness of our method.

Function approximation reinforcement learning of energy management with the fuzzy REINFORCE for fuel cell hybrid electric vehicles

In the paper, a novel self-learning energy management strategy (EMS) is proposed for fuel cell hybrid electric vehicles (FCHEV) to achieve the hydrogen saving and maintain the battery operation. In the EMS, it is proposed to approximate the EMS policy function with fuzzy inference system (FIS) and learn the policy parameters through policy gradient reinforcement learning (PGRL). Thus, a so-called Fuzzy REINFORCE algorithm is first proposed and studied for EMS problem in the paper. Fuzzy REINFORCE is a model-free method that the EMS agent can learn itself through interactions with environment, which makes it independent of model accuracy, prior knowledge, and expert experience. Meanwhile, to stabilize the training process, a fuzzy baseline function is adopted to approximate the value function based on FIS without affecting the policy gradient direction. Moreover, the drawbacks of traditional reinforcement learning such as high computation burden, long convergence time, can also be overcome. The effectiveness of the proposed methods were verified by Hardware-in-Loop experiments. The adaptability of the proposed method to the changes of driving conditions and system states is also verified.