Policy-based Reinforcement Learning Research Articles

Deep deterministic policy gradient algorithm: A systematic review

Deep Reinforcement Learning (DRL) has gained significant adoption in diverse fields and applications, mainly due to its proficiency in resolving complicated decision-making problems in spaces with high-dimensional states and actions. Deep Deterministic Policy Gradient (DDPG) is a well-known DRL algorithm that adopts an actor-critic approach, synthesizing the advantages of value-based and policy-based reinforcement learning methods. The aim of this study is to provide a thorough examination of the latest developments, patterns, obstacles, and potential opportunities related to DDPG. A systematic search was conducted using relevant academic databases (Scopus, Web of Science, and ScienceDirect) to identify 85 relevant studies published in the last five years (2018-2023). We provide a comprehensive overview of the key concepts and components of DDPG, including its formulation, implementation, and training. Then, we highlight the various applications and domains of DDPG, including Autonomous Driving, Unmanned Aerial Vehicles, Resource Allocation, Communications and the Internet of Things, Robotics, and Finance. Additionally, we provide an in-depth comparison of DDPG with other DRL algorithms and traditional RL methods, highlighting its strengths and weaknesses. We believe that this review will be an essential resource for researchers, offering them valuable insights into the methods and techniques utilized in the field of DRL and DDPG.

Learning strategies for underwater robot autonomous manipulation control

Autonomous manipulation operations represent the high intelligent coordination from robotic vision and control, it is also a symbol of the advances of robotic intelligence. The limitations of visual sensing and the increasingly complex experimental conditions make autonomous manipulation operations more difficult, particularly for deep reinforcement learning methods, which can enhance robotic control intelligence but require a lot of training process. Due to the high-dimensional continuous state space and continuous action space characteristics of underwater operations, this paper adopts a policy-based reinforcement learning method as the foundational approach. To address the issues of instability and low convergence efficiency in traditional policy-based reinforcement learning algorithms during the learning process, this paper proposes a novel policy learning method. This method adopts the Proximal Policy Optimization algorithm (PPOClip) and optimizes it through an actor-critic network. The aim is to improve the stability and effectiveness of convergence in the learning process. In the underwater training environment, a new reward shaping scheme has been designed to address the issue of reward sparsity during the training process. The manually crafted dense reward function is utilized as attractive and repulsive potential functions for goal manipulation and obstacle avoidance. On the highly complex underwater manipulation and training environment, transferred learning algorithm has been established to reduce the training times and compensate the differences between the simulation and experiment. Simulations and tank experiments have verified the performance of the proposed strategy learning method.

Policy-Based Reinforcement Learning for Through Silicon Via Array Design in High-Bandwidth Memory Considering Signal Integrity

Policy-Based Reinforcement Learning for Through Silicon Via Array Design in High-Bandwidth Memory Considering Signal Integrity

A Knowledge-Guided End-to-End Optimization Framework Based on Reinforcement Learning for Flow Shop Scheduling

Designing an effective and efficient end-to-end optimization framework with good generalization for shop scheduling is an emerging topic in the informational manufacturing system. Existing end-to-end frameworks have achieved satisfactory results for COPs such as traveling salesman problem and vehicle routing problem. However, the performances of these methods in solving complex COPs such as shop scheduling need to be improved. In this paper, a knowledge-guided end-to-end optimization framework based on reinforcement learning is proposed to solve the permutation flow shop scheduling problem (PFSP). Firstly, a new policy network is designed based on the problem characteristics to deal with different scales of PFSPs and achieve iterative end-to-end generation. Secondly, an improved policy-based reinforcement learning algorithm by using the knowledge accumulated during the training process is designed to enhance the training quality. Thirdly, a knowledge-guided improvement strategy is introduced through the cooperation of local search and supervised learning to improve the learning of the policy. Simulation results and comparisons show that the knowledge-guided end-to-end optimization framework can obtain better results than different kinds of commonly-used optimization methods in limited computation time for solving the PFSP.

A review of reinforcement learning based hyper-heuristics.

The reinforcement learning based hyper-heuristics (RL-HH) is a popular trend in the field of optimization. RL-HH combines the global search ability of hyper-heuristics (HH) with the learning ability of reinforcement learning (RL). This synergy allows the agent to dynamically adjust its own strategy, leading to a gradual optimization of the solution. Existing researches have shown the effectiveness of RL-HH in solving complex real-world problems. However, a comprehensive introduction and summary of the RL-HH field is still blank. This research reviews currently existing RL-HHs and presents a general framework for RL-HHs. This article categorizes the type of algorithms into two categories: value-based reinforcement learning hyper-heuristics and policy-based reinforcement learning hyper-heuristics. Typical algorithms in each category are summarized and described in detail. Finally, the shortcomings in existing researches on RL-HH and future research directions are discussed.

TRL-PROTAC: A pre-trained generator of PROTACs targeting specific proteins optimized by reinforcement learning

Proteolysis-targeting chimeras (PROTACs) introduce a novel paradigm in drug development, incorporating three essential components: the warhead, the E3 ligand, and the linker. The complexity of the ternary structure, particularly the intricate design of the linker, presents a significant challenge in PROTACs drug design. Here an integrated protocol for design and evaluation of PROTACs targeting specific proteins, called TRL-PROTAC is proposed. TRL-PROTAC is focused on the de novo design of complete PROTACs by effectively joining the designed ligands targeting the proteins of interest (POI) with linkers. The ligands for POIs and E3 ligases are generated by a molecular generation model for targeting proteins, and the linker design is generated by a sequence-to-sequence model consisting of a transformer variant and the policy-based reinforcement learning method which is employed to optimize the reward values for generating PROTACs. The three components are then integrated and optimized based on their pharmacokinetic (PK) and degradation (DEG) properties. The experimental results have strongly confirmed that TRL-PROTAC is superior in optimizing properties. For existing PROTACs, TRL-PROTAC improves DEG scores by 0.45 and lowers PK scores by 1.20. Moreover, TRL-PROTAC enhances binding affinity by 2.15 in PROTACs generated from scratch.

Hybrid Policy-Based Reinforcement Learning of Adaptive Energy Management for the Energy Transmission-Constrained Island Group

This paper proposes a hybrid policy-based reinforcement learning (HPRL) adaptive energy management to realize the optimal operation for the island group energy system with energy transmission-constrained environment. An island energy hub (IEH) model that can realize the energy cascade utilization is proposed. Compared with the traditional model, the IEH can satisfy the special energy demand of island meanwhile ensure the energy supply of island. Moreover, an energy management model of islands group (EMIG) based on the IEH is formulated which comprehensively considers the inverse distribution of energy demand and resources, as well as the limited energy transmission. Since the environment model of island is difficult to construct due to the increase of proportion of renewable energy generation and civilian load, the EMIG is transformed into a reinforcement learning (RL) task which features model-free. Considering the limitations of traditional RL in discrete-continuous hybrid action space, HPRL is proposed to achieve optimal operation without simplifying the model. Numerical simulations demonstrate the effectiveness of the proposed adaptive energy management.

Performance, robustness, and portability of imitation-assisted reinforcement learning policies for shading and natural ventilation control

Space heating and cooling account for approximately half of all building-related energy consumption, emitting 3Gt of CO2 annually, or nearly 10% of the global total. Operable shading, natural ventilation, and solar heating are promising strategies for reducing these emissions, leveraging minimal mechanical energy to condition space with cool night air, cold night skies, and solar radiation. However, these strategies are under-utilized because their performance depends on rigorous coordination among their operable elements. Additionally, the individuality of such systems, and the lack of physics-based models suitable for control design, have thwarted the development of widely-applicable control strategies. To address this problem, here we develop a new data-driven strategy for the design of shading and natural ventilation controls in residential buildings using policy-based reinforcement learning (RL). To limit undesirable actions and reduce training time, we first used imitation learning to initialize RL training with expert knowledge, yielding an initial policy that reduced simulated late-spring space conditioning loads by ≥40% in 24 climatically diverse cities. This policy was then trained with RL in four cities representing Mediterranean, semi-arid, humid subtropical, and continental climates. When deployed in cities with unfamiliar yet related climates, these new policies reduced space conditioning loads by ≥50% in the humid subtropics and by ≥90% in the other three climates, showing exceptional portability. Further, their performance was unexpectedly robust to variations in dwelling orientation, glazing, internal heat gain, and air leakage. These results show the extraordinary potential of imitation-assisted RL in developing high-performance policies for dynamic passive heating and cooling control that remain effective in unfamiliar situations, removing a substantial barrier to passive systems advancement in carbon-free building operation.

Equivariant Graph-Representation-Based Actor-Critic Reinforcement Learning for Nanoparticle Design.

We have developed an actor-critic-type policy-based reinforcement learning (RL) method to find low-energy nanoparticle structures and compared its effectiveness to classical basin-hopping. We took a molecule building approach where nanoalloy particles can be regarded as metallic molecules, albeit with much higher flexibility in structure. We explore the strengths of our approach by tasking an agent with the construction of stable mono- and bimetallic clusters. Following physics, an appropriate reward function and an equivariant molecular graph representation framework is used to learn the policy. The agent succeeds in finding well-known stable configuration for small clusters in both single and multicluster experiments. However, for certain use cases the agent lacks generalization to avoid overfitting. We relate this to the pitfalls of actor-critic methods for molecular design and discuss what learning properties an agent will require to achieve universality.

Combining Neural Networks with Logic Rules

How to utilize symbolic knowledge in deep learning is an important problem. Deep neural networks are flexible and powerful, while symbolic knowledge has the virtue of interpretability and intuitiveness. It is necessary to combine the two together to inject symbolic knowledge into neural networks. We propose a novel approach to combine neural networks with logic rules. In this approach, task-specific supervised learning and policy-based reinforcement learning are performed alternately to train a neural model until convergence. The basic idea is to use supervised learning to train a deep model and use reinforcement learning to propel the deep model to meet logic rules. In the process of the policy gradient reinforcement learning, if a predicted output of a deep model meets all logical rules, the deep model is given a positive reward, otherwise, it is given a negative reward. By maximizing the expected rewards, the deep model can be gradually adjusted to meet logical constraints. We conduct experiments on the tasks of named entity recognition. The experimental results demonstrate the effectiveness of our method.

Automated Generation of Ensemble Pipelines using Policy-Based Reinforcement Learning method

At the moment, there are a considerable number of different automated machine learning frameworks. They are often use predefined pipelines and choose the best one among them. However, searching for optimal pipelines can be improved by using methods that generate pipelines step by step. The paper introduces an approach to generate ensemble pipelines using policy-based reinforcement learning. Approach consists of pipeline, environment, state, action and reward representations. This approach was successfully integrated into automatic machine learning framework. The generated pipelines were tested by comparing a baseline model using OpenML datasets, and the proposed approach demonstrated high efficiency, even surpassing the metrics for some datasets. This research has the potential to enhance the existing pipeline generation methods.

PHH: Policy-Based Hyper-Heuristic With Reinforcement Learning

Hyper-heuristics have a high level of generality and adaptability, allowing them to effectively solve a wide range of complex optimization problems. With reinforcement learning, hyper-heuristics can use experience and knowledge gained to tackle unforeseen problems, allowing hyper-heuristics to adapt and improve over time. Our paper proposes a framework for using policy-based reinforcement learning to improve the performance of hyper-heuristics. The framework trains hyper-heuristic agents to select the best generalized constructive low-level heuristics to solve combinatorial optimization problems. The framework evaluation was performed using three benchmarking problems: traveling salesman, capacitated vehicle routing, and bin packing problems. The results showed that the proposed framework can outperform existing meta-heuristic and hyper-heuristic-based algorithms for all large problem instances in all problem domains. The proposed framework was also evaluated by applying it to a cost optimization problem for workflow scheduling on a hybrid cloud with a deadline constraint. Eight agents were trained on medium-sized workflows with two deadlines and tested against traditional meta-heuristic and hyper-heuristic methods to solve smaller and larger workflows with unforeseen deadlines. Four workflow applications, three workflow sizes, and three deadlines were used in the evaluation. The results showed that our proposed framework provided significantly better solutions of up to 98% for benchmarking problems, and up to 22% for cost optimization in workflow scheduling. Moreover, trained with small problem instances, the framework performed well for unforeseen larger problem instances implying its generalization. The proposed framework, thus, has the potential to improve both the generality and performance of solving large combinatorial optimization problems.

Gradient-Enhanced Bayesian Optimization via Acquisition Ensembles with Application to Reinforcement Learning

Bayesian optimization (BO) has shown great promise as a data-efficient strategy for the global optimization of expensive, black-box functions in a plethora of control applications. Traditional BO is derivative-free, as it solely relies on observations of a performance function to find its optimum. Recently, so-called first-order BO methods have been proposed that additionally exploit gradient information of the performance function to accelerate convergence. First-order BO methods mostly utilize standard acquisition functions, while indirectly using gradient information in the kernel structure to learn more accurate probabilistic surrogates for the performance function. In this work, we present a gradient-enhanced BO method that directly exploits performance function (zeroth-order) and its corresponding gradient (first-order) evaluations in the acquisition function. To this end, a novel gradient-based acquisition function is proposed that can identify stationary points of the performance optimization problem. We then leverage ideas from multi-objective optimization to develop an effective strategy for finding query points that optimally tradeoff between a zeroth-order acquisition function and the proposed gradient-based acquisition function. We show how the proposed acquisition-ensemble gradient-enhanced BO (AEGEBO) method enables accelerating convergence of policy-based reinforcement learning by combining noisy observations of the reward function and its gradient that can be directly estimated from closed-loop data. The performance of AEGBO is compared to standard BO and the well-known REINFORCE algorithm on a benchmark LQR problem, for which we consistently observe significantly improved performance over a limited data budget.

Research on a Real-time Job-shop Scheduling Method Based on Reinforcement Learning

At present, manufacturing models are characterized by multi-variety, small batch, and diversification. It is insufficient to use traditional scheduling methods for production management with high performance. A real-time production scheduling system based on reinforcement learning (RL) is suggested in an effort to address the aforementioned issues. A brand-new manufacturing neural network is created to learn the state-action values for production scheduling in real time using high-dimensional data as the input. The detailed setup of network inputs, neural network, action, and reward are also designed. Then, a policy-based reinforcement learning algorithm is proposed to achieve the optimum objective. Finally, By contrasting the proposed scheduling strategy with rule-based approaches in a smart manufacturing environment, its efficacy is demonstrated. according to experimental data, the suggested algorithm can successfully improve performance in the dynamic job-shop environment.

Research on an Adaptive Real-Time Scheduling Method of Dynamic Job-Shop Based on Reinforcement Learning

With the rapid development of modern industrialization in our country and the continuous improvement of people’s living standards, the changing market has put forward new requirements for traditional manufacturing enterprises. The order product demand of many manufacturing enterprises is changing from a single variety, large batch to multiple varieties, small batch. In view of this change, the traditional job-shop scheduling method is far from enough, which greatly affects the efficiency of the production job-shop. In order to solve the above problems, this paper proposes a real-time scheduling method based on reinforcement learning applied in the dynamic job-shop and a new type of neural network is designed at the same time. The neural network is designed with the high-dimensional data in the above problem as input, and a policy-based reinforcement learning algorithm is proposed based on this network. In the process of research, it was found that the reinforcement learning method not only enables the agent to use historical data for learning, but also enables it to explore and learn other possible high-reward actions within a certain range, so as to realize the optimization of production goals under real-time scheduling. The effectiveness of the proposed real-time scheduling method is verified by comparing with other common rule-based scheduling methods in the manufacturing environment.

CeCO: Cost-Efficient Computation Offloading of IoT Applications in Green Industrial Fog Networks

Fog computing is one of the promising technology that could reduce the execution cost and energy consumption of smart industrial Internet of Things (IIoT) devices via a strategy called offloading. However, designing an intelligent offloading strategy for large-scale industrial applications becomes challenging. To address this issue, in this article, we design a novel fog federation, a computation offloading framework for industrial networks called cost-efficient computation offloading ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CeCO</monospace> ), where a master fog controller regulates the network and distributes the IIoT data among the fog devices. In particular, we design our cost optimization function as the sum of weighted energy-delay cost of IIoT devices while reaching several constraints. To determine this optimization problem, we first design a frequency control mechanism for the IIoT devices. Then, we introduce a controller-based device adaptation strategy and a policy-based reinforcement learning technique for efficiently controlling emergency-based service demands and accordingly route them toward the fog devices following the shortest path. Experimental results demonstrate the effectiveness of the <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CeCO</monospace> strategy then the baseline algorithms while maintaining the same and even better cost utilization and performance maximization upto 13%–18% for industrial applications.

Stochastic cubic-regularized policy gradient method

Policy-based reinforcement learning methods have achieved great achievements in real-world decision-making problems. However, the theoretical understanding of policy-based methods is still limited. Specifically, existing works mainly focus on first-order stationary point policies (FOSPs); in some very special reinforcement learning settings (e.g., tabular case and function approximation with restricted parametric policy classes) some works consider globally optimal policy. It is well-known that FOSPs could be undesirable local optima or saddle points, and obtaining a global optimum is generally NP-hard. In this paper, we propose a policy gradient method that provably converges to second-order stationary point policies (SOSPs) for any differentiable policy classes. The proposed method is computationally efficient, and it judiciously uses cubic-regularized subroutines to escape saddle points while at the same time minimizing the Hessian-based computations. We prove that the method enjoys the sample complexity of O˜(ϵ−3.5), which improves upon the current optimal complexity O˜ϵ−4.5. Finally, experimental results are provided to demonstrate the effectiveness of the method.

Policy-Based Reinforcement Learning for Training Autonomous Driving Agents in Urban Areas With Affordance Learning

Learning to drive in urban areas is an open challenge for autonomous vehicles (AVs), as complex decision making requirements are needed in multi-task co-ordinations environments. In this paper, we propose a hybrid framework with a new perception model involving affordance learning to simplify the surrounding urban scenes for training an AV agent, along with a planned trajectory and the associated driving measurements. Our proposed solution encompasses two main aspects. Firstly, a supervised learning network is used to map the input sensory data into affordance predictions. The predicted affordances provide a low-dimensional representation of surrounding scenes of the AV in the form of key perception indicators, e.g., true or false with respect to a traffic light signal. Secondly, a deep deterministic policy gradient model that maps the perception information into a series of actions is devised. We evaluate the proposed solution using the CARLA driving simulator in an urban town and evaluate the performance in a new, unseen town under different weather conditions. The quantitative and qualitative results indicate that our proposed solution can generalize well to cope with different traffic solutions and environmental conditions. Our proposed solution also outperforms other baseline methods in a comparative study in handling various AV driving tasks with different levels of difficulty. In addition, the model trained with simulated scenes yields promising prediction results when testing on recorded video streams on real-world highway and suburb environments with varying traffic and weather conditions.

A General Framework for Learning Mean-Field Games

This paper presents a general mean-field game (GMFG) framework for simultaneous learning and decision making in stochastic games with a large population. It first establishes the existence of a unique Nash equilibrium to this GMFG, and it demonstrates that naively combining reinforcement learning with the fixed-point approach in classical mean-field games yields unstable algorithms. It then proposes value-based and policy-based reinforcement learning algorithms (GMF-V and GMF-P, respectively) with smoothed policies, with analysis of their convergence properties and computational complexities. Experiments on an equilibrium product pricing problem demonstrate that two specific instantiations of GMF-V with Q-learning and GMF-P with trust region policy optimization—GMF-V-Q and GMF-P-TRPO, respectively—are both efficient and robust in the GMFG setting. Moreover, their performance is superior in convergence speed, accuracy, and stability when compared with existing algorithms for multiagent reinforcement learning in the N-player setting. Funding: X. Guo acknowledges the generous support from Coleman Fung Chair endowment fund and the Tsinghua-Berkeley-Shenzhen-Institute.

Improving de novo molecular design with curriculum learning

Reinforcement learning is a powerful paradigm that has gained popularity across multiple domains. However, applying reinforcement learning may come at the cost of multiple interactions between the agent and the environment. This cost can be especially pronounced when the single feedback from the environment is slow or computationally expensive, causing extensive periods of non-productivity. Curriculum learning provides a suitable alternative by arranging a sequence of tasks of increasing complexity, with the aim of reducing the overall cost of learning. Here we demonstrate the application of curriculum learning for drug discovery. We implement curriculum learning in the de novo design platform REINVENT, and apply it to illustrative molecular design problems of different complexities. The results show both accelerated learning and a positive impact on the quality of the output when compared with standard policy-based reinforcement learning. While reinforcement learning can be a powerful tool for complex design tasks such as molecular design, training can be slow when problems are either too hard or too easy, as little is learned in these cases. Jeff Guo and colleagues provide a curriculum learning extension to the REINVENT de novo molecular design framework that provides problems of increasing difficulty over epochs such that the training process is more efficient.