Trust Region Policy Optimization Research Articles

Research on Supply Chain Optimization and Management Based on Deep Reinforcement Learning

This research introduces a groundbreaking approach to supply chain optimization and management, termed as Deep Reinforcement Learning based Supply Chain Optimization and Management (DRL-SCOM). At the core of this approach is the utilization of advancements in Deep Reinforcement Learning (DRL), specifically through the integration of Randomized Ensembled Double Q-learning (REDQ) and Trust Region Policy Optimization (TRPO). DRL-SCOM is designed to effectively tackle the inherent complexities and dynamic challenges that are characteristic of supply chain management. One of the key strengths of DRL-SCOM lies in its use of REDQ, which plays a crucial role in mitigating the overestimation bias commonly associated with traditional Q-learning methods. This results in more accurate value estimation and policy improvement, a critical factor in the effective management of supply chains. Additionally, the integration of TRPO into the framework brings the advantage of safe and stable policy updates. Such stability is vital for maintaining the robustness required in the fluctuating environment of supply chain operations. The combination of REDQ and TRPO in DRL-SCOM creates a powerful synergy. REDQ’s ensembled learning approach, when fused with TRPO’s trust-region method, enables the framework to efficiently navigate the complex and high-dimensional decision space typical of supply chains. This allows for real-time optimization of decisions while staying within operational constraints. The DRL-SCOM methodology shows significant potential in addressing various aspects of supply chain management, from demand forecasting and inventory management to logistics, adeptly handling the nonlinearities and uncertainties that are prevalent in these areas. Thus, the DRL-SCOM framework emerges as an innovative solution, pushing the frontiers of traditional supply chain management. It paves the way for a more agile, responsive, and intelligent system, equipped to adapt to changing market demands and operational challenges. This approach represents a significant stride towards transforming supply chain management into a more advanced, data-driven, and adaptive field.

Innovative energy solutions: Evaluating reinforcement learning algorithms for battery storage optimization in residential settings

The implementation of BESS (battery energy storage systems) and the efficient optimization of their scheduling are crucial research challenges in effectively managing the intermittency and volatility of solar-PV (photovoltaic) systems. Nevertheless, an examination of the existing body of knowledge uncovers notable deficiencies in the ideal arrangement of energy systems' timetables. Most models primarily concentrate on a single aim, whereas only a few tackle the intricacies of multi-objective scenarios. This study examines homes connected to the power grid equipped with a BESS and a solar PV system. It leverages four distinct reinforcement learning (RL) algorithms, selected for their unique training methodologies, to develop effective scheduling models. The findings demonstrate that the RL model using Trust Region Policy Optimization (TRPO) effectively manages the BESS and PV system despite real-world uncertainties. This case study confirms the suitability and effectiveness of this approach. The TRPO-based RL framework surpasses previous models in decision-making by choosing the most optimal BESS scheduling strategies. The TRPO model exhibited the highest mean self-sufficiency rates compared to the A3C (Asynchronous Advantage Actor-Critic), DDPG (Deep Deterministic Policy Gradient), and TAC (Twin Actor Cretic) models, surpassing them by ∼ 3 %, 0.72 %, and 3.5 %, correspondingly. This results in enhanced autonomy and economic benefits by adapting to dynamic real-world conditions. Consequently, our approach was strategically designed to deliver an optimized outcome. This framework is primarily intended for seamless integration into an automated energy plant environment, facilitating regular electricity trading among multiple buildings. Backed by initiatives like the Renewable Energy Certificate weight, this technology is expected to play a crucial role in maintaining a balance between power generation and consumption. The MILP (Mixed Integer Linear Programming) architecture achieved a self-sufficiency rate of 29.12 %, surpassing the rates of A3C, TRPO, DDPG, and TAC by 2.48 %, 0.64 %, 2 %, and 3.04 %, correspondingly.

Multiagent Trust Region Policy Optimization.

We extend trust region policy optimization (TRPO) to cooperative multiagent reinforcement learning (MARL) for partially observable Markov games (POMGs). We show that the policy update rule in TRPO can be equivalently transformed into a distributed consensus optimization for networked agents when the agents' observation is sufficient. By using a local convexification and trust-region method, we propose a fully decentralized MARL algorithm based on a distributed alternating direction method of multipliers (ADMM). During training, agents only share local policy ratios with neighbors via a peer-to-peer communication network. Compared with traditional centralized training methods in MARL, the proposed algorithm does not need a control center to collect global information, such as global state, collective reward, or shared policy and value network parameters. Experiments on two cooperative environments demonstrate the effectiveness of the proposed method.

Reinforcement learning for battery energy management: A new balancing approach for Li-ion battery packs

This study investigates the challenge of cell balancing in battery management systems (BMS) for lithium-ion batteries. Effective cell balancing is crucial for maximizing the usable capacity and lifespan of battery packs, which is essential for the widespread adoption of electric vehicles and the reduction of greenhouse gas emissions. A novel deep reinforcement learning (deep RL) approach is proposed for passive balancing with switched shunt resistors. Notable deep RL algorithms capable of handling discrete actions, such as Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Q-Network (DQN), Augmented Random Search (ARS), and Asynchronous Advantage Actor Critic (A3C), are investigated. TRPO demonstrates superior performance compared to other deep RL algorithms and rule-based methods in both charging and discharging scenarios without requiring fine-tuning, optimizing the balance between cell balancing and switch changes. It achieves up to 16.8% improvement in battery pack capacity, 69.4% reduction in state-of-charge variance among cells, and 40.4% decrease in the number of switching operations in simulation results for five li-ion cells connected in series.The study introduces an innovative application of deep RL for passive balancing, a comprehensive battery cell modeling technique, and a tailored multi-objective reward function that balances cell balancing and switching costs. This work represents a significant advancement in applying deep RL to battery management systems, providing a framework for further research and practical implementation in energy storage systems.

Flexible recommendation for optimizing the debt collection process based on customer risk using deep reinforcement learning

Finance sector loss can be minimized by reducing the number of defaulters who often miss payments during debt collection. Most research focused on the credit risk analysis before approving the loan to the customer, and few papers related to predicting the defaulters. Certain works are focused on minimizing the number of calls made by debt collectors during the collection process. We have developed a Flexible Recommender System using Deep Reinforcement Learning (FRS-DRL) to provide Flexible recommendations based on the risk category of the customers. Initially, we categorized customers into defaulters and non-defaulters. The defaulters are again classified into Very Low, Low, Medium, High, and Very High-Risk customers. Based on this category, we recommend suitable actions to the Field Collection Agent (FCA) to increase the collection rate. The suggestions include how frequently the customer should be contacted, statements, and the dates and times of visits. At the same time, the statements define instructions to FCA for collecting money. Managers are traditionally in charge of this type of allocation; they must understand each customer personally. Our proposed recommendation method eliminates the need for manual mapping. We build a Hybrid Actor-Critic algorithm with an Adaptive Trust region Policy Optimization algorithm for classifying the customer category and providing the appropriate recommendations to reduce the defaulter rate, thereby increasing the collection efficiency and reducing the human effort. This FRS-DRL is tested in a real-time environment for three months and provides an improved reward rate with and without transfer learning at 73.22 % and 62.44 %. The collection percentage improved by 20.34 % in five different regions, with a 16.30 % reduction in the number of visits made by the FCA.

Self-Adaptive Server Anomaly Detection Using Ensemble Meta-Reinforcement Learning

As the user’s behavior changes at any time with cloud computing and network services, abnormal server resource utilization traffic will lead to severe service crashes and system downtime. The traditional single anomaly detection model cannot handle the rapid failure prediction ahead. Therefore, this study proposed ensemble learning combined with model-agnostic meta-reinforcement learning called ensemble meta-reinforcement learning (EMRL) to implement self-adaptive server anomaly detection rapidly and precisely, according to the time series of server resource utilization. The proposed ensemble approach combines hidden Markov model (HMM), variational autoencoder (VAE), temporal convolutional autoencoder (TCN-AE), and bidirectional long short-term memory (BLSTM). The EMRL algorithm trains this combination with several tasks to learn the implicit representation of various anomalous traffic, where each task executes trust region policy optimization (TRPO) to quickly adapt the time-varying data distribution and make rapid decisions precisely for an agent response. As a result, our proposed approach can improve the precision of anomaly prediction by 2.4 times and reduce the model deployment speed by 5.8 times on average because a meta-learner can immediately be applied to new tasks.

Reinforcement learning for Multi-Flight Dynamic Pricing

Dynamic pricing is essential for airline revenue management, requiring quick adaptation to fluctuating market environments and complex customer behaviors. This study addresses the Multi-Flight Dynamic Pricing (MFDP) problem, which presents unique challenges due to interdependent demand between multiple flights and high dimensionality. Traditional studies often assume that the demand function modeling customer behavior is either known in advance or follows a predefined structure, failing to capture the dynamic nature of pricing decisions. To fill this gap, we develop deep reinforcement learning (DRL) algorithms—Deep Q-Network (DQN), Advantage Actor-Critic (A2C), Proximal Policy Optimization (PPO), and Trust Region Policy Optimization (TRPO). By formulating the MFDP problem as a Markov Decision Process (MDP), we design an innovative utility function for the Multinomial Logit (MNL) model that captures realistic features of the airline market, such as competition from high-speed rail, the effect of reference fares, and travel time. We compare the performance of our DRL algorithms with traditional algorithms, including Dynamic Programming (DP), Price Pooling (PP), Inventory Pooling (IP), and Inventory and Price Pooling (IPP). Our experiments demonstrate that DRL algorithms alleviate the curse of dimensionality faced by traditional algorithms, expedite the learning process, and deliver satisfactory performance without relying on predefined demand functions. Among these algorithms, TRPO shows superior performance, achieving 99% of the theoretical optimal revenue, proving its adaptability and stability in dynamic pricing applications. We also highlight the importance of considering the null price in the action space of MFDP problems. The larger the market scale, the more pronounced the effect of the null price in accelerating RL algorithm convergence, leading to more efficient computational resource utilization.

Deep reinforcement learning for maintenance optimization of a scrap-based steel production line

This paper presents a Deep Reinforcement Learning (DRL)-based optimization approach for determining the optimal inspection and maintenance planning of a scrap-based steel production line. The DRL-based optimization maintenance recommends the adequate time for inspections and maintenance activities based on the monitoring conditions of the production line, such as machine productivity, buffer level, and production demand. Some practical aspects of the system, such as such uncertainty of the maintenance duration and the variable production rate of the machines, were considered. A scrap-based steel production line was modeled as a multi-component system considering components dependencies. A simulation model was developed to simulate the dynamics of the system and assist with the development of the DRL maintenance approach. The proposed DRL-based maintenance is compared with traditional maintenance policies, such reactive maintenance, time-based maintenance, and condition-based maintenance. In addition, different DRL algorithms such as PPO (Proximal Policy Optimization), TRPO (Trust Region Policy Optimization), and DQN (Deep Q-Network) are investigated in the case-based scenario. The findings indicated the potential for significant financial savings. Therefore, the proposed maintenance approach demonstrates system adaptability and has the potential to be a powerful tool for industrial competitiveness.

Evaluating the stealth of reinforcement learning-based cyber attacks against unknown scenarios using knowledge transfer techniques

Organizations are vulnerable to cyber attacks as they rely on computer networks and the internet for communication and data storage. While Reinforcement Learning (RL) is a widely used strategy to simulate and learn from these attacks, RL-guided offensives against unknown scenarios often lead to early exposure due to low stealth resulting from mistakes during the training phase. To address this issue, this work evaluates if the use of Knowledge Transfer Techniques (KTT), such as Transfer Learning and Imitation Learning, reduces the probability of early exposure by smoothing mistakes during training. This study developed a laboratory platform and a method to compare RL-based cyber attacks using KTT for unknown scenarios. The experiments simulated 2 unknown scenarios using 4 traditional RL algorithms and 4 KTT. In the results, although some algorithms using KTT obtained superior results, they were not so significant for stealth during the initial epochs of training. Nevertheless, experiments also revealed that throughout the entire learning cycle, Trust Region Policy Optimization (TRPO) is a promising algorithm for conducting cyber offensives based on Reinforcement Learning.

Trust region policy optimization via entropy regularization for Kullback–Leibler divergence constraint

Trust region policy optimization (TRPO) is one of the landmark policy optimization algorithms in deep reinforcement learning. Its purpose is to maximize a surrogate objective based on an advantage function, subject to the limited Kullback–Leibler (KL) divergence of two consecutive policies. Although there have been many successful applications of this algorithm in the literature, the approach has often been criticized for suppressing the exploration ability of some application environments due to its strict divergence constraint. As such, most researchers prefer to use entropy regularization, which is added to the expected discounted rewards or the surrogate objectives. That said, there is much debate about whether there might be an alternative strategy for regularizing TRPOs. In this paper, we present just that. Our strategy is to regularize the KL divergence-based constraint via Shannon entropy. This approach enlarges the difference between two consecutive policies and thus derives a new TRPO scheme with entropy regularization for use with KL divergence constraint. Next, the surrogate objective and Shannon entropy are approximated linearly, while the KL divergence is expanded quadratically. An efficient conjugate gradient optimization procedure then solves two sets of linear equations, providing a detailed code-level implementation that can be used for a fair experimental comparison. Extensive experiments within eight benchmark environments demonstrate that our proposed method is superior to both the original TRPO and the entropy regularized objective TRPO. Further, theoretical and experimental analysis shows that three TRPO-like methods have an equal time complexity and a close computational burden.

Deep reinforcement learning for dynamic distributed job shop scheduling problem with transfers

Dynamic events and transportation constraints would significantly affect the full utilization of resources and the reduction of production costs in distributed job shops. Therefore, in this paper, a deep reinforcement learning algorithm (DRL)-based real-time scheduling method is developed to minimize the mean tardiness of the dynamic distributed job shop scheduling problem with transfers (DDJSPT) considering random job arrivals. Firstly, the proposed DDJSPT is modeled as a Markov decision process (MDP). Then, ten problem-oriented state features covering four aspects of factories, machines, jobs, and operations are elaborately extracted from the dynamic distributed job shop. After that, eleven composite rules considering the uniqueness of DDJSPT are constructed as a pool of actions to intelligently prioritize unfinished jobs and allocate the selected job to an appropriate factory. Moreover, a justified reward function adapted from the objective is designed for better convergence of DRLs. Subsequently, five DRLs are employed to address the DDJSPT, encompassing deep Q-network (DQN), double DQN (DDQN), dueling DQN (DlDQN), trust region policy optimization (TRPO), and proximal policy optimization (PPO). Finally, grounded in numerical comparison experiments under 243 production configurations of the DDJSPT, the effectiveness and generalization of DRL-based scheduling methods are credibly verified and confirmed.

Military Decision Support with Actor and Critic Reinforcement Learning Agents

While the recent advanced military operational concept requires an intelligent support of command and control, Reinforcement Learning (RL) has not been actively studied in the military domain. This study points out the limitations of RL for military applications from literature review and aims at improving the understanding of RL for military decision support under the limitations. Most of all, the black box characteristic of Deep RL makes the internal process difficult to understand in addition to complex simulation tools. A scalable weapon selection RL framework is built which can be solved either by a tabular form or a neural network form. The transition of the Deep Q-Network (DQN) solution to the tabular form makes it easier to compare the result to the Q-learning solution. Furthermore, rather than using one or two RL models selectively as before, RL models are divided as an actor and a critic, and systematically compared. A random agent, Q-learning and DQN agents as a critic, a Policy Gradient (PG) agent as an actor, Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) agents as an actor-critic approach are designed, trained, and tested. The performance results show that the trained DQN and PPO agents are the best decision supporter candidates for the weapon selection RL framework.

Delta robot control by learning systems: Harnessing the power of deep reinforcement learning algorithms

The present study focuses on the implementation of Deep Reinforcement Learning (Deep-RL) techniques for a parallel manipulator robot, specifically the Delta Robot, within a simulated setting. We introduced a simulation framework designed to guide the Delta Robot’s end-effector to a designated spatial point accurately. Within this environment, the robotic agent undergoes a learning process grounded in trial and error. It garners positive rewards for successful predictions regarding the next action and faces negative repercussions for inaccuracies. Through this iterative learning mechanism, the robot refines its strategies, thereby establishing improved decision-making rules based on the ever-evolving environment states. Our investigation delved into three distinct Deep-RL algorithms: the Deep Q-Network Algorithm (DQN), the Double Deep Q-Network (DDQN), and the Trust Region Policy Optimization Algorithm (TRPO). All three methodologies were adept at addressing the challenge presented, and a comprehensive discussion of the findings is encapsulated in the subsequent sections of the paper.

A Novel Trajectory Planning Method Based on Trust Region Policy Optimization

A Novel Trajectory Planning Method Based on Trust Region Policy Optimization

A Reinforcement Learning Approach for Continuum Robot Control

Rigid joint manipulators are limited in their movement and degrees of freedom (DOF), while continuum robots possess a continuous backbone that allows for free movement and multiple DOF. Continuum robots move by bending over a section, taking inspiration from biological manipulators such as tentacles and trunks. This paper presents an implementation of a forward kinematics and velocity kinematics model to describe the planar continuum robot, along with the application of reinforcement learning (RL) as a control algorithm. In this paper, we have adopted the planar constant curvature representation for the forward kinematic modeling. This choice was made due to its straightforward implementation and its potential to fill the literature gap in the field RL-based control for planar continuum robots. The intended control mechanism is achieved through the use of Deep Deterministic Policy Gradient (DDPG), a RL algorithm that is suited for learning controls in continuous action spaces. After simulating the algorithm, it was observed that the planar continuum robot can autonomously move from any initial point to any desired goal point within the task space of the robot. By analyzing the results, we wanted to recommend a future direction for research in the field of continuum robot control, specifically in the application of RL algorithms. One potential area of focus could be the integration of sensory feedback, such as vision or force sensing, to improve the robot’s ability to navigate complex environments. Additionally, exploring the use of different RL algorithms, such as Proximal Policy Optimization (PPO) or Trust Region Policy Optimization (TRPO), could lead to further advancements in the field. Overall, this paper demonstrates the potential for RL-based control of continuum robots and highlights the importance of continued research in this area.

Deep Q-Network (DQN): Reinforcement Learning Based Approach for Secure Social Distancing Adherence with SARS-CoV-2 in Public Places

The estimates taken far and wide to deal with the SARS-CoV-2 pandemic, limiting travel, shuttering superfluous organizations and implementing all social separating arrangements, are having serious monetary consequences. a noteworthy decrease in economic action spread over the economy the world, lasting in excess of a few months, typically clear in genuine GDP. Where it is formally announced a downturn. To quicken a strong expected recuperation with rising protectionism and unilateralism. There is a requirement for individuals to come out and face the circumstance. Despite the fact that it is established that separating individuals and investigating their contacts would be inadequate to control the SARS-CoV-2 pandemic, in light of the fact that there would be an excess of deferral between the beginning of indications and seclusion. Consequently, in these sorts of conditions it is to keep people groups from infection influence and early anticipation of these tainted individuals may prompt re development the economy too. We built up a numerical model utilizing profound Deep reinforcement learning (DRL) which is poised to revolutionize the field of artificial intelligence and the use of central algorithms in deep RL, specifically the deep Q-network (DQN), trust region policy optimization (TRPO). The proposed astute checking framework can be utilized as a reciprocal apparatus to be introduced at better places and consequently screen individuals receiving the security rules. With these prudent estimations, people will have the option to win this battle against SARS-CoV-2.

Robust Reinforcement Learning-Based Multiple Inputs and Multiple Outputs Controller for Wind Turbines

The control of variable-speed wind turbines that generate electricity from the kinetic energy of the wind involves subsystems that need to be controlled simultaneously, namely, the blade pitch angle controllers and the generator torque controllers. The presented study solves the control problem with multiple inputs and multiple outputs (MIMO), using the method of reinforcement learning–based Trust Region Policy Optimization, through which the control parameters of both subsystems are simultaneously optimized. In this case, the robust control problem is transformed into a constrained optimal control problem with an appropriate choice of value functions for the nominal system. The study aims to synthesize a robust controller, with the aim of maximizing the generated energy (power) and minimizing unwanted forces (thrust). The innovative control architecture uses an extended input space, which allows fine-tuning of parameters for each operating state. Test calculations carried out in simulation experiments using models of the 5 MW NREL wind turbine and the 4 MW Enercon E-126 EP3 wind turbine are presented to illustrate the performance and practicality of the proposed approach.

Vibration and Position Control of a Two-Link Flexible Manipulator Using Reinforcement Learning

In recent years, industries have increasingly emphasized the need for high-speed, energy-efficient, and cost-effective solutions. As a result, there has been growing interest in developing flexible link manipulator robots to meet these requirements. However, reducing the weight of the manipulator leads to increased flexibility which, in turn, causes vibrations. This research paper introduces a novel approach for controlling the vibration and motion of a two-link flexible manipulator using reinforcement learning. The proposed system utilizes trust region policy optimization to train the manipulator’s end effector to reach a desired target position, while minimizing vibration and strain at the root of the link. To achieve the research objectives, a 3D model of the flexible-link manipulator is designed, and an optimal reward function is identified to guide the learning process. The results demonstrate that the proposed approach successfully suppresses vibration and strain when moving the end effector to the target position. Furthermore, the trained model is applied to a physical flexible manipulator for real-world control verification. However, it is observed that the performance of the trained model does not meet expectations, due to simulation-to-real challenges. These challenges may include unanticipated differences in dynamics, calibration issues, actuator limitations, or other factors that affect the performance and behavior of the system in the real world. Therefore, further investigations and improvements are recommended to bridge this gap and enhance the applicability of the proposed approach.

Deep-reinforcement-learning-based gait pattern controller on an uneven terrain for humanoid robots

Although conventional gait pattern control in humanoid robots is typically performed on flat terrains, the roads that people walk on every day have bumps and potholes. Therefore, to make humanoid robots more similar to humans, the movement parameters of these robots should be modified to allow them to adapt to uneven terrains. In this study, to solve this problem, reinforcement learning (RL) was used to allow humanoid robots to engage in self-training and automatically adjust their parameters for ultimate gait pattern control. However, RL has multiple types, and each type has its own benefits and shortcomings. Therefore, a series of experiments were performed, and the results indicated that proximal policy optimization (PPO), combining advantage actor-critic and trust region policy optimization, was the most suitable method. Hence, an improved version of PPO, called PPO2, was used, and the experimental results indicated that the combination of deep RL with data preprocessing methods, such as wavelet transform and fuzzification, facilitated the gait pattern control and balance of humanoid robots.

Relative sparsity for medical decision problems.

Existing statistical methods can estimate a policy, or a mapping from covariates to decisions, which can then instruct decision makers (eg, whether to administer hypotension treatment based on covariates blood pressure and heart rate). There is great interest in using such data-driven policies in healthcare. However, it is often important to explain to the healthcare provider, and to the patient, how a new policy differs from the current standard of care. This end is facilitated if one can pinpoint the aspects of the policy (ie, the parameters for blood pressure and heart rate) that change when moving from the standard of care to the new, suggested policy. To this end, we adapt ideas from Trust Region Policy Optimization (TRPO). In our work, however, unlike in TRPO, the difference between the suggested policy and standard of care is required to be sparse, aiding with interpretability. This yields "relative sparsity," where, as a function of a tuning parameter, , we can approximately control the number of parameters in our suggested policy that differ from their counterparts in the standard of care (eg, heart rate only). We propose a criterion for selecting , perform simulations, and illustrate our method with a real, observational healthcare dataset, deriving a policy that is easy to explain in the context of the current standard of care. Our work promotes the adoption of data-driven decision aids, which have great potential to improve health outcomes.