Proximal Policy Optimization Research Articles

Deep reinforcement learning-based spatio-temporal graph neural network for solving job shop scheduling problem

AbstractThe job shop scheduling problem (JSSP) is a well-known NP-hard combinatorial optimization problem that focuses on assigning tasks to limited resources while adhering to certain constraints. Currently, deep reinforcement learning (DRL)-based solutions are being widely used to solve the JSSP by defining the problem structure on disjunctive graphs. Some of the proposed approaches attempt to leverage the structural information of the JSSP to capture the dynamics of the environment without considering the time dependency within the JSSP. However, learning graph representations only from the structural relationship of nodes results in a weak and incomplete representation of these graphs which does not provide an expressive representation of the dynamics in the environment. In this study, unlike existing frameworks, we defined the JSSP as a dynamic graph to explicitly consider the time-varying aspect of the JSSP environment. To this end, we propose a novel DRL framework that captures both the spatial and temporal attributes of the JSSP to construct rich and complete graph representations. Our DRL framework introduces a novel attentive graph isomorphism network (Attentive-GIN)-based spatial block to learn the structural relationship and a temporal block to capture the time dependency. Additionally, we designed a gated fusion block that selectively combines the learned representations from the two blocks. We trained the model using the proximal policy optimization algorithm of reinforcement learning. Experimental results show that our trained model exhibits significant performance enhancement compared to heuristic dispatching rules and learning-based solutions for both randomly generated datasets and public benchmarks.

Spatio‐temporal dynamic navigation for electric vehicle charging using deep reinforcement learning

AbstractThis paper considers the real‐time spatio‐temporal electric vehicle charging navigation problem in a dynamic environment by utilizing a shortest path‐based reinforcement learning approach. In a data sharing system including transportation network, an electric vehicle (EV) and EV charging stations (EVCSs), it is aimed to determine the most convenient EVCS and the optimal path for reducing the travel, charging and waiting costs. To estimate the waiting times at EVCSs, Gaussian process regression algorithm is integrated using a real‐time dataset comprising of state‐of‐charge and arrival‐departure times of EVs. The optimization problem is modelled as a Markov decision process with unknown transition probability to overcome the uncertainties arising from time‐varying variables. A recently proposed on‐policy actor–critic method, phasic policy gradient (PPG) which extends the proximal policy optimization algorithm with an auxiliary optimization phase to improve training by distilling features from the critic to the actor network, is used to make EVCS decisions on the network where EV travels through the optimal path from origin node to EVCS by considering dynamic traffic conditions, unit value of EV owner and time‐of‐use charging price. Three case studies are carried out for 24 nodes Sioux‐Falls benchmark network. It is shown that phasic policy gradient achieves an average of 9% better reward compared to proximal policy optimization and the total time decreases by 7–10% when EV owner cost is considered.

Reinforcement learning for dynamic pricing and capacity allocation in monetized customer wait-skipping services

ABSTRACT We consider how to facilitate a dynamically-priced premium service option that enables customer parties to shorten their wait in a queue. Offering such a service requires that some of a business’s capacity be reserved continuously and kept ready for premium customers. In tandem with capacity reservation, pricing must be coordinated. Hence, a joint dynamic pricing and capacity allocation problem lies at the heart of this service. We propose a conceptual solution architecture and employ Proximal Policy Optimization (PPO) for dynamic pricing and capacity allocation to maximize total revenue. Simulation experiments over multiple scenarios compared PPO against a human-engineered policy and a baseline policy having no premium option. The human-engineered policy led to significantly greater revenues than the baseline policy in each scenario, illustrating the potential increase in revenues afforded by this concept. The PPO agent substantially improved upon the human-engineered policy advantage, with improvements ranging from 28% to 161%.

Application of Reinforcement Learning in Controlling Quadrotor UAV Flight Actions

Most literature has extensively discussed reinforcement learning (RL) for controlling rotorcraft drones during flight for traversal tasks. However, most studies lack adequate details regarding the design of reward and punishment mechanisms, and there is a limited exploration of the feasibility of applying reinforcement learning in actual flight control following simulation experiments. Consequently, this study focuses on the exploration of reward and punishment design and state input for RL. The simulation environment is constructed using AirSim and Unreal Engine, with onboard camera footage serving as the state input for reinforcement learning. The research investigates three RL algorithms suitable for discrete action training. The Deep Q Network (DQN), Advantage Actor–Critic (A2C), and Proximal Policy Optimization (PPO) were combined with three different reward and punishment design mechanisms for training and testing. The results indicate that employing the PPO algorithm along with a continuous return method as the reward mechanism allows for effective convergence during the training process, achieving a target traversal rate of 71% in the testing environment. Furthermore, this study proposes integrating the YOLOv7-tiny object detection (OD) system to assess the applicability of reinforcement learning in real-world settings. Unifying the state inputs of simulated and OD environments and replacing the original simulated image inputs with a maximum dual-target approach, the experimental simulation achieved a target traversal rate of 52% ultimately. In summary, this research formulates a set of logical frameworks for an RL reward and punishment design deployed with real-time Yolo’s OD implementation synergized as a useful aid for related RL studies.

A novel reinforcement learning-based method for structure optimization

With the rapid development of deep learning technology, Reinforcement Learning (RL) has garnered considerable acclaim within the realm of structural optimization owing to its excellent exploration mechanism. However, the widespread application of RL in this field is limited owing to the excessive number of iterations required to converge and the expensive computational cost it brings. To address these challenges, this article presents a novel RL framework for structural optimization, combining Monte Carlo tree search with the proximal policy optimization method, called LMPOM. The key contributions of LMPOM encompass: (1) an enhanced Monte Carlo tree search strategy for partitioning the hybrid design space; (2) a strategy for adaptively updating surrogate models to reduce simulation costs; and (3) the introduction of a novel termination condition for the RL algorithms. Through tests on three benchmark problems, compared with previous RL algorithms, LMPOM consistently shows fewer iterations and better optimization results.

Enhancing stability and explainability in reinforcement learning with machine learning

Abstract. In the field of reinforcement learning, training agents using machine learning algorithms to learn and perform tasks in complex environments has become a prevalent approach. However, reinforcement learning faces challenges such as training instability and decision opacity, which limit its feasibility in real-world applications. To solve the problems of stability and transparency in reinforcement learning, this project will use advanced algorithms like Proximal Policy Optimization (PPO), Q-DAGGER, and Gradient Boosting Decision Trees to set up reinforcement learning agents in the OpenAI Gymnasium environment. Specifically, the study selected the Atari game Breakout as the testbed, enhancing training efficiency and game performance by refining reward structures and decision-making processes, and integrating interpretable models to provide explanations for agent decisions. This study has successfully developed robust reinforcement learning agents that excel in complex environments. By employing advanced algorithms like PPO, Q-DAGGER, and Gradient Boosting Decision Trees, the study has addressed issues of training instability, and improved game performance through optimized reward structures and decision processes. Additionally, by integrating interpretable models, the study has provided insights into the learned strategies of the agents, thereby enhancing decision transparency. These findings provide crucial support for the broader application of reinforcement learning in real-world scenarios and offer valuable insights for tackling other complex tasks.

Vehicle-To-Grid (V2G) Charging and Discharging Strategies of an Integrated Supply–Demand Mechanism and User Behavior: A Recurrent Proximal Policy Optimization Approach

With the increasing global demand for renewable energy and heightened environmental awareness, electric vehicles (EVs) are rapidly becoming a popular clean and efficient mode of transportation. However, the widespread adoption of EVs has presented several challenges, such as the lagging development of charging infrastructure, the impact on the power grid, and the dynamic changes in user charging behavior. To address these issues, this paper first proposes a vehicle-to-grid (V2G) optimization framework that responds to regional dynamic pricing. It also considers power balancing in charging and discharging stations when a large number of EVs are involved in scheduling, with the aim of maximizing the benefits for EV owners. Next, by leveraging the interaction between environmental states and the dynamic behavior of EVs, we design an optimization algorithm that combines the recurrent proximal policy optimization (RPPO) algorithm and long short-term memory (LSTM) networks. This approach enhances system convergence and improves grid stability while maximizing benefits for EV owners. Finally, a simulation platform is used to validate the practical application of the RPPO algorithm in optimizing V2G and grid-to-vehicle (G2V) charging strategies, providing significant theoretical foundations and technical support for the development of smart grids and sustainable transportation systems.

Intelligent design of steel–concrete composite beams based on deep reinforcement learning

In this work, an automated member design method is proposed based on deep reinforcement learning (DRL). Using steel–concrete composite beam design as a case study, the design procedure is conceptualized as a sequential optimization process and modeled as a Markov Decision Process (MDP). A design agent equipped with two deep neural networks, is trained using the proximal policy optimization (PPO) method to complete the complex design task of steel–concrete composite beams, adhering to the Chinese standard for design of steel structures GB50017-2017 provisions. The structural provisions are integrated into a simulated design environment, which can respond to the agent’s actions. A universal reward shaping function is introduced to guide the agent towards success and minimize material cost. The design quality and generation performance of trained PPO agent are compared with those of the Differential Evolution (DE) in 100 random training cases, 100 random testing cases, and 95 real-world engineering cases, demonstrating significant promise and reduced time consumption for automated steel–concrete composite beam design. An approach combining PPO with DE is proposed to enhance the robustness and reliability of the original agent. Finally, a brief discussion is conducted on the essence and prospects of the proposed method.

Simulation and Optimization of Automated Guided Vehicle Charging Strategy for U-Shaped Automated Container Terminal Based on Improved Proximal Policy Optimization

As the decarbonization strategies of automated container terminals (ACTs) continue to advance, electrically powered Battery-Automated Guided Vehicles (B-AGVs) are being widely adopted in ACTs. The U-shaped ACT, as a novel layout, faces higher AGV energy consumption due to its deep yard characteristics. A key issue is how to adopt charging strategies suited to varying conditions to reduce the operational capacity loss caused by charging. This paper proposes a simulation-based optimization method for AGV charging strategies in U-shaped ACTs based on an improved Proximal Policy Optimization (PPO) algorithm. Firstly, Gated Recurrent Unit (GRU) structures are incorporated into the PPO to capture temporal correlations in state information. To effectively limit policy update magnitudes in the PPO, we improve the clipping function. Secondly, a simulation model is established by mimicking the operational process of the U-shaped ACTs. Lastly, iterative training of the proposed method is conducted based on the simulation model. The experimental results indicate that the proposed method converges faster than standard PPO and Deep Q-network (DQN). When comparing the proposed method-based charging threshold with a fixed charging threshold strategy across six different scenarios with varying charging rates, the proposed charging strategy demonstrates better adaptability to terminal condition variations in two-thirds of the scenarios.

Using Deep Reinforcement Learning (DRL) for minimizing power consumption in Video-on-Demand (VoD) storage systems

As video streaming services such as Netflix become popular, resolving the problem of high power consumption arising from both large data size and high bandwidth in video storage systems has become important. However, because various factors, such as the power characteristics of heterogeneous storage devices, variable workloads, and disk array models, influence storage power consumption, reducing power consumption with deterministic policies is ineffective. To address this, we present a new deep reinforcement learning (DRL)-based file placement algorithm for replication-based video storage systems, which aims to minimize overall storage power consumption. We first model the video storage system with time-varying streaming workloads as the DRL environment, in which the agent aims to find power-efficient file placement. We then propose a proximal policy optimization (PPO) algorithm, consisting of (1) an action space that determines the placement of each file; (2) an observation space that allows the agent to learn a power-efficient placement based on the current I/O bandwidth utilization; (3) a reward model that assigns a greater penalty for increased power consumption for each action; and (4) an action masking model that supports effective learning by preventing agents from selecting unnecessary actions. Extensive simulations were performed to evaluate the proposed scheme under various solid-state disk (SSD) models and replication configurations. Results show that our scheme reduces storage power consumption by 5% to 25.8% (average 12%) compared to existing benchmark methods known to be effective for file placement.

Optimization of High-Frequency Trading Strategies Using Deep Reinforcement Learning

This study presents a new method for optimising high-risk trading (HFT) strategies using deep learning (DRL). We propose a multi-time DRL framework integrating advanced neural network architectures with sophisticated business data processing techniques. The framework employs a combination of convolutional neural networks for manual order analysis, short-term memory networks for time series processing, and a multi-head listening mechanism for body fusion. We formulate the HFT problem based on Markov Decision Processes and use the Proximal Policy Optimization algorithm for training. The model is evaluated using tick-by-tick data from the NASDAQ exchange, including ten liquid stocks in 6 months. The experimental results show the superiority of our method, achieving a Sharpe ratio of 3.42, outperforming the learning model and machine learning based on benchmarks up to 33%. The proposed strategy has demonstrated strong performance across a wide range of regulatory markets and has shown potential for strategic objectives. Sensitivity analysis confirms the model's stability across a range of hyperparameters. Our findings suggest that the DRL-based approach can improve HFT performance and provide better market adaptation and risk management. This research leads to the continuous evolution of algorithmic trading strategies and shows the potential of AI-driven approaches in financial markets.

COPSA: a computation offloading strategy based on PPO algorithm and self-attention mechanism in MEC-empowered smart factories

With the dawn of Industry 5.0 upon us, the smart factory emerges as a pivotal element, playing a crucial role in the realm of intelligent manufacturing. Meanwhile, mobile edge computing is proposed to alleviate the computational burden presented by substantial workloads in smart factories. Nonetheless, it is very challenging to effectively incorporate edge computing resources to improve the efficiency of resource deployment in smart factories. Accordingly, we devise a novel approach based on Proximal Policy Optimization algorithm with the Self-Attention Mechanism to implement computing resource allocation in MEC-Empowered Smart Factories. More specifically, the self-attention mechanism is incorporated to enable dynamic focus on state information, accelerates convergence and facilitates global control. A great number of experiments conducted on both simulated and real datasets have verified the superiority of our proposed approach compared to the state-of-the-art baselines.

Digital Twin‐Enabled Deep Reinforcement Learning for Safety‐Guaranteed Flocking Motion of UAV Swarm

ABSTRACTMulti‐agent deep reinforcement learning (MADRL) has become a typical paradigm for the flocking motion of UAV swarm in dynamic, stochastic environments. However, sim‐to‐real problems, such as reality gap, training efficiency, and safety issues, restrict the application of MADRL in flocking motion scenarios. To address these problems, we first propose a digital twin (DT)‐enabled training framework. With the assistance of high‐fidelity digital twin simulation, effective policies can be efficiently trained. Based on the multi‐agent proximal policy optimization (MAPPO) algorithm, we then design the learning approach for flocking motion with matching observation space, action space, and reward function. Afterward, we employ a distributed flocking center estimation algorithm based on position consensus. The estimated center is used as a policy input to improve the aggregation behavior. Moreover, we introduce a repulsion scheme, which applies an additional repulsion force to the action to prevent UAVs from colliding with neighbors and obstacles. Simulation results show that our method performs well in maintaining flocking formation and avoiding collisions, and has better decision‐making ability in near‐realistic environments.

Meta-learning and proximal policy optimization driven two-stage emergency allocation strategy for multi-energy system against typhoon disasters

To achieve resilience improvement of the multi-energy system against typhoon disasters, this study designs a novel two-stage optimization framework that considers the emergency allocation of distributed resources under typhoon disasters to fully exploit the potential of distributed resources for resilience enhancement. Firstly, we formulate the emergency allocation of distributed resources as a Markov decision process. Then, a meta-learning-driven proximal policy optimization method is utilized to solve it. Different from that the existing reinforcement learning methods always ignore the unpredictable change caused by typhoon and keep multi-energy system dynamics invariant, limiting its control performance. The proposed method embeds meta-learning to fine-tune the pre-trained allocation policy to new tasks with high adaptability and few interactions. Finally, comparison results with other benchmark methods are carried out and shows that the proposed method can learn the appropriate resource allocation policy for multi-energy system and achieve better resilience enhancement, yielding fast application efficiency and good generalization ability for emergency fault conditions.

Comparison of Empirical and Reinforcement Learning (RL)-Based Control Based on Proximal Policy Optimization (PPO) for Walking Assistance: Does AI Always Win?

Comparison of Empirical and Reinforcement Learning (RL)-Based Control Based on Proximal Policy Optimization (PPO) for Walking Assistance: Does AI Always Win?

Robust solar sail trajectories using proximal policy optimization

Reinforcement learning is used to design minimum-time trajectories of solar sails subject to the typical sources of uncertainty associated with such a propulsion system, i.e., inaccurate knowledge of the sail’s optical properties and the presence of wrinkles on the sail membrane. A proximal policy optimization (PPO) algorithm is used to train the agent and derive the control policy that associates the optimal sail attitude with each dynamic state. First, the agent is trained assuming deterministic unperturbed dynamics, and the results are compared with optimal solutions found by an indirect optimization method, thus demonstrating the effectiveness of this approach. Next, two stochastic scenarios are analyzed. In the first, the optical coefficients of the sail are assumed to be random variables with Gaussian distribution, which leads to random variations in the sail characteristic acceleration. In the second scenario, wrinkles on the sail membrane are taken into account, resulting in a misalignment of the thrust vector with respect to a perfectly smooth surface. Both phenomena are modeled based on experimental measurements available in the literature in order to perform realistic analyses. In the stochastic scenarios, Monte Carlo simulations are performed using the trained policies, demonstrating that the reinforcement learning approach is capable of finding near time-optimal solutions, while also being robust to the sources of uncertainty considered.

Proximal Policy Optimization with Population-based Variable Neighborhood Search Algorithm for Coordinating Photo-Etching and Acid-Etching Processes in Sustainable Storage Chip Manufacturing

In the complex process of manufacturing storage chips, the photo-etching and acid-etching stages play a crucial role, significantly affecting energy consumption and environmental impact. This paper introduces a novel Bi-Level Programming Model for Storage Chip Manufacturing (BLPM-SCM) aimed at optimizing the coordination between these two stages. The upper-level model focuses on minimizing the time it takes to complete wafer production, while the lower-level model seeks to reduce the number of acid-etching tanks used, thereby balancing production efficiency with resource utilization. To address the inherent complexity of the bi-level model, we present a hybrid meta-heuristic algorithm that combines Proximal Policy Optimization (PPO) with a Population-based Variable Neighborhood Search (PVNS) method. The PPO-PVNS algorithm enhances the intensification phase by adaptively selecting shaking and local search strategies, while PVNS supports the diversification phase, ensuring comprehensive exploration of the search space through iterative updates of the solution population. Extensive numerical experiments demonstrate the algorithm's superior performance and generalization capabilities in optimizing the manufacturing process. It significantly improves the coordination between the photo-etching and acid-etching stages, achieving dual optimization of energy consumption and environmental benefits. Furthermore, this study provides valuable insights and decision-making tools for industry practitioners, offering innovative solutions for scheduling optimization in the semiconductor sector and promoting more sustainable and efficient production practices.

Probing an LSTM-PPO-Based reinforcement learning algorithm to solve dynamic job shop scheduling problem

With the growth of personalized demand and the continuous improvement in social productivity, the large-scale and few-variety centralized production model is gradually transitioning towards a personalized model of small batches and multiple varieties, which makes the manufacturing process of the job shop increasingly complex. Furthermore, disruptive events such as machinery failures and rush orders in the job shop increase the uncertainty and variability of the production environment. Traditional scheduling methods are usually based on fixed rules and heuristic algorithms, which are difficult to adapt to constantly changing production environments and demands. This may lead to inaccurate scheduling decisions and hinder the optimal allocation of job shop resources. To solve the dynamic job shop scheduling problem (JSP) more effectively, this paper proposes a Reinforcement Learning (RL) optimization algorithm integrating long short-term memory (LSTM) neural network and proximal policy optimization (PPO). It can dynamically adjust scheduling strategies according to the changing production environment, achieving comprehensive status awareness of the job shop environment to make optimal scheduling decisions. First, a state-aware network framework based on LSTM-PPO is proposed to achieve real-time perception of job shop state changes. Then, the state and action space of the job shop are described within the context of the state-aware network framework. Finally, an experimental environment is established to verify the algorithm’s effectiveness. Training the LSTM-PPO algorithm makes it feasible to achieve better performance than other scheduling methods. By comparing the initial planning time with the actual completion time of the rescheduling decision under different dynamic disturbances, the efficiency of the proposed algorithm is verified for the dynamic JSP

CIPPO: Contrastive Imitation Proximal Policy Optimization for Recommendation Based on Reinforcement Learning

CIPPO: Contrastive Imitation Proximal Policy Optimization for Recommendation Based on Reinforcement Learning

Reinforcement Learning for EV Fleet Smart Charging with On-Site Renewable Energy Sources

In 2020, the transportation sector was the second largest source of carbon emissions in the UK and in Newcastle upon Tyne, responsible for about 33% of total emissions. To support the UK’s target of reaching net zero emissions by 2050, electric vehicles (EVs) are pivotal in advancing carbon-neutral road transportation. Optimal EV charging requires a better understanding of the unpredictable output from on-site renewable energy sources (ORES). This paper proposes an integrated EV fleet charging schedule with a proximal policy optimization method based on a framework for deep reinforcement learning. For the design of the reinforcement learning environment, mathematical models of wind and solar power generation are created. In addition, the multivariate Gaussian distributions derived from historical weather and EV fleet charging data are utilized to simulate weather and charging demand uncertainty in order to create large datasets for training the model. The optimization problem is expressed as a Markov decision process (MDP) with operational constraints. For training artificial neural networks (ANNs) through successive transition simulations, a proximal policy optimization (PPO) approach is devised. The optimization approach is deployed and evaluated on a real-world scenario comprised of council EV fleet charging data from Leicester, UK. The results show that due to the design of the rewards function and system limitations, the charging action is biased towards the time of day when renewable energy output is maximum (midday). The charging decision by reinforcement learning improves the utilization of renewable energy by 2–4% compared to the random charging policy and the priority charging policy. This study contributes to the reduction in battery charging and discharging, electricity sold to the grid to create benefits and the reduction in carbon emissions.