Reinforcement Learning Environment Research Articles

Reinforcement Learning for EV Fleet Smart Charging with On-Site Renewable Energy Sources

In 2020, the transportation sector was the second largest source of carbon emissions in the UK and in Newcastle upon Tyne, responsible for about 33% of total emissions. To support the UK’s target of reaching net zero emissions by 2050, electric vehicles (EVs) are pivotal in advancing carbon-neutral road transportation. Optimal EV charging requires a better understanding of the unpredictable output from on-site renewable energy sources (ORES). This paper proposes an integrated EV fleet charging schedule with a proximal policy optimization method based on a framework for deep reinforcement learning. For the design of the reinforcement learning environment, mathematical models of wind and solar power generation are created. In addition, the multivariate Gaussian distributions derived from historical weather and EV fleet charging data are utilized to simulate weather and charging demand uncertainty in order to create large datasets for training the model. The optimization problem is expressed as a Markov decision process (MDP) with operational constraints. For training artificial neural networks (ANNs) through successive transition simulations, a proximal policy optimization (PPO) approach is devised. The optimization approach is deployed and evaluated on a real-world scenario comprised of council EV fleet charging data from Leicester, UK. The results show that due to the design of the rewards function and system limitations, the charging action is biased towards the time of day when renewable energy output is maximum (midday). The charging decision by reinforcement learning improves the utilization of renewable energy by 2–4% compared to the random charging policy and the priority charging policy. This study contributes to the reduction in battery charging and discharging, electricity sold to the grid to create benefits and the reduction in carbon emissions.

Deep reinforcement learning-based multi-objective optimization for electricity–gas–heat integrated energy systems

With the increasing global attention on energy efficiency and carbon emissions, the optimization of integrated energy systems (IES) has become the key to improve energy efficiency and reduce pollution emissions. However, most of the existing optimization methods cannot effectively deal with the complexity of high dimensional continuous action space. Therefore, this paper focuses on a novel multi-objective optimization strategy for the electricity–gas–heat integrated energy systems (EGH-IES). Firstly, considering the absorption capacity of wind power and the emission of pollutant gases, a multi-objective optimization model is constructed based on the mechanism model and operation constraints of each device in EGH-IES, in which the integrated operation cost and the environmental factors are taken as optimization objectives. Then, the multi-objective optimization problem is designed as the optimal strategy of interaction learning between agent and environment in reinforcement learning, and the output power of the devices constitutes the action of reinforcement learning. Additionally, the Ornstein–Uhlenbeck process is introduced to enhance the training efficiency and exploration performance of the agent, and the deep deterministic policy gradients (DDPG) algorithm is employed to optimize the action, thus the output power of the appliances could be obtained. Finally, the simulation results show that compared with deep Q network (DQN) method and proximal policy optimization (PPO) method, the reward function value of the proposed method increases by 2.43% and 6.09%, respectively, which represents a reduction in economic cost and pollutant emissions. These verify the effectiveness and superiority of the proposed multi-objective optimization scheme in cost reduction and benefit improvement for the EGH-IES.

A knowledge-guided process planning approach with reinforcement learning

With the widespread application of computer-aided technologies such as CAD and CAM in the manufacturing industry, a growing number of process documents and design documents generate multi-source process knowledge and expert experience. However, due to the diverse and complex representation of process knowledge, there is a need for more effective methods to mine a large amount of multi-source information and to exploit the explicit and implicit relationships between the knowledge contained in process knowledge. Effective knowledge reuse in process planning still needs to be improved. This paper proposes a reinforcement learning approach that combines knowledge graphs and process decision-making activities in process planning to exploit the learning potential of process knowledge graphs. Firstly, a reinforcement learning environment for process planning is introduced to model the process planning decision-making phase as a sequential recommendation of process knowledge. Then, this paper designs in detail the state representation method that combines process decision sequences and potential relationships between processes. This paper also creates the composite reward function that combines the process planning environment. In addition, a new algorithm is proposed for learning the proposed model more efficiently. Experimental results show that the network structure has more accurate recommendation results than other methods.

Priority-based intelligent resolution method of multi-aircraft flight conflicts

Abstract The rising demand for air traffic will inevitably result in a surge in both the number and complexity of flight conflicts, necessitating intelligent strategies for conflict resolution. This study addresses the critical challenges of scalability and real-time performance in multi-aircraft flight conflict resolution by proposing a comprehensive method that integrates a priority ranking mechanism with a conflict resolution model based on the Markov decision process (MDP). Within this framework, the proximity between aircraft in a multi-aircraft conflict set is dynamically assessed to establish a conflict resolution ranking mechanism. The problem of multi-aircraft conflict resolution is formalised through the MDP, encompassing the design of state space, discrete action space and reward function, with the transition function implemented via simulation prediction using model-free methods. To address the positional uncertainty of aircraft in real-time scenarios, the conflict detection mechanism introduces the aircraft’s positional error. A deep reinforcement learning (DRL) environment is constructed incorporating actual airspace structures and traffic densities, leveraging the Actor Critic using Kronecker-factored Trust Region (ACKTR) algorithm to determine resolution actions. The experimental results indicate that with 20–30 aircraft in the airspace, the success rate can reach 94% for the training set and 85% for the test set. Furthermore, this study analyses the impact of varying aircraft numbers on the success rate within a specific airspace scenario. The outcomes of this research provide valuable insights for the automation of flight conflict resolution.

Reinforcement learning-based assimilation of the WOFOST crop model

Crop model assimilation is a crucial technique for improving the accuracy and precision of crop models by integrating observational data and crop models. Although conventional data assimilation methods such as Kalman filtering and variational methods have been widely applied, these methods often face limitations in data quality, model bias, and high computational complexity. This study explored the potential of reinforcement learning (RL) for crop model assimilation, which has the advantage of not requiring large datasets. Based on the WOFOST crop model, two RL environments were constructed: a Daily-Data Driven approach and a Time-Series Driven approach. The Proximal Policy Optimization (PPO) algorithm was used to train these environments for 100,000 iterations. The assimilation results were compared with the commonly used SUBPLEX optimization algorithm using four-year field measurement data and a public dataset with added random errors. Our results demonstrate that the Time-Series Driven RL model achieved assimilation accuracy comparable to the SUBPLEX optimization algorithm, with an average MAE of 0.65 compared to 0.76 for SUBPLEX, and a slight decrease in RMSE, while significantly reducing the computational burden by 365 times. In a multi-year stability test, the Time-Series Driven RL model and SUBPLEX had similar assimilation performance. This study demonstrates the potential of RL for crop model assimilation, providing a novel approach to overcome the limitations of conventional assimilation algorithms. The findings suggest that RL-based crop model assimilation can improve model accuracy and efficiency, with potential for practical applications in precision agriculture.

Reinforcement Learning for Fuselage Shape Control during Aircraft Assembly

Critical safety requirements necessitate ultra-high precision quality control during the assembly of large aerospace components to reduce the mismatch between parts to be joined. Traditional methods use heuristic shape adjustment or surrogate model-based control. These methods are limited by reliance on accurate model learning and inadequate robustness to varying initial assembly conditions. To address these limitations, this paper proposes a model-free reinforcement learning approach for adaptive fuselage shape control during aircraft assembly. The trained reinforcement learning agent directly adjusts the aircraft components in response to their part variations and enables an autonomous system (like AlphaGo) to learn the optimal shape control policy. Specifically, the reinforcement learning environment is built on the finite element simulator. A reward function is developed to capture the optimization objective and introduces a scheme to enforce the original constraints. The proximal policy optimization algorithm is modified to speed up the learning progress and achieve better final performance. In the case study, the root-mean-square gap between components is reduced by 98.4% on average compared with their initial shape mismatch. Our proposed method outperforms the benchmark methods with smaller final shape errors, smaller maximum forces, and lower variations across different test samples.

Optimizing Attention and Cognitive Control Costs Using Temporally Layered Architectures.

The current reinforcement learning framework focuses exclusively on performance, often at the expense of efficiency. In contrast, biological control achieves remarkable performance while also optimizing computational energy expenditure and decision frequency. We propose a decision-bounded Markov decision process (DB-MDP) that constrains the number of decisions and computational energy available to agents in reinforcement learning environments. Our experiments demonstrate that existing reinforcement learning algorithms struggle within this framework, leading to either failure or suboptimal performance. To address this, we introduce a biologically inspired, temporally layered architecture (TLA), enabling agents to manage computational costs through two layers with distinct timescales and energy requirements. TLA achieves optimal performance in decision-bounded environments and in continuous control environments, matching state-of-the-art performance while using a fraction of the computing cost. Compared to current reinforcement learning algorithms that solely prioritize performance, our approach significantly lowers computational energy expenditure while maintaining performance. These findings establish a benchmark and pave the way for future research on energy and time-aware control.

PenGym: Realistic training environment for reinforcement learning pentesting agents

Penetration testing, or pentesting, refers to assessing network system security by trying to identify and exploit any existing vulnerabilities. Reinforcement Learning (RL) has recently become an effective method for creating autonomous pentesting agents. However, RL agents are typically trained in a simulated network environment. This can be challenging when deploying them in a real network infrastructure due to the lack of realism of the simulation-trained agents.In this paper, we present PenGym, a framework for training pentesting RL agents in realistic network environments. The most significant features of PenGym are its support for real pentesting actions, full automation of the network environment creation, and good execution performance. The results of our experiments demonstrated the advantages and effectiveness of using PenGym as a realistic training environment in comparison with a simulation approach (NASim). For the largest scenario, agents trained in the original NASim environment behaved poorly when tested in a real environment, having a high failure rate. In contrast, agents trained in PenGym successfully reached the pentesting goal in all our trials. Even after fixing logical modeling issues in simulation to create the revised version NASim(rev.), experiment results with the largest scenario indicated that agents trained in PenGym slightly outperformed, and were more stable, than those trained in NASim(rev.). Thus, the average number of steps required to reach the pentesting goal was 1.4 to 8 steps better for PenGym. Consequently, PenGym provides a reliable and realistic training environment for pentesting RL agents, eliminating the need to model agent actions via simulation.

Deep learning-based rapid prediction of temperature field and intelligent control of molten pool during directed energy deposition process

In this work, deep learning-based approaches are proposed to provide promising solutions to address the challenges in realizing intelligent manufacturing and digital twins for directed energy deposition (DED) process. Firstly, a rapid and accurate prediction of part temperature is realized by innovatively combining graph neural networks (GNNs) and recurrent neural networks (RNNs). Twenty parts with different structures are selected for demonstration. GPU parallel computing technique is adopted to accelerate the thermal finite element analysis, which is used to quickly construct the simulated graph dataset with sufficient samples. By embedding the memory optimization method into the GNN block, deeper GNNs with more trainable parameters are successfully trained with a 79.4% lower GPU memory footprint, which solves the difficulty of deeper GNNs are hard to train on large graph datasets, and the accuracy of temperature prediction on unseen DED parts is significantly improved. Secondly, for intelligent molten pool regulation, a semi-analytic temperature solution method is used to create an efficient DED environment in reinforcement learning (RL) workflows. The intelligent control of molten pool depth under complex deposition strategy is realized based on the environmental state represented by molten pool images. A tailored convolutional neural networks (CNNs) model is employed as the agent to output varying laser power and continuously interact with the dynamic environment. Compared with the traditional artificial neural network agent, the total reward scored by the CNN agent is improved by 9.7% in the zigzag deposition process, mitigating the fluctuations in the controlled molten pool depths. Moreover, CNNs are more compatible with in-situ thermal images. This work can provide theoretical and technical support for realizing real-time and even ahead-of-time temperature prediction and the corresponding feedback control during DED process.

Method of Motion Planning for Digital Twin Navigation and Cutting of Shearer.

To further enhance the intelligence level of coal mining faces and achieve the autonomous derivation, learning, and optimization of shearer navigation cutting, this paper proposes the methods of shearer digital twin navigation cutting motion planning based on the concept of shearer autonomous navigation cutting technology and intelligent coal mining face digital twins. This study includes the digital twin theory and the construction method of the shearer digital twin navigation cutting motion planning system based on this theory. Based on the digital twin theory, a shearer digital twin navigation cutting motion planning system was constructed. This system supports the service functions of the shearer cutting digital twin, dynamic navigation map digital twin, reinforcement learning environment construction, and motion planning through the physical perception layer, comprehensive data layer, and digital-model fusion analysis layer. Finally, by comparing the effects of the DQN-NAF and DDPG deep reinforcement learning algorithms in the shearer motion planning task within the constructed digital twin environment, the results show that the DQN-NAF algorithm demonstrates better performance and stability in solving the shearer digital twin motion planning task.

Safe Spacecraft Inspection via Deep Reinforcement Learning and Discrete Control Barrier Functions

While reinforcement learning (RL) offers high-performance solutions to complex tasks, its trial-and-error paradigm presents safety concerns when exploring unsafe states may be unacceptable. Safety filtering or run-time assurance (RTA) approaches can be used to monitor the RL agent’s desired control and prevent the agent from taking unsafe actions by satisfying safety constraints during and after training. This study investigates the problem of on-orbit inspection of a chief spacecraft using a deputy spacecraft controlled by an RL agent with safety constraints. Several constraints on the state space are enforced using discrete control barrier functions and an optimization-based RTA system. This assures the safety of the deputy spacecraft and its sensors even with a neural-network-based primary controller. A detailed analysis of RL agent performance for eight separate experiments is presented, including each safety constraint individually, all constraints simultaneously, and no constraints. Results demonstrate that the RL agent can complete the inspection task while adhering to safety constraints in a simulated RL environment. These results also show that the RTA system can aid the RL agent in learning a successful policy over fewer time steps but may lead to higher fuel usage, depending on the constraint(s) enforced.

Reward shaping via expectation maximization method

Reinforcement Learning (RL) has found widespread applications in various decision making tasks. However, it still faces challenges such as the deadly triad, slow convergence, and rewards drop, which limit its practical scope. This paper primarily addresses the issues of slow convergence and rewards drop observed during RL training. Our proposed solution involves a reward shaping method composed of two distinct components, each serving a specific purpose: speeding up training and enhancing stability. These two components are interconnected through hyper-parameters, and it has been observed that the choice of hyper-parameters plays a critical role in determining the final performance of the RL algorithm. To optimize these hyper-parameters effectively, we employ a discrete sampling technique to cover the value ranges of these parameters. This discrete sampling creates a sparse set of data points within the reward matrix. Subsequently, we introduce a fitting approach based on the Expectation Maximization (EM) algorithm to estimate the global maximum of the reward matrix along with the corresponding hyper-parameters combination. This EM method significantly reduces computational complexity. Our extensive experimental results across various RL environments have demonstrated the effectiveness of our proposed method. It successfully mitigates the issue of rewards drop while simultaneously accelerating the convergence speed of the RL algorithm.

Two-step dynamic obstacle avoidance

Dynamic obstacle avoidance (DOA) is a fundamental challenge for any autonomous vehicle, independent of whether it operates in sea, air, or land. This paper proposes a two-step architecture for handling DOA tasks by combining supervised and reinforcement learning (RL). In the first step, we introduce a data-driven approach to estimate the collision risk (CR) of an obstacle using a recurrent neural network, which is trained in a supervised fashion and offers robustness to non-linear obstacle movements. In the second step, we include these CR estimates into the observation space of an RL agent to increase its situational awareness. We illustrate the power of our two-step approach by training different RL agents in a challenging environment that requires to navigate amid multiple obstacles. The non-linear movements of obstacles are exemplarily modeled based on stochastic processes and periodic patterns, although our architecture is suitable for any obstacle dynamics. The experiments reveal that integrating our CR metrics into the observation space doubles the performance in terms of reward, which is equivalent to halving the number of collisions in the considered environment. We also perform a generalization experiment to validate the proposal in an RL environment based on maritime traffic and real-world vessel trajectory data. Furthermore, we show that the architecture’s performance improvement is independent of the applied RL algorithm.

Learning the optimal power flow: Environment design matters

To solve the optimal power flow (OPF) problem, reinforcement learning (RL) emerges as a promising new approach. However, the RL-OPF literature is strongly divided regarding the exact formulation of the OPF problem as an RL environment. In this work, we collect and implement diverse environment design decisions from the literature regarding training data, observation space, episode definition, and reward function choice. In an experimental analysis, we show the significant impact of these environment design options on RL-OPF training performance. Further, we derive some first recommendations regarding the choice of these design decisions. The created environment framework is fully open-source and can serve as a benchmark for future research in the RL-OPF field.

A Reinforcement Learning approach to study climbing plant behaviour

A plant’s structure is the result of constant adaptation and evolution to the surrounding environment. From this perspective, our goal is to investigate the mass and radius distribution of a particular plant organ, namely the searcher shoot, by providing a Reinforcement Learning (RL) environment, that we call Searcher-Shoot, which considers the mechanics due to the mass of the shoot and leaves. We uphold the hypothesis that plants maximize their length, avoiding a maximal stress threshold. To do this, we explore whether the mass distribution along the stem is efficient, formulating a Markov Decision Process. By exploiting this strategy, we are able to mimic and thus study the plant’s behavior, finding that shoots decrease their diameters smoothly, resulting in an efficient distribution of the mass. The strong accordance between our results and the experimental data allows us to remark on the strength of our approach in the analysis of biological systems traits.

Discrete Space Deep Reinforcement Learning Algorithm Based on Support Vector Machine Recursive Feature Elimination

Algorithms for training agents with experience replay have advanced in several domains, primarily because prioritized experience replay (PER) developed from the double deep Q-network (DDQN) in deep reinforcement learning (DRL) has become a standard. PER-based algorithms have achieved significant success in the image and video domains. However, the exceptional results observed in images and videos are not as effective in many domains with simple action spaces and relatively small states, particularly in discrete action spaces with sparse rewards. Moreover, most advanced techniques may improve sampling efficiency using deep learning algorithms rather than reinforcement learning. However, there is growing evidence that deep learning algorithms cannot generalize during training. Therefore, this study proposes an algorithm suitable for discrete action space environments that uses the sample efficiency of PER based on DDQN but incorporates support vector machine recursive feature elimination (SVM-RFE) without enhancing the sampling efficiency through deep learning algorithms. The proposed algorithm exhibited considerable performance improvements in classical OpenAI Gym environments that did not use images or videos as inputs. In particular, simple discrete space environments with reflection symmetry, such as Cart–Pole, exhibited a faster and more stable learning process. These results suggest that the application of SVM-RFE, which leverages the orthogonality of support vector machines (SVMs) across learning patterns, can be appropriate when the data in the reinforcement learning environment demonstrate symmetry.

RCI-Seg: Robust click-based interactive segmentation framework with deep reinforcement learning for biomedical images

Recently, interactive segmentation models have achieved remarkable success in the field of biomedical images. However, these models rely on the accurate and high-quality interaction information provided by users, otherwise the segmentation performance will be seriously affected. This problem is more severe in multi-target biomedical images, which means that an image contains multiple targets of interest. It is extremely challenging for users to always maintain high-quality interactions. In this paper, we propose a novel two-stage segmentation model with robust interaction points for biomedical images. In the first stage, we implement robust interaction points based on user initial interaction points and the deep reinforcement learning (DRL) model. Specifically, we build a reinforcement learning environment to simulate the movement of interaction points with agents, and obtain improved interaction points (clue points) that are beneficial for segmentation. In the second stage, we use a convolutional neural network (CNN) model to achieve segmentation by combining clue points and biomedical image. We validate the performance of our approach on five public biomedical image datasets. The experimental results show that the proposed approach outperforms several SOTA methods in multiple metrics.

An innovative multi-agent approach for robust cyber–physical systems using vertical federated learning

Federated learning presents a compelling approach to training artificial intelligence systems in decentralized settings, prioritizing data safety over traditional centralized training methods. Understanding correlations among higher-level threats exhibiting abnormal behavior in the data stream becomes paramount to developing cyber–physical systems resilient to diverse attacks within a continuous data exchange framework. This work introduces a novel vertical federated multi-agent learning framework to address the challenges of modeling attacker and defender agents in stationary and non-stationary vertical federated learning environments. Our approach uniquely applies synchronous Deep Q-Network (DQN) based agents in stationary environments, facilitating convergence towards optimal strategies. Conversely, in non-stationary contexts, we employ synchronous Advantage Actor–Critic (A2C) based agents, adapting to the dynamic nature of multi-agent vertical federated reinforcement learning (VFRL) environments. This methodology enables us to simulate and analyze the adversarial interplay between attacker and defender agents, ensuring robust policy development. Our exhaustive analysis demonstrates the effectiveness of our approach, showcasing its capability to learn optimal policies in both static and dynamic setups, thus significantly advancing the field of cyber-security in federated learning contexts. To evaluate the effectiveness of our approach, we have done a comparative analysis with its baseline schemes. The findings of our study show significant enhancements compared to the standard methods, confirming the efficacy of our methodology. This progress dramatically enhances the area of cyber-security in the context of federated learning by facilitating the formulation of substantial policies. The proposed scheme attains 15.93%, 32.91%, 31.02%, and 47.26% higher results as compared to the A3C, DDQN, DQN, and Reinforce, respectively.

Offloading Strategy Based on Graph Neural Reinforcement Learning in Mobile Edge Computing

In the mobile edge computing (MEC) architecture, base stations with computational capabilities are subject to service coverage limitations, and the mobility of devices leads to dynamic changes in their connections, directly impacting the offloading decisions of agents. The connections between base stations and mobile devices, as well as the connections between base stations themselves, are abstracted into an MEC structural diagram due to the difficulty of deep reinforcement learning (DRL) in capturing the complex relationships between nodes and their multi-order neighboring nodes in the graph; decisions solely generated by DRL have limitations. To address this issue, this study proposes a hierarchical mechanism strategy based on Graph Neural Reinforcement Learning (M-GNRL) under multiple constraints. Specifically, the MEC structural graph constructed with the current device as an observation point aggregates to learn node features, thus comprehensively considering the contextual information of nodes, and the learned graph information serves as the environment for deep reinforcement learning, effectively integrating a graph neural network (GNN) with DRL. In the M-GNRL strategy, edge features from GNN are introduced into the architecture of the DRL network to enhance the accuracy of agents’ decision-making. Additionally, this study proposes an updated algorithm to obtain graph data that change with observation points. Comparative experiments demonstrate that the M-GNRL algorithm outperforms other baseline algorithms in terms of system cost and convergence performance.

Deep reinforcement learning based fast charging and thermal management optimization of an electric vehicle battery pack

Existing optimal strategies for fast-charging electric vehicle batteries are predominantly at the cell level. The proposed high-fidelity methods for extending available cell models to pack level are associated with a large computational burden, making real-world implementation impossible. Further, fast charging optimization and thermal management problems are dependent. That is, the cooling system reduces battery temperature, allowing for higher charging current, while optimal current minimizes the need for cooling and, in turn, reduces thermal system power consumption. There is a lack of studies where fast charging optimization and battery thermal management problems are jointly solved. Therefore, this paper proposes a simulation study using a deep reinforcement learning (RL) approach that concurrently solves fast charging and thermal management problems for a battery pack with low computational complexity. In this regard, we formulate each cell using an electro-thermal-aging model, which accounts for the heat exchange between adjacent cells. The electro-thermal-aging model plays the role of the environment for RL and is not the focus of novelty in this work. The RL agent is then trained to output the optimal charging current and coolant mass flow rate. Moreover, the proposed methodology is examined through a numerical study where the outperformance of our model is showcased by comparing it with baseline algorithms of model predictive control (MPC) and CC–CV. Consequently, three battery packs comprising 20, 444, and 7104 cells are used, respectively. We demonstrate that RL requires less than a second to finish the simulation for 20 cells, while MPC requires more than 80 min. In addition, RL keeps the cells’ core temperature below 33 °C, but MPC results reach 40 °C. The RL performance in charging packs with 444 and 7104 cells is then compared with that of CC–CV. In terms of computation time, RL and CC–CV are nearly the same, while regarding the average core and surface temperature of the cells as well as the cell aging, RL attains better outcomes, extending the battery pack’s life by up to two years after 1000 fast charging cycles.