Reinforcement Learning Methodology Research Articles

Expert knowledge data-driven based actor–critic reinforcement learning framework to solve computationally expensive unit commitment problems with uncertain wind energy

With the expansion of power grid, unaffordable computational cost and time will pose serious challenges of time-efficient scheduling in unit commitment problem (UCP). However, existing optimization methods, i.e., mathematical programming methods and meta-heuristic algorithms, are powerless and time-consuming to handle computationally expensive UCP (CEUCP). Thus, reinforcement learning methods with strong inference and time-saving performances are motivated to solve the computationally expensive challenges in tackling CEUCPs. In this paper, a novel expert knowledge data-driven based actor–critic (AC) reinforcement learning methodology is proposed for solving CEUCPs. Specifically, in the proposed AC reinforcement learning methodology, expert knowledge, data-driven surrogate model, and improved meta-heuristic algorithm are integrated for further performance enhancement. Firstly, a novel action selection mechanism (based on the expert knowledge of thermal units characteristic) is integrated into AC to improve the efficiency of network training. Secondly, an improved extreme learning machine (ELM) data-driven surrogate model is proposed to build reward function in AC. In detail, original cost function in reward is replaced by a lightweight ELM model. Shape distance is integrated into ELM for enhancing accuracy. Finally, original marine predators algorithm (MPA) is improved for obtaining optimal dispatching decisions and rewards of AC method quickly and correctly. Original search pattern is replaced by quantum based representation for boosting convergence. The excellent performances of the proposed AC framework are verified by simulations of 10-units, 100-units, and 100-units with wind energy test systems.

Application of reinforcement learning to deduce nuclear power plant severe accident scenario

Severe accident scenarios for nuclear power plants are determined through probabilistic safety analysis (PSA). In this process, it is possible to identify the failure sequence of specific components, but assessing the impact of component failure time on the severity of accident remains a challenge. In this study, a novel approach is presented that utilizes machine learning methodologies such as reinforcement learning (RL) to complement traditional PSA. The proposed process was validated by comparing whether the most severe accident scenarios obtained through critical accident simulations can be reproduced by a RL, since this is a novel use of machine learning techniques. The comparison shows feasibility of exploring critical accident scenarios using RL. To implement the reinforcement learning methodology based on the existing system code, supervised learning model that can predict the remaining time of reactor vessel failure was implemented in this study. Based on this prediction model and data from existing catastrophic accident simulation, RL was implemented. The results obtained from RL were subsequently validated with the results of severe accident code simulation. In summary, new methodology for applying machine learning techniques to the nuclear accident analysis process was presented, and the feasibility and potential of the proposed methodology were discussed.

Floorplanning with Edge-aware Graph Attention Network and Hindsight Experience Replay

In this article, we focus on chip floorplanning, which aims to determine the location and orientation of circuit macros simultaneously, so the chip area and wirelength are minimized. As the highest level of abstraction in hierarchical physical design, floorplanning bridges the gap between the system-level design and the physical synthesis, whose quality directly influences downstream placement and routing. To tackle chip floorplanning, we propose an end-to-end reinforcement learning (RL) methodology with a hindsight experience replay technique. An edge-aware graph attention network (EAGAT) is developed to effectively encode the macro and connection features of the netlist graph. Moreover, we build a hierarchical decoder architecture mainly consisting of transformer and attention pointer mechanism to output floorplan actions. Since the RL agent automatically extracts knowledge about the solution space, the previously learned policy can be quickly transferred to optimize new unseen netlists. Experimental results demonstrate that, compared with state-of-the-art floorplanners, the proposed end-to-end methodology significantly optimizes area and wirelength on public GSRC and MCNC benchmarks.

Spacecraft relative on-off control via reinforcement learning

The article investigates the task of spacecraft relative control using reactive actuators, the output of which has two states, “on” or “off”. For cases where the resolution of the thrusters does not provide an accurate approximation of linear control laws using a pulse-width thrust modulator, the possibility of applying reinforcement learning methods for direct finding of control laws that map the state vector and the on-off thruster commands has been investigated. To implement such an approach, a model of controlled relative motion of two satellites in the form of a Markov decision process was obtained. The intelligent agent is presented in the form of “actor” and “critic” neural networks, and the architecture of these modules is defined. It is proposed to use a cost function with variable weights of control actions, which allows for optimizing the number of thruster firings explicitly. To improve the control performance, it is proposed to use an extended input vector for the “actor” and “critic” neural networks of the intelligent agent, which, in addition to the state vector, also includes information about the control action on the previous control step and the control step number. To reduce the training time, the agent was pre-trained on the data obtained using conventional control algorithms. Numerical results demonstrate that the reinforcement learning methodology allows the agent to outperform the results provided by the linear controller with the pulse-width modulator in terms of control accuracy, response time, and number of thruster firings.

Production rescheduling via explorative reinforcement learning while considering nervousness

Nervousness-aware rescheduling is essential in maximizing the profitability and stability of processes in manufacturing industries. It involves re-optimization to meet scheduling goals while minimizing deviations from the base schedule. However, conventional mathematical optimization becomes impractical due to high computational costs and the inability to handle real-time rescheduling. Here, we propose an online rescheduling agent trained by explorative reinforcement learning that autonomously optimizes schedules while considering schedule nervousness. In a static scheduling environment, our model consistently achieves over 90% of the cost objective with scalability and flexibility. A computational time comparison proves that the reinforcement learning methodology makes near-optimal decisions rapidly, irrespective of the complexity of the scheduling problem. Furthermore, we present several realistic rescheduling scenarios that demonstrate the capability of our methodology. Our study illustrates the significant potential of reinforcement learning methodology in expediting digital transformation and process automation within real-world manufacturing systems.

Artificial intelligence-based position control: reinforcement learning approach in spring mass damper systems

This work examines the use of deep Reinforcement Learning (RL) in mass-spring system position control, providing a fresh viewpoint that goes beyond conventional control techniques. Mass-spring systems are widely used in many sectors and are basic models in control theory. The novel aspect of this approach is the thorough examination of the impact of several optimizer algorithms on the RL methodology, which reveals the optimal control tactics. The research applies a Deep Deterministic Policy Gradient (DDPG) algorithm for continuous action spaces, where the actor and critic networks are important components in assessing the agent’s performance. The RL agent is trained to follow a reference trajectory using the Simulink environment for system modeling. The study provides insights into the agent’s learning approach and performance optimization by evaluating the training process using force-time graphs, reward graphs, and Episode Manager charts. Furthermore, the effect of different combinations of optimizers on the control performance of the agent is examined. The outcomes highlight the importance of optimizer selection in the learning process by revealing significant variations in training times. As a result, a better understanding of the relationship between various optimizers and control performance is provided by this study’s novel application of reinforcement learning in mass-spring system control. The results raise the possibility of more potent methods for controlling complex systems and add to the expanding field of study at the interface of control theory and deep learning.

Exploring optimal pathways for enterprise procurement management systems based on fast neural modeling and semantic segmentation

Corporate procurement management assumes a pivotal role within the contemporary business landscape, yet confronts an array of challenges as markets continue to evolve and globalize. Conventional procurement management systems frequently grapple with issues of inefficiency, resource depletion, and noncompliance, necessitating the exploration of innovative avenues for optimization. This paper delves into the realm of risk mitigation associated with collusion behavior in the administration of intelligent procurement systems, presenting a novel procurement collusion identification model founded on a convolutional neural network (CNN) with reinforcement learning techniques. This framework commences with the application of a CNN and Long Short-Term Memory (LSTM) network for in-depth feature analysis and initial identification of historical procurement data, subsequently leveraging reinforcement learning methodologies to enhance the model's autonomy and intelligence for the purpose of optimization. Throughout the experimental phase, diverse domains of procurement data were meticulously selected for analysis. The empirical findings unequivocally demonstrate the model's proficiency, with an average recognition accuracy of 95.1% across five publicly available datasets. This performance surpasses existing machine learning methodologies employed in contemporary research and common recognition networks, thereby offering a pioneering reference point for the intelligent administration and optimization of future procurement systems.

Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management

The challenge of maintaining optimal comfort in residents while minimizing energy consumption has long been a focal point for researchers and practitioners. As technology advances, reinforcement learning (RL)—a branch of machine learning where algorithms learn by interacting with the environment—has emerged as a prominent solution to this challenge. However, the modern literature exhibits a plethora of RL methodologies, rendering the selection of the most suitable one a significant challenge. This work focuses on evaluating various RL methodologies for saving energy while maintaining adequate comfort levels in a residential setting. Five prominent RL algorithms—Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Deep Q-Network (DQN), Advantage Actor-Critic (A2C), and Soft Actor-Critic (SAC)—are being thoroughly compared towards a baseline conventional control approach, exhibiting their potential to improve energy use while ensuring a comfortable living environment. The integrated comparison between the different RL methodologies emphasizes the subtle strengths and weaknesses of each algorithm, indicating that the best selection relies heavily on particular energy and comfort objectives.

Reinforcement learning-based energy storage management in smart grids

This study investigates the use of reinforcement learning (RL) techniques as a dynamic control mechanism to enhance the management of energy storage in smart grid systems. The research aims to optimize the efficiency of energy storage operations by analyzing collected data from different time intervals in a simulated smart grid scenario. An evaluation of the energy storage status reveals a consistent upward trend in the quantity of stored energy, with a 30% cumulative growth across time intervals. An examination of the demand and supply of the grid indicates a persistent insufficiency of energy, with an average shortfall of 15% in meeting the requirements of the system. Through the use of reinforcement learning (RL) methodologies, the system exhibits a remarkable 450% improvement in cumulative rewards, providing substantiation of its capacity to acquire knowledge and adjust its behavior over time. The system's actions indicate a purposeful shift in strategy, with 75% of instances involving charging procedures, emphasizing a commitment to energy preservation and the buildup of stored energy. Despite a shift in approach, persistent disparities between grid demand and supply need the implementation of more accurate technologies for effective energy management. The findings highlight the effectiveness of using reinforcement learning (RL) for managing energy storage in smart grids. This approach improves energy reserves and optimizes energy storage by altering actions accordingly. These insights contribute to the advancement of adaptive energy management strategies, resulting in the development of sustainable and resilient smart grid infrastructures.

Boosting Deep Reinforcement Learning Agents with Generative Data Augmentation

Data augmentation is a promising technique in improving exploration and convergence speed in deep reinforcement learning methodologies. In this work, we propose a data augmentation framework based on generative models for creating completely novel states and increasing diversity. For this purpose, a diffusion model is used to generate artificial states (learning the distribution of original, collected states), while an additional model is trained to predict the action executed between two consecutive states. These models are combined to create synthetic data for cases of high and low immediate rewards, which are encountered less frequently during the agent’s interaction with the environment. During the training process, the synthetic samples are mixed with the actually observed data in order to speed up agent learning. The proposed methodology is tested on the Atari 2600 framework, producing realistic and diverse synthetic data which improve training in most cases. Specifically, the agent is evaluated on three heterogeneous games, achieving a reward increase of up to 31%, although the results indicate performance variance among the different environments. The augmentation models are independent of the learning process and can be integrated to different algorithms, as well as different environments, with slight adaptations.

An examination of the reinforcement learning methodology of machine learning to establish a schedule considering major KPIs such as delivery date in a high-tech manufacturing line

An examination of the reinforcement learning methodology of machine learning to establish a schedule considering major KPIs such as delivery date in a high-tech manufacturing line

Basal-bolus advisor for type 1 diabetes (T1D) patients using multi-agent reinforcement learning (RL) methodology

This study presents in-silico design and verification of an advanced multi-agent reinforcement learning (RL) strategy for personalized glucose regulation in individuals diagnosed with type 1 diabetes (T1D). The proposed framework realizes a closed-loop system, encompassing a blood glucose (BG) metabolic model acting as virtual patient, and a sophisticated multi-agent soft actor–critic RL controller serving as the basal-bolus advisor. The efficacy of the RL agents in meeting the recommended glycemic targets is benchmarked against traditional therapeutic approaches and rigorously assessed across three distinct scenarios: (A) nominal, three meals per day, (B) robustness against meal disturbances, and (C) robustness against insulin sensitivity disturbances. Key evaluation metrics are minimum, maximum and mean values of glucose concentrations, the temporal distribution across various BG intervals, and the daily averages of both bolus and basal insulin administrations. Empirical results underscore the superior performance of the RL-driven basal-bolus advisor, manifesting in diminished glycemic fluctuations and an augmented duration within the optimal range of 70–180 mg/dL. Specifically, in scenarios A, B, and C, the time within this desired range witnessed a surge from 66.66±34.97% to 92.55±4.05%, 64.13±33.84% to 93.91±6.03%, and 58.85±34.67% to 78.34±13.28%, respectively. The RL paradigm also exhibited a marked proficiency in curtailing severe hyperglycemia incidents (p ≤0.05) and diminishing hypoglycemia occurrences. In scenarios A and B, hypoglycemic incidents were curtailed from 14.2% ±32.27% to 3.77% ±4.01% and 16.59% ±32.42% to 2.63% ±4.09%, respectively. Intriguingly, scenario C recorded zero hypoglycemic incidents for both methodologies, attributed to a decline in insulin sensitivity. Additionally, a statistically significant decrement in the mean daily basal insulin dosage was observed with the RL agent in comparison to traditional therapy (p ≤0.05). Collectively, the outcomes accentuate the potential of the multi-agent RL strategy in enhancing glucose regulation and minimizing the peril of severe hyperglycemia for T1D patients.

A Novel Hybrid Security Framework (HSF) with Vshield Based Firewall to Secure Cloud Computing Environment

Cloud Computing is an emerging technology that provides an enormous amount of computing resources which includes networks, servers and storages which are accessed through the internet. In addition it allows useful provisioning of the resources based on the user’s demands. A crucial aspect of cloud computing infrastructure is to provide secure and reliable services. The main challenge lies in the security issues is to reduce the impact of third party attacks in the cloud computing environment. Hence a novel Hybrid Security Framework(HSF) based on Reinforcement Learning (RL) Methodology with Vshield Firewall is proposed for securing the cloud environment. The RL method is used for deep packet inspection and VShiled based firewall is established to deny the attacks which are malicious when authenticating the signature of incoming packets. The bipartite pattern matching approach is integrated with the RL method to verify the signatures for obtaining the decisions quickly. The simulation results shows that the hybrid security framework is effective when compared with the existing methods by considering response time, resource utilization and denial of malicious attacks. This indicates that our proposed framework achieves not only better security but also attains better efficiency in cloud computing environment.

A Reinforcement Learning Approach to Robust Scheduling of Permutation Flow Shop.

The permutation flow shop scheduling problem (PFSP) stands as a classic conundrum within the realm of combinatorial optimization, serving as a prevalent organizational structure in authentic production settings. Given that conventional scheduling approaches fall short of effectively addressing the intricate and ever-shifting production landscape of PFSP, this study proposes an end-to-end deep reinforcement learning methodology with the objective of minimizing the maximum completion time. To tackle PFSP, we initially model it as a Markov decision process, delineating pertinent states, actions, and reward functions. A notably innovative facet of our approach involves leveraging disjunctive graphs to represent PFSP state information. To glean the intrinsic topological data embedded within the disjunctive graph's underpinning, we architect a policy network based on a graph isomorphism network, subsequently trained through proximal policy optimization. Our devised methodology is compared with six baseline methods on randomly generated instances and the Taillard benchmark, respectively. The experimental results unequivocally underscore the superiority of our proposed approach in terms of makespan and computation time. Notably, the makespan can save up to 183.2 h in randomly generated instances and 188.4 h in the Taillard benchmark. The calculation time can be reduced by up to 18.70 s for randomly generated instances and up to 18.16 s for the Taillard benchmark.

Optimal planning of hybrid energy storage systems using curtailed renewable energy through deep reinforcement learning

Energy management systems are becoming increasingly important to utilize the continuously growing curtailed renewable energy. Promising energy storage systems, such as batteries and green hydrogen, should be employed to maximize the efficiency of energy stakeholders. However, optimal decision-making, i.e., planning the leveraging between different strategies, is confronted with the complexity and uncertainties of large-scale problems. A sophisticated deep reinforcement learning methodology with a policy-based algorithm is proposed to achieve real-time optimal energy storage systems planning under the curtailed renewable energy uncertainty. A quantitative performance comparison proved that the deep reinforcement learning agent outperforms the scenario-based stochastic optimization algorithm, even with a wide action and observation space. A robust performance, with maximizing net profit and a stable system, confirmed the uncertainty rejection capability of the deep reinforcement learning under a large uncertainty of the curtailed renewable energy. Action mapping was performed to visually assess the action the deep reinforcement learning agent took according to the state. The corresponding results confirmed that the deep reinforcement learning agent learns how the deterministic solution performs and demonstrates more than 90% profit accuracy compared to the solution.

Intelligent Control of Wastewater Treatment Plants Based on Model-Free Deep Reinforcement Learning

In this work, deep reinforcement learning methodology takes advantage of transfer learning methodology to achieve a reasonable trade-off between environmental impact and operating costs in the activated sludge process of Wastewater treatment plants (WWTPs). WWTPs include complex nonlinear biological processes, high uncertainty, and climatic disturbances, among others. The dynamics of complex real processes are difficult to accurately approximate by mathematical models due to the complexity of the process itself. Consequently, model-based control can fail in practical application due to the mismatch between the mathematical model and the real process. Control based on the model-free reinforcement deep learning (RL) methodology emerges as an advantageous method to arrive at suboptimal solutions without the need for mathematical models of the real process. However, convergence of the RL method to a reasonable control for complex processes is data-intensive and time-consuming. For this reason, the RL method can use the transfer learning approach to cope with this inefficient and slow data-driven learning. In fact, the transfer learning method takes advantage of what has been learned so far so that the learning process to solve a new objective does not require so much data and time. The results demonstrate that cumulatively achieving conflicting objectives can efficiently be used to approach the control of complex real processes without relying on mathematical models.

Model‐free closed‐loop wind farm control using reinforcement learning with recursive least squares

AbstractWind farms experience significant power losses due to wake interactions between turbines. Research shows that wake steering can alleviate these losses by redirecting the flow through the farm. However, dynamic closed‐loop implementations of wake steering are rarely presented. We present a model‐free closed‐loop control method using reinforcement learning methodology known as policy gradients in combination with recursive least squares to perform real‐time wake steering in a wind farm. We present dynamic simulations of a four‐turbine wind farm row using HAWC2Farm, implementing the reinforcement learning control method for various inflow conditions and controller configurations. By controlling the three most upstream turbines, mean power gains of and (95% confidence interval) are observed in partial wake and full wake conditions respectively at 7.5% turbulence intensity. The study helps to bridge the gap between theoretical wind farm control and real‐world wind farm systems.

Optimal navigation for AGVs: A soft actor–critic-based reinforcement learning approach with composite auxiliary rewards

In this paper, we address the problem of real-time navigation and obstacle avoidance for automated guided vehicles (AGVs) in dynamic environments, which is a primary research area in collaborative control systems for AGVs. To overcome the computational inefficiency of recalculating optimal paths every time, we propose an improved Soft Actor–Critic (SAC)-based reinforcement learning methodology. This methodology utilizes a novel composite auxiliary reward structure and sum-tree prioritized experience replay (SAC-SP) to achieve real-time optimal feedback control. First, we formulate the navigation task as a Markov Decision Process that considers both static and dynamic obstacles. To accelerate the active learning of AGVs, we propose a novel strategy that uses composite auxiliary rewards. Next, we train the AGVs using the proposed SAC-SP methodology to handle real-time navigation with the composite auxiliary reward structure. The well-trained policy network can generate effective on-board optimal feedback actions given obstacle positions, targets, and AGV states. Simulation experiments demonstrate that our proposed method can steer AGVs to the destination with high robustness to original conditions and various obstacle restrictions, generating optimal feedback actions in the shortest amount of time.

FPGA-Based Accelerator for AI-Toolbox Reinforcement Learning Library

In Reinforcement Learning (RL) an agent interacts with the environment based on sequential decisions. This agent receives a reward from the environment according to decisions and tries to maximize the reward. RL is used in several domains such as production, autonomous driving, business management, education, games, healthcare, natural language processing and robotics, among others. RL methodologies require processing large volumes of data and computational power. To speed up these applications, field-programmable gate array (FPGA) are widely employed in the literature. This paper proposes an accelerator for the Markov Decision Process (MDP) implemented in the AI-Toolbox public library using high-level synthesis tools, using the tiger-antelope problem as use case. Our approach shows an acceleration greater than 7x compared to the original version.

Automation of unstructured production environment by applying reinforcement learning

Implementation of Machine Learning (ML) to improve product and production development processes poses a significant opportunity for manufacturing industries. ML has the capability to calibrate models with considerable adaptability and high accuracy. This capability is specifically promising for applications where classical production automation is too expensive, e.g., for mass customization cases where the production environment is uncertain and unstructured. To cope with the diversity in production systems and working environments, Reinforcement Learning (RL) in combination with lightweight game engines can be used from initial stages of a product and production development process. However, there are multiple challenges such as collecting observations in a virtual environment which can interact similar to a physical environment. This project focuses on setting up RL methodologies to perform path-finding and collision detection in varying environments. One case study is human assembly evaluation method in the automobile industry which is currently manual intensive to investigate digitally. For this case, a mannequin is trained to perform pick and place operations in varying environments and thus automating assembly validation process in early design phases. The next application is path-finding of mobile robots including an articulated arm to perform pick and place operations. This application is expensive to setup with classical methods and thus RL enables an automated approach for this task as well.