Reinforcement Learning-based Approach Research Articles

Deep Reinforcement Learning-Based Approach With Varying-Scale Generalization for the Earth Observation Satellite Scheduling Problem Considering Resource Consumptions and Supplements

Deep Reinforcement Learning-Based Approach With Varying-Scale Generalization for the Earth Observation Satellite Scheduling Problem Considering Resource Consumptions and Supplements

Joint Drone Access and LEO Satellite Backhaul for a Space–Air–Ground Integrated Network: A Multi-Agent Deep Reinforcement Learning-Based Approach

The space–air–ground integrated network can provide services to ground users in remote areas by utilizing high-altitude platform (HAP) drones to support stable user access and using low earth orbit (LEO) satellites to provide large-scale traffic backhaul. However, the rapid movement of LEO satellites requires dynamic maintenance of the matching relationship between LEO satellites and HAP drones. Additionally, different traffic types generated at HAP drones hold varying levels of values. Therefore, a tripartite matching problem among LEO satellites, HAP drones, and traffic types jointly considering multi-dimensional characteristics such as remaining visible time, channel condition, handover latency, and traffic storage capacity is formulated as mixed integer nonlinear programming to maximize the average transmitted traffic value. The traffic generation state for HAP drones is modeled as a mixture of stochasticity and determinism, which aligns with real-world scenarios, posing challenges for traditional optimization solvers. Thus, the original problem is decoupled into two independent sub-problems: traffic–drone matching and LEO–drone matching, which are addressed by mathematical simplification and multi-agent deep reinforcement learning with centralized training and decentralized execution, respectively. Simulation results verify the effectiveness and superiority of the proposed tripartite matching approach.

Online EVs Vehicle-to-Grid Scheduling Coordinated with Multi-Energy Microgrids: A Deep Reinforcement Learning-Based Approach

The integration of electric vehicles (EVs) into vehicle-to-grid (V2G) scheduling offers a promising opportunity to enhance the profitability of multi-energy microgrid operators (MMOs). MMOs aim to maximize their total profits by coordinating V2G scheduling and multi-energy flexible loads of end-users while adhering to operational constraints. However, scheduling V2G strategies online poses challenges due to uncertainties such as electricity prices and EV arrival/departure patterns. To address this, we propose an online V2G scheduling framework based on deep reinforcement learning (DRL) to optimize EV battery utilization in microgrids with different energy sources. Firstly, our approach proposes an online scheduling model that integrates the management of V2G and multi-energy flexible demands, modeled as a Markov Decision Process (MDP) with an unknown transition. Secondly, a DRL-based Soft Actor-Critic (SAC) algorithm is utilized to efficiently train neural networks and dynamically schedule EV charging and discharging activities in response to real-time grid conditions and energy demand patterns. Extensive simulations are conducted in case studies to testify to the effectiveness of our proposed approach. The overall results validate the efficacy of the DRL-based online V2G scheduling framework, highlighting its potential to drive profitability and sustainability in multi-energy microgrid operations.

Multi-objective reinforcement learning-based approach for pressurized water reactor optimization

A novel method, the Pareto Envelope Augmented with Reinforcement Learning (PEARL), has been developed to address the challenges posed by multi-objective problems, particularly in the field of engineering where the evaluation of candidate solutions can be time-consuming. PEARL distinguishes itself from traditional policy-based multi-objective Reinforcement Learning methods by learning a single policy, eliminating the need for multiple neural networks to independently solve simpler sub-problems. Several versions inspired from deep learning and evolutionary techniques have been crafted, catering to both unconstrained and constrained problem domains. Curriculum Learning (CL) is harnessed to effectively manage constraints in these versions. PEARL’s performance is first evaluated on classical multi-objective benchmarks. Additionally, it is tested on two practical PWR core Loading Pattern (LP) optimization problems to showcase its real-world applicability. The first problem involves optimizing the Cycle length (LC) and the rod-integrated peaking factor (FΔh) as the primary objectives, while the second problem incorporates the mean average enrichment as an additional objective. Furthermore, PEARL addresses three types of constraints related to boron concentration (Cb), peak pin burnup (Bumax), and peak pin power (Fq). The results are systematically compared against conventional approaches from stochastic optimization. Notably, PEARL, specifically the PEAL-NdS ( ▪ ) variant, efficiently uncovers a Pareto front without necessitating additional efforts from the algorithm designer, as opposed to a single optimization with scaled objectives. It also outperforms the classical approach across multiple performance metrics, including the Hyper-volume. Future works will encompass a sensitivity analysis of hyper-parameters with statistical analysis to optimize the application of PEARL and extend it to more intricate problems.

A Deep Reinforcement Learning-Based Preemptive Approach for Cost-Aware Cloud Job Scheduling

A Deep Reinforcement Learning-Based Preemptive Approach for Cost-Aware Cloud Job Scheduling

Fall prediction, control, and recovery of quadruped robots

When legged robots perform complex tasks in unstructured environments, falls are inevitable due to unknown external disturbances. However, current research mainly focuses on the locomotion control of legged robots without falling. This paper proposes a comprehensive decision-making and control framework to address the falling over of quadruped robots. First, a capturability-based fall prediction algorithm is derived for planar single-contact and 3D multi-contact locomotion with a predefined gait sequence. For safe fall control, a novel contact-implicit trajectory optimization method is proposed to generate both state and input trajectories and contact mode sequences. Specifically, incorporating uncertainty into the system and terrain models enables mitigating the non-smoothness of contact dynamics while improving the robustness of the resulting trajectories. Furthermore, a model-free deep reinforcement learning-based approach is presented to achieve fall recovery after the robot completes a fall. Experimental results demonstrate that the proposed fall prediction algorithm accurately predicts robot falls with up to 95% accuracy approximately 395ms in advance. Compared to classical locomotion controllers, which often struggle to maintain balance under significant pushes or terrain perturbations, the presented framework can autonomously switch to the fall controller approximately 0.06s after the perturbation, effectively preventing falls or achieving recovery with a threefold reduction in touchdown impact velocity. These findings highlight the effectiveness of the proposed framework in enhancing the stability and safety of legged robots in unstructured environments.

Minimizing occupant loads in vehicle crashes through reinforcement learning-based restraint system design: assessing performance and transferability

AbstractThe optimization of mechanical behavior in safety systems during crash scenarios consistently poses challenges in vehicle development. Hence, a reinforcement learning-based approach for optimizing restraint systems in frontal impacts is proposed. The trained agent, which adjusts five parameters simultaneously, is capable of minimizing loads on a seen and unseen anthropomorphic test device on the co-driver position and is thus able of transferring knowledge. A hundred times higher rate of convergence to reach a similar optimum compared to a global optimization algorithm has been achieved.

Fundamentals of Arthroscopic Surgery Training and beyond: a reinforcement learning exploration and benchmark.

This work presents FASTRL, a benchmark set of instrument manipulation tasks adapted to the domain of reinforcement learning and used in simulated surgical training. This benchmark enables and supports the design and training of human-centric reinforcement learning agents which assist and evaluate human trainees in surgical practice. Simulation tasks from the Fundamentals of Arthroscopic Surgery Training (FAST) program are adapted to the reinforcement learning setting for the purpose of training virtual agents that are capable of providing assistance and scoring to the surgical trainees. A skill performance assessment protocol is presented based on the trained virtual agents. The proposed benchmark suite presents an API for training reinforcement learning agents in the context of arthroscopic skill training. The evaluation scheme based on both heuristic and learned reward functions robustly recovers the ground truth ranking on a diverse test set of human trajectories. The presented benchmark enables the exploration of a novel reinforcement learning-based approach to skill performance assessment and in-procedure assistance for simulated surgical training scenarios. The evaluation protocol based on the learned reward model demonstrates potential for evaluating the performance of surgical trainees in simulation.

A Model-Free Deep Reinforcement Learning-Based Approach for Assessment of Real-Time PV Hosting Capacity

Assessments of the hosting capacity of electricity distribution networks are of paramount importance, as they facilitate the seamless integration of rooftop photovoltaic systems into the grid, accelerating the transition towards a more carbon neutral and sustainable system. This paper employs a deep reinforcement learning-based approach to evaluate the real-time hosting capacity of low voltage distribution networks in a model-free manner. The proposed approach only requires real-time customer voltage data and solar irradiation data to provide a fast and accurate estimate of real-time hosting capacity at each customer connection point. This study addresses the imperative for accurate electrical models, which are frequently unavailable, in evaluating the hosting capacity of electricity distribution networks. To meet this challenge, the proposed approach utilizes a deep neural network-based, data-driven model of a low-voltage electricity distribution network. This proposed methodology incorporates model-free elements, enhancing its adaptability and robustness. In addition, a comparative analysis between model-based and model-free hosting capacity assessment methods is presented, highlighting their respective strengths and weaknesses. The utilization of the proposed hosting capacity estimation model enables distribution network service providers to make well-informed decisions regarding grid planning, leading to cost minimization.

Diffusion model-based inverse design for thermal transparency

Generative models in the field of artificial intelligence and their applications and deployment have demonstrated their great strength in the past few years. Of the vast spectrum of generative models, diffusion probabilistic models have proven to be particularly powerful and productive, transforming notions such as text-to-image and text-to-video generation from ideas into practical applications. In our previous works, we proposed a thermal metamaterial-based periodic interparticle interaction mechanism for heat management, with a specific application in thermal transparency. To address the challenging problems associated with the inverse design of thermal metamaterial structures, we employed an autoencoder-based machine learning approach and a reinforcement learning-based approach successfully. In this work, we demonstrate that our particular problems with the inverse design of thermal metamaterial-based periodic lattices for the realization of thermal transparency can also be reframed and efficiently solved by training a generative diffusion probabilistic model that can generate the design parameters corresponding to the desired response. Furthermore, we show that for a specific response, multiple sets of design parameters can be obtained by simply performing multiple inferences with the generative diffusion probabilistic model, enabling us to select the ones that can be more economical to fabricate and implement. Our work is among the first to use a diffusion model for the inverse design of thermal metamaterial-based structures and demonstrates the effectiveness of generating low-dimensional design parameters through a diffusion model.

RLPeri: Accelerating Visual Perimetry Test with Reinforcement Learning and Convolutional Feature Extraction

Visual perimetry is an important eye examination that helps detect vision problems caused by ocular or neurological conditions. During the test, a patient's gaze is fixed at a specific location while light stimuli of varying intensities are presented in central and peripheral vision. Based on the patient's responses to the stimuli, the visual field mapping and sensitivity are determined. However, maintaining high levels of concentration throughout the test can be challenging for patients, leading to increased examination times and decreased accuracy. In this work, we present RLPeri, a reinforcement learning-based approach to optimize visual perimetry testing. By determining the optimal sequence of locations and initial stimulus values, we aim to reduce the examination time without compromising accuracy. Additionally, we incorporate reward shaping techniques to further improve the testing performance. To monitor the patient's responses over time during testing, we represent the test's state as a pair of 3D matrices. We apply two different convolutional kernels to extract spatial features across locations as well as features across different stimulus values for each location. Through experiments, we demonstrate that our approach results in a 10-20% reduction in examination time while maintaining the accuracy as compared to state-of-the-art methods. With the presented approach, we aim to make visual perimetry testing more efficient and patient-friendly, while still providing accurate results.

Resilience-oriented approach of dynamic production and maintenance scheduling optimisation considering operational uncertainty

ABSTRACT One of the main challenges in operating multistate manufacturing systems (MMSs) is maintaining stable and robust production against various disruptions. Therefore, an urgent need exists for an operation and maintenance (O&M) method that optimises MMS resilience, i.e. the capability of withstanding or recover from disruptions of various sources. Consequently, this study proposes an integrated resilience-oriented production and maintenance scheduling approach for MMSs. This approach enhances MMS resilience by reinforcing its adaptivity to the variation in production requirements. Based on a conceptual investigation of operational uncertainty and its mechanism of disruption, this study devotes to (i) formulating a performance loss-based resilience measurement with consideration of operational uncertainties and (ii) proposing a reinforcement learning-based approach to schedule MMS operation and maintenance activities for MMS components of different performance states. An industrial case study of a ferrite phase shifter manufacturing system is subsequently conducted to validate the proposed approach. Results demonstrate the effectiveness of the proposed approach in the resilience optimisation of MMSs.

DDT: A Reinforcement Learning Approach to Dynamic Flow Timeout Assignment in Software Defined Networks

OpenFlow-compliant commodity switches face challenges in efficiently managing flow rules due to the limited capacity of expensive high-speed memories used to store them. The accumulation of inactive flows can disrupt ongoing communication, necessitating an optimized approach to flow rule timeouts. This paper proposes Delayed Dynamic Timeout (DDT), a Reinforcement Learning-based approach to dynamically adjust flow rule timeouts and enhance the utilization of a switch’s flow table(s) for improved efficiency. Despite the dynamic nature of network traffic, our DDT algorithm leverages advancements in Reinforcement Learning algorithms to adapt and achieve flow-specific optimization objectives. The evaluation results demonstrate that DDT outperforms static timeout values in terms of both flow rule match rate and flow rule activity. By continuously adapting to changing network conditions, DDT showcases the potential of Reinforcement Learning algorithms to effectively optimize flow rule management. This research contributes to the advancement of flow rule optimization techniques and highlights the feasibility of applying Reinforcement Learning in the context of SDN.

Hybrid supervised and reinforcement learning for the design and optimization of nanophotonic structures.

From higher computational efficiency to enabling the discovery of novel and complex structures, deep learning has emerged as a powerful framework for the design and optimization of nanophotonic circuits and components. However, both data-driven and exploration-based machine learning strategies have limitations in their effectiveness for nanophotonic inverse design. Supervised machine learning approaches require large quantities of training data to produce high-performance models and have difficulty generalizing beyond training data given the complexity of the design space. Unsupervised and reinforcement learning-based approaches on the other hand can have very lengthy training or optimization times associated with them. Here we demonstrate a hybrid supervised learning and reinforcement learning approach to the inverse design of nanophotonic structures and show this approach can reduce training data dependence, improve the generalizability of model predictions, and significantly shorten exploratory training times. The presented strategy thus addresses several contemporary deep learning-based challenges, while opening the door for new design methodologies that leverage multiple classes of machine learning algorithms to produce more effective and practical solutions for photonic design.

Postdisaster Routing of Movable Energy Resources for Enhanced Distribution System Resilience: A Deep Reinforcement Learning-Based Approach

Postdisaster Routing of Movable Energy Resources for Enhanced Distribution System Resilience: A Deep Reinforcement Learning-Based Approach

Real-Time Joint Regulations of Frequency and Voltage for TSO-DSO Coordination: A Deep Reinforcement Learning-Based Approach

Real-Time Joint Regulations of Frequency and Voltage for TSO-DSO Coordination: A Deep Reinforcement Learning-Based Approach

Reinforcement learning for distributed transient frequency control with stability and safety guarantees

This paper proposes a reinforcement learning-based approach for optimal transient frequency control in power systems with stability and safety guarantees. Building on Lyapunov stability theory and safety-critical control, we derive sufficient conditions on the distributed controller design that ensure the stability and transient frequency safety of the closed-loop system. Our idea of distributed dynamic budget assignment makes these conditions less conservative than those in recent literature, so that they can impose less stringent restrictions on the search space of control policies. We construct neural network controllers that parameterize such control policies and use reinforcement learning to train an optimal one. Simulations on the IEEE 39-bus network illustrate the guaranteed stability and safety properties of the controller along with its significantly improved optimality.

Load balancing and topology dynamic adjustment strategy for power information system network: a deep reinforcement learning-based approach

As power information systems play an increasingly critical role in modern society, higher requirements are placed on the performance and reliability of their network infrastructure. In order to cope with the growing data traffic and network attack threats in the power information system, we select the power information system data center network as the research object and design an overall system solution based on software defined network, including the application layer, control layer and infrastructure layer. A typical fat tree network topology is simulated and analyzed. We define the load balancing and network topology dynamic adjustment problem as a Markov decision process, and design a data flow path acquisition method based on breadth-first search to construct the action space of each host. Then, a deep reinforcement learning algorithm based on deep Q-network, priority experience replay and target network is introduced to provide solutions for optimizing the performance of power information systems and responding to network attacks. Simulation results show that the proposed method is better than the traditional equal-cost multi-path algorithm in terms of average bandwidth utilization, average jitter and average packet loss, and can reduce the probability of network nodes being attacked by more than 11%.

Addressing Challenges in Dynamic Modeling of Stewart Platform using Reinforcement Learning-Based Control Approach

In this paper, we focus on enhancing the performance of the controller utilized in the Stewart platform by investigating the dynamics of the platform. Dynamic modeling is crucial for control and simulation, yet challenging for parallel robots like the Stewart platform due to closed-loop kinematics. We explore classical methods to solve its inverse dynamical model, but conventional approaches face difficulties, often resulting in simplified and inaccurate models. To overcome this limitation, we propose a novel approach by replacing the classical feedforward inverse dynamic block with a reinforcement learning (RL) agent, which, to our knowledge, has not been tried yet in the context of the Stewart platform control. Our proposed methodology utilizes a hybrid control topology that combines RL with existing classical control topologies and inverse kinematic modeling. We leverage three deep reinforcement learning (DRL) algorithms and two model-based RL algorithms to achieve improved control performance, highlighting the versatility of the proposed approach. By incorporating the learned feedforward control topology into the existing PID controller, we demonstrate enhancements in the overall control performance of the Stewart platform. Notably, our approach eliminates the need for explicit derivation and solving of the inverse dynamic model, overcoming the drawbacks associated with inaccurate and simplified models. Through several simulations and experiments, we validate the effectiveness of our reinforcement learning-based control approach for the dynamic modeling of the Stewart platform. The results highlight the potential of RL techniques in overcoming the challenges associated with dynamic modeling in parallel robot systems, promising improved control performance. This enhances accuracy and reduces the development time of control algorithms in real-world applications. Nonetheless, it requires a simulation step before practical implementations.

Multi-agent deep reinforcement learning-based maintenance optimization for multi-dependent component systems

Manufacturing systems consist of a set of interdependent components. However, addressing the dependence between these components remains a challenge in both maintenance modeling and the optimization process. In this paper, we propose a multi-agent deep reinforcement learning-based maintenance approach for a manufacturing system, taking into consideration both stochastic and economic dependencies between components. In this manner, we introduce a novel state interactions model, suggesting that the degradation state of one component may influence the degradation process of others. Subsequently, a maintenance planning approach based on multi-agent deep reinforcement learning is developed to optimize maintenance decisions in both fully and partially observed states. The deployed multi-agent deep reinforcement algorithm, specifically Weighted QMIX, ensures scalability and efficient consideration of state interactions and economic dependencies between components. The feasibility and performance of the proposed maintenance approach are investigated through various numerical studies. When compared to traditional maintenance approaches, such as value iteration method, Dueling Double Deep Q Network, and Multi-Agent Deep Q Network, our proposed approach consistently demonstrates superior results.