Application Of Reinforcement Learning Research Articles

A DRL-Based Satellite Service Allocation Method in LEO Satellite Networks

Satellite computing represents a recent computational paradigm in the development of low Earth orbit (LEO) satellites. It aims to augment the capabilities of LEO satellites beyond their current transparent relay functions by enabling real-time processing, thereby providing low-latency computational services to end users. In LEO constellations, a significant deployment of computationally capable satellites is orchestrated to offer enhanced computational resources. Challenges arise in the optimal allocation of terminal services to the most suitable satellite due to overlapping coverage among neighboring satellites, compounded by constraints on satellite energy and computational resources. The satellite service allocation (SSA) problem is recognized as NP-hard, yet assessing allocation methods through results allows for the application of deep reinforcement learning (DRL) to obtain improved solutions, partially addressing the SSA challenge. In this paper, we introduce a satellite computing capability model to quantify satellite computational resources. A DRL model is proposed to address service demands, computational resources, and resolve service allocation conflicts, strategically placing each service on appropriate servers. Through simulation experiments, numerical results demonstrate the superiority of our proposed method over baseline approaches in service allocation and satellite resource utilization, showcasing advancements in this field.

Control of Qubit Dynamics Using Reinforcement Learning

The progress in machine learning during the last decade has had a considerable impact on many areas of science and technology, including quantum technology. This work explores the application of reinforcement learning (RL) methods to the quantum control problem of state transfer in a single qubit. The goal is to create an RL agent that learns an optimal policy and thus discovers optimal pulses to control the qubit. The most crucial step is to mathematically formulate the problem of interest as a Markov decision process (MDP). This enables the use of RL algorithms to solve the quantum control problem. Deep learning and the use of deep neural networks provide the freedom to employ continuous action and state spaces, offering the expressivity and generalization of the process. This flexibility helps to formulate the quantum state transfer problem as an MDP in several different ways. All the developed methodologies are applied to the fundamental problem of population inversion in a qubit. In most cases, the derived optimal pulses achieve fidelity equal to or higher than 0.9999, as required by quantum computing applications. The present methods can be easily extended to quantum systems with more energy levels and may be used for the efficient control of collections of qubits and to counteract the effect of noise, which are important topics for quantum sensing applications.

Application of reinforcement learning to deduce nuclear power plant severe accident scenario

Severe accident scenarios for nuclear power plants are determined through probabilistic safety analysis (PSA). In this process, it is possible to identify the failure sequence of specific components, but assessing the impact of component failure time on the severity of accident remains a challenge. In this study, a novel approach is presented that utilizes machine learning methodologies such as reinforcement learning (RL) to complement traditional PSA. The proposed process was validated by comparing whether the most severe accident scenarios obtained through critical accident simulations can be reproduced by a RL, since this is a novel use of machine learning techniques. The comparison shows feasibility of exploring critical accident scenarios using RL. To implement the reinforcement learning methodology based on the existing system code, supervised learning model that can predict the remaining time of reactor vessel failure was implemented in this study. Based on this prediction model and data from existing catastrophic accident simulation, RL was implemented. The results obtained from RL were subsequently validated with the results of severe accident code simulation. In summary, new methodology for applying machine learning techniques to the nuclear accident analysis process was presented, and the feasibility and potential of the proposed methodology were discussed.

Application of deep reinforcement learning in electricity demand response market: Demand response decision-making of load aggregator

With the large-scale integration of renewable energy generation, developing the potential of demand response is becoming more and more significant. However, the number of consumers is huge, the electricity consumption behaviors are various and the information of consumers is incomplete, thus there are great difficulties in modeling and processing demand response by the conventional mathematical methods. Therefore, how to accurately predict the response behaviors of consumers and select the appropriate consumers for demand response is worthy of in-depth discussion. In this paper, the decision-making method of the load aggregators who participant in demand response market on behalf of the small and medium-sized consumers based on Deep Q-network algorithm is proposed, supporting the aggregators to properly guide the potential demand response consumers. At the same time, a dynamic consumer classification method based on self-organizing maps algorithm is also proposed, supporting the aggregators to accurately predict the response behaviors of the consumers. Simulation results show that the proposed method can effectively realize the classification of the consumers and result in a more beneficial demand response.•The proposed method does not require complete information, such as demand response behavior parameters of the consumers.•Self-organizing maps can realize a dynamic classification of demand response consumers as the case may be.•DQN algorithm can effectively realize the demand response decision-making of aggregators under the condition of incomplete information.

Deep reinforcement learning in chemistry: A review.

Reinforcement learning (RL) has been applied to various domains in computational chemistry and has found wide-spread success. In this review, we first motivate the application of RL to chemistry and list some broad application domains, for example, molecule generation, geometry optimization, and retrosynthetic pathway search. We set up some of the formalism associated with reinforcement learning that should help the reader translate their chemistry problems into a form where RL can be used to solve them. We then discuss the solution formulations and algorithms proposed in recent literature for these problems, the advantages of one over the other, together with the necessary details of the RL algorithms they employ. This article should help the reader understand the state of RL applications in chemistry, learn about some relevant actively-researched open problems, gain insight into how RL can be used to approach them and hopefully inspire innovative RL applications in Chemistry.

Reinforcement Learning for Efficient Power Systems Planning: A Review of Operational and Expansion Strategies

The efficient planning of electric power systems is essential to meet both the current and future energy demands. In this context, reinforcement learning (RL) has emerged as a promising tool for control problems modeled as Markov decision processes (MDPs). Recently, its application has been extended to the planning and operation of power systems. This study provides a systematic review of advances in the application of RL and deep reinforcement learning (DRL) in this field. The problems are classified into two main categories: Operation planning including optimal power flow (OPF), economic dispatch (ED), and unit commitment (UC) and expansion planning, focusing on transmission network expansion planning (TNEP) and distribution network expansion planning (DNEP). The theoretical foundations of RL and DRL are explored, followed by a detailed analysis of their implementation in each planning area. This includes the identification of learning algorithms, function approximators, action policies, agent types, performance metrics, reward functions, and pertinent case studies. Our review reveals that RL and DRL algorithms outperform conventional methods, especially in terms of efficiency in computational time. These results highlight the transformative potential of RL and DRL in addressing complex challenges within power systems.

Automating the assembly planning process to enable design for assembly using reinforcement learning

AbstractThis paper introduces a new concept for the automation of the assembly planning process, to enable Design for Assembly (DfA). The approach involves the application of reinforcement learning (RL) to assembly sequence planning (ASP) based on a 3D-CAD model. The ASP algorithm determines assembly sequences through assembly by disassembly. The assembly sequence is then used for the generation of subassemblies by considering the product contact information. The approach aims to support the creation of the manufacturing bill of materials (MBOM) by automating the assembly planning process.

Reinforcement learning in building controls: A comparative study of algorithms considering model availability and policy representation

Reinforcement Learning (RL) has presented considerable potential as an advanced control technique in building controls to enable buildings operating more energy-efficient. As various types of RL algorithms have been studied on their performance of building controls, benchmarking these algorithms across the entire spectrum of features is essential to provide an overview and deepen the understanding of RL applications. Therefore, this study aims to compare and analyze the effectiveness of various RL algorithms, encompassing the entire RL categories featured by value-based, policy gradient, actor-critic and model-based RL considering model availability and policy representation. To provide a comprehensive analysis, in addition to the control performance quantified by the cumulative rewards based on the cost function of RL, data demand and robustness of hyperparameter tuning were investigated. The open-source Gym-Eplus framework was selected as the virtual environment to train and test different RL agents. The results showed that both model-free and model-based RL agents outperformed the baseline rule-based control in terms of energy consumption and thermal comfort, and RL agents were capable of evaluating both short-term and long-term rewards to achieve adaptive control optimization continuously along with the online control process. Model-based RL agent improved the data sampling efficiency but presented a relatively sacrificed control performance during the tested summer days.

Quarl: A Learning-Based Quantum Circuit Optimizer

Optimizing quantum circuits is challenging due to the very large search space of functionally equivalent circuits and the necessity of applying transformations that temporarily decrease performance to achieve a final performance improvement. This paper presents Quarl, a learning-based quantum circuit optimizer. Applying reinforcement learning (RL) to quantum circuit optimization raises two main challenges: the large and varying action space and the non-uniform state representation. Quarl addresses these issues with a novel neural architecture and RL-training procedure. Our neural architecture decomposes the action space into two parts and leverages graph neural networks in its state representation, both of which are guided by the intuition that optimization decisions can be mostly guided by local reasoning while allowing global circuit-wide reasoning. Our evaluation shows that Quarl significantly outperforms existing circuit optimizers on almost all benchmark circuits. Surprisingly, Quarl can learn to perform rotation merging—a complex, non-local circuit optimization implemented as a separate pass in existing optimizers.

Reinforcement Learning Algorithm to Optimize Players' Tactical Decisions and Round Planning in Tennis Matches

This paper explores the application of the Ranking Genetic Optimization Reinforcement Learning (RGORL) algorithm to optimize players' tactical decisions and round planning in tennis matches. Leveraging evolutionary principles and reinforcement learning techniques, RGORL offers a data-driven framework for enhancing on-court performance. Extensive simulations demonstrate the algorithm's effectiveness in improving match outcomes, points won percentages, and games won percentages. Results illustrate a steady improvement in fitness scores over successive generations, indicating RGORL's ability to evolve and refine strategies over time. Analysis of tactical decisions reveals the superiority of strategies such as the "Net Approach" in terms of win rates, points won percentages, and games won percentages. Through extensive simulations, RGORL demonstrates a notable improvement in match outcomes, with a maximum increase of 13% in win rates. Analysis of tactical decisions reveals significant enhancements in points won percentages, with improvements of up to 34% observed across various strategies, notably the "Net Approach." Furthermore, the algorithm achieves substantial gains in games won percentages, with increases of up to 25% recorded.

SPACECRAFT RELATIVE ON-OFF CONTROL VIA REINFORCEMEN T LEARNING

The article investigates the task of spacecraft relative control using reactive actuators, the output of which has two states, “on” or “off”. For cases where the resolution of the thrusters does not provide an accurate approximation of linear control laws using a pulse-width thrust modulator, the possibility of applying reinforcement learning methods for direct finding of control laws that map the state vector and the on-off thruster commands has been investigated. To implement such an approach, a model of controlled relative motion of two satellites in the form of a Markov decision process was obtained. The intelligent agent is presented in the form of “actor” and “critic” neural networks, and the architecture of these modules is defined. It is proposed to use a cost function with variable weights of control actions, which allows for optimizing the number of thruster firings explicitly. To improve the control performance, it is proposed to use an extended input vector for the “actor” and “critic” neural networks of the intelligent agent, which, in addition to the state vector, also includes information about the control action on the previous control step and the control step number. To reduce the training time, the agent was pre-trained on the data obtained using conventional control algorithms. Numerical results demonstrate that the reinforcement learning methodology allows the agent to outperform the results provided by the linear controller with the pulse-width modulator in terms of control accuracy, response time, and number of thruster firings.

Reinforcement Learning-Driven Decision-Making for Live Virtual Machine Migration in Fog Computing

Virtualization is an essential mechanism in fog computing that enables elasticity and isolation, which in turn helps achieve resource efficiency. To bring high flexibility in a fog environment, migration of virtual machines from one node to another is required. This can be achieved by live virtual machine migration to reduce downtime and delays. Multiple existing studies have discussed live virtual machine migration in a fog environment. However, these studies have some limitations, such as pre-migrating the virtual machines based on mobility prediction only or based on the load only, which causes an issue of late and early handover. Due to the dynamic nature of fog environments, VM migration decisions require consideration of multiple factors. Hence, there is a need to develop a system that considers multiple factors to decide to migrate a virtual machine or not to solve the issue of early and late handover. This study proposes a novel approach to live virtual machine migration that applies reinforcement learning for decision-making. Experiments show that the proposed approach significantly reduces the latency of time-critical applications. The proposed system, outperforms the existing systems in terms of total average reward. The system outperformed the mobility-only-based system by 97% when tested with two fog nodes and by 80% when tested with sixteen fog nodes in terms of average reward. Further, the proposed system outperforms the load-based system by 50% and 75% when the environment consists of two fog nodes and sixteen fog nodes, respectively. This proved that considering multiple factors in deciding virtual machine migration in a fog environment can be effectively applied in time-critical applications to reduce latency.

Advancing Frontiers: An In-Depth Analysis of Reinforcement Learning Applications in Recommendation System Technologies

This article commences by elucidating the concept and algorithms underpinning reinforcement learning (RL), laying the foundation for a comprehensive understanding of RL's principles. It then draws a detailed comparison between RL and traditional machine learning paradigms, using illustrative examples to highlight the distinctive methodologies and outcomes of these approaches. This comparison not only clarifies the unique attributes of RL but also contextualizes its position within the broader landscape of machine learning techniques. Subsequently, the focus shifts to recommendation systems (RS), where both the conceptual framework and algorithmic foundations are thoroughly examined. This exploration is not limited to the technical mechanics of RS but extends to an appraisal of their diverse applications, showcasing how these systems have become integral in various domains, from e-commerce to content curation. The core of the discussion then converges on the application of RL within the realm of Chinese RS. This section delves into how RL's dynamic learning capabilities and adaptability enhance the functionality and effectiveness of RS, particularly in the context of the unique market dynamics and user behaviors observed in China. The synergy between RL and RS in this context is dissected, offering insights into how RL-driven RS can lead to more personalized, context-aware, and efficient user experiences.

Intelligent Resource Allocation Scheme Using Reinforcement Learning for Efficient Data Transmission in VANET.

Vehicular ad hoc networks (VANETs) use multiple channels to communicate using wireless access in vehicular environment (WAVE) standards to provide a variety of vehicle-related applications. The current IEEE 802.11p WAVE communication channel structure is composed of one control channel (CCH) and several service channels (SCHs). SCHs are used for non-safety data transmission, while the CCH is used for broadcasting beacons, control, and safety. WAVE devices transmit data that alternate between CCHs and SCHs, and each channel is active for a duration called the CCH interval (CCHI) and SCH interval (SCHI), respectively. Currently, both intervals are fixed at 50 ms. However, fixed-length intervals cannot effectively respond to dynamically changing traffic loads. Additionally, when many vehicles are simultaneously using the limited channel resources for data transmission, the network performance significantly degrades due to numerous packet collisions. Herein, we propose an adaptive resource allocation technique for efficient data transmission. The technique dynamically adjusts the SCHI and CCHI to improve network performance. Moreover, to reduce data collisions and optimize the network's backoff distribution, the proposed scheme applies reinforcement learning (RL) to provide an intelligent channel access algorithm. The simulation results demonstrate that the proposed scheme can ensure high throughputs and low transmission delays.

Promoting sustainable tourism by recommending sequences of attractions with deep reinforcement learning

Developing Recommender Systems (RSs) is particularly interesting in the tourist domain, where one or more attractions have to be suggested to users based on preferences, contextual dimensions, and several other constraints. RSs usually rely on the availability of a vast amount of historical information about users’ past activities. However, this is not usually the case in the tourist domain, where acquiring complete and accurate information about the user’s behavior is complex, and providing personalized suggestions is frequently practically impossible. Moreover, even though most available Touristic RSs (T-RSs) are user-focused, the touristic domain also requires the development of systems that can promote a more sustainable form of tourism. The concept of sustainable tourism covers many aspects, from economic, social, and environmental issues to the attention to improving tourists’ experience and the needs of host communities. In this regard, one of the most important aspects is the prevention of overcrowded situations in attractions or locations (over-tourism). For this reason, this paper proposes a different kind of T-RS, which focuses more on the tourists’ impact on the destinations, trying to improve their experiences by offering better visit conditions. Moreover, instead of suggesting the next Point of Interest (PoI) to visit in a given situation, it provides a suggestion about a complete sequence of PoIs (tourist itinerary) that covers an entire day or vacation period. The proposed technique is based on the application of Deep Reinforcement Learning, where the tourist’s reward depends on the specific spatial and temporal context in which the itinerary has to be performed. The solution has been evaluated with a real-world dataset regarding the visits conducted by tourists in Verona (Italy) from 2014 to 2023 and compared with three baselines.

Application of Reinforcement Learning to Dyeing Processes for Residual Dye Reduction

Application of Reinforcement Learning to Dyeing Processes for Residual Dye Reduction

Security Assessment of Industrial Control System Applying Reinforcement Learning

Industrial control systems are often used to assist and manage an industrial operation. These systems’ weaknesses in the various hierarchical structures of the system components and communication backbones make them vulnerable to cyberattacks that jeopardize their security. In this paper, the security of these systems is studied by employing a reinforcement learning extended attack graph to efficiently reveal the subsystems’ flaws. Specifically, an attack graph that mimics the environment is constructed for the system using the state–action–reward–state–action technique, in which the agent is regarded as the attacker. Attackers may cause the greatest amount of system damage with the fewest possible actions if they have the highest cumulative reward. The worst-case assault scheme with a total reward of 42.9 was successfully shown in the results, and the most badly affected subsystems were recognized.

Design of an Assisted Driving System for Obstacle Avoidance Based on Reinforcement Learning Applied to Electrified Wheelchairs

Driving a motorized wheelchair is not without risk and requires high cognitive effort to obtain good environmental perception. Therefore, people with severe disabilities are at risk, potentially lowering their social engagement, and thus, affecting their overall well-being. Therefore, we designed a cooperative driving system for obstacle avoidance based on a trained reinforcement learning (RL) algorithm. The system takes the desired direction and speed from the user via a joystick and the obstacle distribution from a LiDAR placed in front of the wheelchair. Considering both inputs, the system outputs a pair of forward and rotational speeds that ensure obstacle avoidance while being as close as possible to the user commands. We validated it through simulations and compared it with a vector field histogram (VFH). The preliminary results show that the RL algorithm does not disruptively alter the user intention, reduces the number of collisions, and provides better door passages than a VFH; furthermore, it can be integrated on an embedded device. However, it still suffers from higher jerkiness.

Exoatmospheric Evasion Guidance Law with Total Energy Limit via Constrained Reinforcement Learning

Due to the lack of aerodynamic forces, the available propulsion for exoatmospheric pursuit-evasion problem is strictly limited, which has not been thoroughly investigated. This paper focuses on the evasion guidance in an exoatmospheric environment with total energy limit. A Constrained Reinforcement Learning (CRL) method is proposed to solve the problem. Firstly, the acceleration commands of the evader are defined as cost and an Actor-Critic-Cost (AC2) network structure is established to predict the accumulated cost of a trajectory. The learning objective of the agent becomes to maximize cumulative rewards while satisfying the cost constraint. Secondly, a Maximum-Minimum Entropy Learning (M2EL) method is proposed to minimize the randomness of acceleration commands while preserving the agent’s exploration capability. Our approaches address two challenges in the application of reinforcement learning: constraint specification and precise control. The well-trained agent is capable of generating accurate commands while satisfying the specified constraints. The simulation results indicate that the CRL and M2EL methods can effectively control the agent’s energy consumption within the specified constraints. The robustness of the agent under information error is also validated.

Reinforcement Learning Techniques in Optimizing Energy Systems

Reinforcement learning (RL) techniques have emerged as powerful tools for optimizing energy systems, offering the potential to enhance efficiency, reliability, and sustainability. This review paper provides a comprehensive examination of the applications of RL in the field of energy system optimization, spanning various domains such as energy management, grid control, and renewable energy integration. Beginning with an overview of RL fundamentals, the paper explores recent advancements in RL algorithms and their adaptation to address the unique challenges of energy system optimization. Case studies and real-world applications demonstrate the efficacy of RL-based approaches in improving energy efficiency, reducing costs, and mitigating environmental impacts. Furthermore, the paper discusses future directions and challenges, including scalability, interpretability, and integration with domain knowledge. By synthesizing the latest research findings and identifying key areas for further investigation, this paper aims to inform and inspire future research endeavors in the intersection of reinforcement learning and energy system optimization.