Application Of Reinforcement Learning Research Articles

Automation of unstructured production environment by applying reinforcement learning

Implementation of Machine Learning (ML) to improve product and production development processes poses a significant opportunity for manufacturing industries. ML has the capability to calibrate models with considerable adaptability and high accuracy. This capability is specifically promising for applications where classical production automation is too expensive, e.g., for mass customization cases where the production environment is uncertain and unstructured. To cope with the diversity in production systems and working environments, Reinforcement Learning (RL) in combination with lightweight game engines can be used from initial stages of a product and production development process. However, there are multiple challenges such as collecting observations in a virtual environment which can interact similar to a physical environment. This project focuses on setting up RL methodologies to perform path-finding and collision detection in varying environments. One case study is human assembly evaluation method in the automobile industry which is currently manual intensive to investigate digitally. For this case, a mannequin is trained to perform pick and place operations in varying environments and thus automating assembly validation process in early design phases. The next application is path-finding of mobile robots including an articulated arm to perform pick and place operations. This application is expensive to setup with classical methods and thus RL enables an automated approach for this task as well.

Application of Deep Reinforcement Learning to Major Solar Flare Forecasting

In this study, we present the application of deep reinforcement learning to the forecasting of major solar flares. For this, we consider full-disk magnetograms at 00:00 UT from the Solar and Heliospheric Observatory/Michelson Doppler Imager (1996–2010) and the Solar Dynamics Observatory/Helioseismic and Magnetic Imager (2011–2019), as well as Geostationary Operational Environmental Satellite X-ray flare data. We apply Deep Q-Network (DQN) and Double DQN, which are popular deep reinforcement learning methods, to predict “Yes or No” for daily M- and X-class flare occurrence. The reward functions, consisting of four rewards for true positive, false positive, false negative, and true negative, are used for our models. The major results of this study are as follows. First, our deep-learning models successfully predict major solar flares with good skill scores, such as HSS, F1, TSS, and ApSS. Second, the performance of our models depends on the reward function, learning method, and target agent update time. Third, the performance of our deep-learning models is noticeably better than that of a convolutional neural network (CNN) model with the same structure: 0.38 (CNN) to 0.44 (ours) for HSS, 0.47 to 0.52 for F1, 0.53 to 0.59 for TSS, and 0.09 to 0.12 for ApSS.

Emergency load shedding strategy for high renewable energy penetrated power systems based on deep reinforcement learning

Traditional event-driven emergency load shedding determines the quantitative strategy by simulation of a specific set of expected faults, which requires high model accuracy and operation mode matching. However, due to the model complexity of renewable power generators and fluctuating power generation, traditional event-driven load shedding strategy faces the risk of mismatching in high renewable energy penetrated power systems. To address these challenges, this paper proposes an emergency load shedding method based on data-driven strategies and deep reinforcement learning (RL). Firstly, the reason for the possible mismatch of the event-driven load shedding strategy in the renewable power system is analyzed, and a typical mismatch scenario is constructed. Then, the emergency load shedding strategy is transformed into a Markov Decision Process (MDP), and the decision process’s action space, state space, and reward function are designed. On this basis, an emergency control strategy platform based on the Gym framework is established for application of deep reinforcement learning in the power system emergency control strategy. In order to enhance the adaptability and efficiency of the RL intelligence agent to multi-fault scenarios, the Proximal Policy Optimization (PPO) is adopted to optimize the constructed MDP. Finally, the proposed reinforcement learning-based emergency load shedding strategy is trained and verified through a modified IEEE 39-bus system. The results show that the proposed strategy can effectively make correct strategies to restore system frequency in the event-driven load shedding mismatch scenario, and have good adaptability for different faults and operation scenarios.

Dynamic Service Function Chain Deployment and Readjustment Method Based on Deep Reinforcement Learning

With the advent of Software Defined Network (SDN) and Network Functions Virtualization (NFV), network operators can offer Service Function Chain (SFC) flexibly to accommodate the diverse network function (NF) requirements of their users. However, deploying SFCs efficiently on the underlying network in response to dynamic SFC requests poses significant challenges and complexities. This paper proposes a dynamic SFC deployment and readjustment method based on deep Q network (DQN) and M Shortest Path Algorithm (MQDR) to address this problem. We develop a model of the dynamic deployment and readjustment of the SFC problem on the basis of the NFV/SFC network to maximize the request acceptance rate. We transform the problem into a Markov Decision Process (MDP) and further apply Reinforcement Learning (RL) to achieve this goal. In our proposed method (MQDR), we employ two agents that dynamically deploy and readjust SFCs collaboratively to enhance the service request acceptance rate. We reduce the action space for dynamic deployment by applying the M Shortest Path Algorithm (MSPA) and decrease the action space for readjustment from two dimensions to one. By reducing the action space, we decrease the training difficulty and improve the actual training effect of our proposed algorithm. The simulation experiments show that MDQR improves the request acceptance rate by approximately 25% compared with the original DQN algorithm and 9.3% compared with the Load Balancing Shortest Path (LBSP) algorithm.

Applying Reinforcement Learning for Enhanced Cybersecurity against Adversarial Simulation

Cybersecurity is a growing concern in today's interconnected world. Traditional cybersecurity approaches, such as signature-based detection and rule-based firewalls, are often limited in their ability to effectively respond to evolving and sophisticated cyber threats. Reinforcement learning (RL) has shown great potential in solving complex decision-making problems in various domains, including cybersecurity. However, there are significant challenges to overcome, such as the lack of sufficient training data and the difficulty of modeling complex and dynamic attack scenarios hindering researchers' ability to address real-world challenges and advance the state of the art in RL cyber applications. In this work, we applied a deep RL (DRL) framework in adversarial cyber-attack simulation to enhance cybersecurity. Our framework uses an agent-based model to continuously learn from and adapt to the dynamic and uncertain environment of network security. The agent decides on the optimal attack actions to take based on the state of the network and the rewards it receives for its decisions. Our experiments on synthetic network security show that the DRL approach outperforms existing methods in terms of learning optimal attack actions. Our framework represents a promising step towards the development of more effective and dynamic cybersecurity solutions.

Multi‐agent reinforcement learning for process control: Exploring the intersection between fields of reinforcement learning, control theory, and game theory

AbstractThe application of reinforcement learning (RL) in process control has garnered increasing research attention. However, much of the current literature is focused on training and deploying a single RL agent. The application of multi‐agent reinforcement learning (MARL) has not been fully explored in process control. This work aims to: (i) develop a unique RL agent configuration that is suitable in a MARL control system for multiloop control, (ii) demonstrate the efficacy of MARL systems in controlling multiloop process that even exhibit strong interactions, and (iii) conduct a comparative study of the performance of MARL systems trained with different game‐theoretic strategies. First, we propose a design of an RL agent configuration that combines the functionalities of a feedback controller and a decoupler in a control loop. Thereafter, we deploy two such agents to form a MARL system that learns how to control a two‐input, two‐output system that exhibits strong interactions. After training, the MARL system shows effective control performance on the process. With further simulations, we examine how the MARL control system performs with increasing levels of process interaction and when trained with reward function configurations based on different game‐theoretic strategies (i.e., pure cooperation and mixed strategies). The results show that the performance of the MARL system is weakly dependent on the reward function configuration for systems with weak to moderate loop interactions. The MARL system with mixed strategies appears to perform marginally better than MARL under pure cooperation in systems with very strong loop interactions.

A bibliometric analysis and review on reinforcement learning for transportation applications

Transportation is the backbone of the economy and urban development. Improving the efficiency, sustainability, resilience, and intelligence of transportation systems is critical and also challenging. The constantly changing traffic conditions, the uncertain influence of external factors (e.g. weather, accidents), and the interactions among multiple travel modes and multi-type flows result in the dynamic and stochastic natures of transportation systems. The planning, operation, and control of transportation systems require flexible and adaptable strategies in order to deal with uncertainty, non-linearity, variability, and high complexity. In this context, Reinforcement Learning (RL) that enables autonomous decision-makers to interact with the complex environment, learn from the experiences, and select optimal actions has been rapidly emerging as one of the most useful approaches for smart transportation applications. This paper conducts a bibliometric analysis to identify the development of RL-based methods for transportation applications, representative journals/conferences, and leading topics in recent 10 years. Then, this paper presents a comprehensive literature review on applications of RL in transportation based on specific topics. The potential future research directions of RL applications and developments are also discussed.

Safe Reinforcement Learning for Model-Reference Trajectory Tracking of Uncertain Autonomous Vehicles With Model-Based Acceleration

Applying reinforcement learning (RL) algorithms to control systems design remains a challenging task due to the potential unsafe exploration and the low sample efficiency. In this paper, we propose a novel safe model-based RL algorithm to solve the collision-free model-reference trajectory tracking problem of uncertain autonomous vehicles (AVs). Firstly, a new type of robust control barrier function (CBF) condition for collision-avoidance is derived for the uncertain AVs by incorporating the estimation of the system uncertainty with Gaussian process (GP) regression. Then, a robust CBF-based RL control structure is proposed, where the nominal control input is composed of the RL policy and a model-based reference control policy. The actual control input obtained from the quadratic programming problem can satisfy the constraints of collision-avoidance, input saturation and velocity boundedness simultaneously with a relatively high probability. Finally, within this control structure, a Dyna-style safe model-based RL algorithm is proposed, where the safe exploration is achieved through executing the robust CBF-based actions and the sample efficiency is improved by leveraging the GP models. The superior learning performance of the proposed RL control structure is demonstrated through simulation experiments.

Model-Free Safe Reinforcement Learning Through Neural Barrier Certificate

Safety is a critical concern when applying reinforcement learning (RL) to real-world control tasks. However, existing safe RL works either only consider expected safety constraint violations and fail to maintain safety guarantees, or use overly conservative safety certificate tools borrowed from safe control theory, which sacrifices reward optimization and relies on analytic system models. This letter proposes a model-free safe RL algorithm that achieves near-zero constraint violations with high rewards. Our key idea is to jointly learn a policy and a neural barrier certificate under stepwise state constraint setting. The barrier certificate is learned in a model-free manner by minimizing the violations of appropriate barrier properties on transition data collected by the policy. We extend the single-step invariant property of the barrier certificate to a multi-step version and construct the corresponding multi-step invariant loss. This loss balances the bias and variance of the barrier certificate and enhances both the safety and performance of the policy. The policy is optimized under the constraint of the multi-step invariant property using the Lagrangian method. We optimize the policy in a model-free manner by introducing an importance sampling weight in the constraint. We test our algorithm on multiple problems, including classic control tasks, robot collision avoidance, and autonomous driving. Results show that our algorithm achieves near-zero constraint violations and high performance compared to the baselines. Moreover, the learned barrier certificates successfully identify the feasible regions on multiple tasks.

Application of Reinforcement Learning to Realize Highly Flexible Microsystem

We herein report on our recent investigation on the development of flexible control of micro (fluidic) system by the application of machine learning (reinforcement learning). Classical peristaltic micropump is employed as platform of the studies and Q-learning algorism, which is also classical algorism of reinforcement learning, is applied to the system. The acquiring of optimal micropumping behavior and manipulation of microbead in microchannel are demonstrated on the platform. The acquired micropumping sequence realize higher flow rate than typical sequences proposed in earlier studies. It is understood that the unique characteristics of the system are considered to acquire the sequence. The efficient micromanipulation of the microbead is also demonstrated on the same platform, even the microdevice is originally designed for the micropumping. Therefore, it could be concluded that the application of the reinforcement learning to a microsystem could be effective to extend the versatility by bringing out the potential of the system.

Applicability of Deep Reinforcement Learning for Efficient Federated Learning in Massive IoT Communications

To build intelligent model learning in conventional architecture, the local data are required to be transmitted toward the cloud server, which causes heavy backhaul congestion, leakage of personalization, and insufficient use of network resources. To address these issues, federated learning (FL) is introduced by offering a systematical framework that converges the distributed modeling process between local participants and the parameter server. However, the challenging issues of insufficient participant scheduling, aggregation policies, model offloading, and resource management still remain within conventional FL architecture. In this survey article, the state-of-the-art solutions for optimizing the orchestration in FL communications are presented, primarily querying the deep reinforcement learning (DRL)-based autonomy approaches. The correlations between the DRL and FL mechanisms are described within the optimized system architectures of selected literature approaches. The observable states, configurable actions, and target rewards are inquired into to illustrate the applicability of DRL-assisted control toward self-organizing FL systems. Various deployment strategies for Internet of Things applications are discussed. Furthermore, this article offers a review of the challenges and future research perspectives for advancing practical performances. Advanced solutions in these aspects will drive the applicability of converged DRL and FL for future autonomous communication-efficient and privacy-aware learning.

A Data-Driven Approach Towards the Application of Reinforcement Learning Based HVAC Control

Refrigeration applications consume a significant share of total electricity demand, with a high indirect impact on global warming through greenhouse gas emissions. Modern technology can help reduce the high power consumption and optimize the cooling control. This paper presents a case study of machine-learning for controlling a commercial refrigeration system. In particular, an approach to reinforcement learning is implemented, trained and validated utilizing a model of a real chiller plant. The reinforcement-learning controller learns to operate the plant based on its interactions with the modeled environment. The validation demonstrates the functionality of the approach, saving around 7% of the energy demand of the reference control. Limitations of the approach were identified in the discretization of the real environment and further model-based simplifications and should be addressed in future research.

Aero-Engine Modeling and Control Method with Model-Based Deep Reinforcement Learning

Due to the strong representation ability and capability of learning from data measurements, deep reinforcement learning has emerged as a powerful control method, especially for nonlinear systems, such as the aero-engine control system. In this paper, a novel application of deep reinforcement learning (DRL) is presented for aero-engine control. In addition, transition dynamic characteristic information of the aero-engine is extracted from the replay buffer of deep reinforcement learning to train a neural-network dynamic prediction model for the aero-engine. In turn, the dynamic prediction model is used to improve the learning efficiency of reinforcement learning. The practical applicability of the proposed control system is demonstrated by the numerical simulations. Compared with the traditional control system, this novel aero-engine control system has faster response speed, stronger self-learning ability, and avoids the complicated manual parameter adjustment without sacrificing the control performance. Moreover, the dynamic prediction model has satisfactory prediction accuracy, and the model-based method can achieve higher learning efficiency than the model-free method.

Safe deep reinforcement learning in diesel engine emission control.

A deep reinforcement learning application is investigated to control the emissions of a compression ignition diesel engine. The main purpose of this study is to reduce the engine-out nitrogen oxide emissions and to minimize fuel consumption while tracking a reference engine load. First, a physics-based engine simulation model is developed in GT-Power and calibrated using experimental data. Using this model and a GT-Power/Simulink co-simulation, a deep deterministic policy gradient is developed. To reduce the risk of an unwanted output, a safety filter is added to the deep reinforcement learning. Based on the simulation results, this filter has no effect on the final trained deep reinforcement learning; however, during the training process, it is crucial to enforce constraints on the controller output. The developed safe reinforcement learning is then compared with an iterative learning controller and a deep neural network-based nonlinear model predictive controller. This comparison shows that the safe reinforcement learning is capable of accurately tracking an arbitrary reference input while the iterative learning controller is limited to a repetitive reference. The comparison between the nonlinear model predictive control and reinforcement learning indicates that for this case reinforcement learning is able to learn the optimal control output directly from the experiment without the need for a model. However, to enforce output constraint for safe learning reinforcement learning, a simple model of system is required. In this work, reinforcement learning was able to reduce emissions more than the nonlinear model predictive control; however, it suffered from slightly higher error in load tracking and a higher fuel consumption.

Reinforcement Learning in Game Industry—Review, Prospects and Challenges

This article focuses on the recent advances in the field of reinforcement learning (RL) as well as the present state–of–the–art applications in games. First, we give a general panorama of RL while at the same time we underline the way that it has progressed to the current degree of application. Moreover, we conduct a keyword analysis of the literature on deep learning (DL) and reinforcement learning in order to analyze to what extent the scientific study is based on games such as ATARI, Chess, and Go. Finally, we explored a range of public data to create a unified framework and trends for the present and future of this sector (RL in games). Our work led us to conclude that deep RL accounted for roughly 25.1% of the DL literature, and a sizable amount of this literature focuses on RL applications in the game domain, indicating the road for newer and more sophisticated algorithms capable of outperforming human performance.

Multi-Agent Reinforcement Learning for Traffic Signal Control: A Cooperative Approach

The rapid growth of urbanization and the constant demand for mobility have put a great strain on transportation systems in cities. One of the major challenges in these areas is traffic congestion, particularly at signalized intersections. This problem not only leads to longer travel times for commuters, but also results in a significant increase in local and global emissions. The fixed cycle of traffic lights at these intersections is one of the primary reasons for this issue. To address these challenges, applying reinforcement learning to coordinating traffic light controllers has become a highly researched topic in the field of transportation engineering. This paper focuses on the traffic signal control problem, proposing a solution using a multi-agent deep Q-learning algorithm. This study introduces a novel rewarding concept in the multi-agent environment, as the reward schemes have yet to evolve in the following years with the advancement of techniques. The goal of this study is to manage traffic networks in a more efficient manner, taking into account both sustainability and classic measures. The results of this study indicate that the proposed approach can bring about significant improvements in transportation systems. For instance, the proposed approach can reduce fuel consumption by 11% and average travel time by 13%. The results of this study demonstrate the potential of reinforcement learning in improving the coordination of traffic light controllers and reducing the negative impacts of traffic congestion in urban areas. The implementation of this proposed solution could contribute to a more sustainable and efficient transportation system in the future.

Increasing the Flexibility of Hydropower with Reinforcement Learning on a Digital Twin Platform

The increasing demand for flexibility in hydropower systems requires pumped storage power plants to change operating modes and compensate reactive power more frequently. In this work, we demonstrate the potential of applying reinforcement learning (RL) to control the blow-out process of a hydraulic machine during pump start-up and when operating in synchronous condenser mode. Even though RL is a promising method that is currently getting much attention, safety concerns are stalling research on RL for the control of energy systems. Therefore, we present a concept that enables process control with RL through the use of a digital twin platform. This enables the safe and effective transfer of the algorithm’s learning strategy from a virtual test environment to the physical asset. The successful implementation of RL in a test environment is presented and an outlook on future research on the transfer to a model test rig is given.

A deep semi-dense compression network for reinforcement learning based on information theory

Although reinforcement learning (RL) can solve complex tasks after training, it’s difficult to extend trained agents to target environments with perturbations. This lack of generalization ability has caused the dilemma of large-scale application of RL. Rich neural networks have brought huge improvements to deep learning, but these aren’t adapted to RL and even bring negative effects. Therefore, RL algorithms have few choices of neural network, which greatly limits the representation and generalization ability of RL. To overcome these limitations, we propose a deep semi-dense compression network (DSCN) to improve RL generalization ability. First, we perform a structural extension on a general network model for RL. Then based on the information theory, we propose a semi-dense connection to enhance the neural information flow (NIF) of initial features. Finally, drawing on the ideas of course learning, we propose a channel compression approach to filter the redundant information of the deep network. In addition, we innovatively extend the experimental environment and evaluation metrics of the existing platform, which can fully evaluate the performance of DSCN. The experimental results show that our model achieves stable and significant generalization performance improvement.

Railway infrastructure maintenance efficiency improvement using deep reinforcement learning integrated with digital twin based on track geometry and component defects

Railway maintenance is a complex and complicated task in the railway industry due to the number of its components and relationships. Ineffective railway maintenance results in excess cost, defective railway structure and components, longer possession time, poorer safety, and lower passenger comfort. Of the three main maintenance approaches, predictive maintenance is the trendy one, and is proven that it provides the highest efficiency. However, the implementation of predictive maintenance for the railway industry cannot be done without an efficient tool. Normally, railway maintenance is corrective when some things fail or preventive when maintenance is routine. A novel approach using an integration between deep reinforcement learning and digital twin is proposed in this study to improve the efficiency of railway maintenance which other techniques such as supervised and unsupervised learning cannot provide. In the study, Advantage Actor Critic (A2C) is used to develop a reinforcement learning model and agent to fulfill the need of the study. Real-world field data over four years and 30 km. is obtained and applied for developing the reinforcement learning model. Track geometry parameters, railway component defects, and maintenance activities are used as parameters to develop the reinforcement learning model. Rewards (or penalties) are calculated based on maintenance costs and occurring defects. The new breakthrough exhibits that using reinforcement learning integrated with digital twin can reduce maintenance activities by 21% and reduce the occurring defects by 68%. Novelties of the study are the use of A2C which is faster and provides better results than other traditional techniques such as Deep Q-learning (DQN), each track geometry parameter is considered without combining into a track quality index, filed data are used to develop the reinforcement learning model, and seven independent actions are included in the reinforcement learning model. This study is the world’s first to contribute a new guideline for applying reinforcement learning and digital twins to improve the efficiency of railway maintenance, reduce the number of defects, reduce the maintenance cost, reduce the possession time for railway maintenance, improve the overall safety of the railway operation, and improve the passenger comfort which can be seen from its results.

Graph learning-based generation of abstractions for reinforcement learning

AbstractThe application of reinforcement learning (RL) algorithms is often hindered by the combinatorial explosion of the state space. Previous works have leveraged abstractions which condense large state spaces to find tractable solutions. However, they assumed that the abstractions are provided by a domain expert. In this work, we propose a new approach to automatically construct abstract Markov decision processes (AMDPs) for potential-based reward shaping to improve the sample efficiency of RL algorithms. Our approach to constructing abstract states is inspired by graph representation learning methods, it effectively encodes the topological and reward structure of the ground-level MDP. We perform large-scale quantitative experiments on a range of navigation and gathering tasks under both stationary and stochastic settings. Our approach shows improvements of up to 8.5 times in sample efficiency and up to 3 times in run time over the baseline approach. Besides, with our qualitative analyses of the generated AMDPs, we are able to visually demonstrate the capability of our approach to preserve the topological and reward structure of the ground-level MDP.