Application Of Deep Reinforcement Learning Research Articles

Deep Reinforcement Learning-Based Retinal Imaging in Alzheimer's Disease: Potential and Perspectives.

Alzheimer's disease (AD) remains a global health challenge in the 21st century due to its increasing prevalence as the major cause of dementia. State-of-the-art artificial intelligence (AI)-based tests could potentially improve population-based strategies to detect and manage AD. Current retinal imaging demonstrates immense potential as a non-invasive screening measure for AD, by studying qualitative and quantitative changes in the neuronal and vascular structures of the retina that are often associated with degenerative changes in the brain. On the other hand, the tremendous success of AI, especially deep learning, in recent years has encouraged its incorporation with retinal imaging for predicting systemic diseases. Further development in deep reinforcement learning (DRL), defined as a subfield of machine learning that combines deep learning and reinforcement learning, also prompts the question of how it can work hand in hand with retinal imaging as a viable tool for automated prediction of AD. This review aims to discuss potential applications of DRL in using retinal imaging to study AD, and their synergistic application to unlock other possibilities, such as AD detection and prediction of AD progression. Challenges and future directions, such as the use of inverse DRL in defining reward function, lack of standardization in retinal imaging, and data availability, will also be addressed to bridge gaps for its transition into clinical use.

Conceptual Reinforcement Learning for Language-Conditioned Tasks

Despite the broad application of deep reinforcement learning (RL), transferring and adapting the policy to unseen but similar environments is still a significant challenge. Recently, the language-conditioned policy is proposed to facilitate policy transfer through learning the joint representation of observation and text that catches the compact and invariant information across various environments. Existing studies of language-conditioned RL methods often learn the joint representation as a simple latent layer for the given instances (episode-specific observation and text), which inevitably includes noisy or irrelevant information and cause spurious correlations that are dependent on instances, thus hurting generalization performance and training efficiency. To address the above issue, we propose a conceptual reinforcement learning (CRL) framework to learn the concept-like joint representation for language-conditioned policy. The key insight is that concepts are compact and invariant representations in human cognition through extracting similarities from numerous instances in real-world. In CRL, we propose a multi-level attention encoder and two mutual information constraints for learning compact and invariant concepts. Verified in two challenging environments, RTFM and Messenger, CRL significantly improves the training efficiency (up to 70%) and generalization ability (up to 30%) to the new environment dynamics.

AutoCost: Evolving Intrinsic Cost for Zero-Violation Reinforcement Learning

Safety is a critical hurdle that limits the application of deep reinforcement learning to real-world control tasks. To this end, constrained reinforcement learning leverages cost functions to improve safety in constrained Markov decision process. However, constrained methods fail to achieve zero violation even when the cost limit is zero. This paper analyzes the reason for such failure, which suggests that a proper cost function plays an important role in constrained RL. Inspired by the analysis, we propose AutoCost, a simple yet effective framework that automatically searches for cost functions that help constrained RL to achieve zero-violation performance. We validate the proposed method and the searched cost function on the safety benchmark Safety Gym. We compare the performance of augmented agents that use our cost function to provide additive intrinsic costs to a Lagrangian-based policy learner and a constrained-optimization policy learner with baseline agents that use the same policy learners but with only extrinsic costs. Results show that the converged policies with intrinsic costs in all environments achieve zero constraint violation and comparable performance with baselines.

Implementability improvement of deep reinforcement learning based congestion control in cellular network

The application of deep reinforcement learning to improve the adaptability of congestion control is promising. However, the state-of-the-art method indicates a high packet loss and requires a high CPU to handle a flow. Those hinder the implementation of deep reinforcement learning-based congestion control in the production network. Therefore, we propose modifications in the agent’s deployment design, specifically in the monitoring component, interval, and transport protocol to reduce packet loss and CPU usage. Unfortunately, those agent modifications yield a tradeoff in throughput performance. In order to compensate for the tradeoff, we re-train the policy model using ns-3 (network-simulator-3) as a gym environment and custom reward function to improve the throughput. Our work shows that the proposed method evaluated in cellular networks is able to reduce packet loss by up to 50.7×, suppress CPU (central processing unit) usage by up to 4.13×, and increase the throughput by up to 6.94%. We hope our contribution can drive the adoption of deep reinforcement learning-based congestion control to the production network.

A reinforcement learning approach to airfoil shape optimization

Shape optimization is an indispensable step in any aerodynamic design. However, the inherent complexity and non-linearity associated with fluid mechanics as well as the high-dimensional design space intrinsic to such problems make airfoil shape optimization a challenging task. Current approaches relying on gradient-based or gradient-free optimizers are data-inefficient in that they do not leverage accumulated knowledge, and are computationally expensive when integrating Computational Fluid Dynamics (CFD) simulation tools. Supervised learning approaches have addressed these limitations but are constrained by user-provided data. Reinforcement learning (RL) provides a data-driven approach bearing generative capabilities. We formulate the airfoil design as a Markov decision process (MDP) and investigate a Deep Reinforcement Learning (DRL) approach to airfoil shape optimization. A custom RL environment is developed allowing the agent to successively modify the shape of an initially provided 2D airfoil and to observe the associated changes in aerodynamic metrics such as lift-to-drag (L/D), lift coefficient (Cl) and drag coefficient (Cd). The learning abilities of the DRL agent are demonstrated through various experiments in which the agent’s objective-maximizing L/D, maximizing Cl or minimizing Cd-as well as the initial airfoil shape are varied. Results show that the DRL agent is able to generate high performing airfoils within a limited number of learning iterations. The strong resemblance between the artificially produced shapes and those found in the literature highlights the rationality of the decision-making policy learned by the agent. Overall, the presented approach demonstrates the relevance of DRL to airfoil shape optimization and brings forward a successful application of DRL to a physics-based aerodynamics problem.

Deep reinforcement learning in stock portfolios

This paper investigates stock portfolios by application of Deep Reinforcement Learning (DRL) Models to achieve an optimal tactical asset allocation. The research problem is described as an optimization scenario that seeks to maximize the portfolio risk adjusted returns for a given portfolio asset allocation. The problem is set up with an initial capital investment which is invested in a set of assets. The initial strategic allocation is determined, which in our case is the equal weight allocation, and all of the capital is invested in the set of assets. At each point in time, the assets are reallocated according to the allocation which will increase the portfolio value. Two DRL models are implemented. The performance of the DRL models is compared with the uniform weights portfolio. The results show that, generally, two DRL models have higher cumulative returns.

Deep Reinforcement Learning for Economics: Progress and Challenges

We discuss the application of deep reinforcement learning to economic domains in general, and to bargaining on eBay in particular.

Production Scheduling Using Deep Reinforcement Learning and Discrete Event Simulation

AbstractScheduling in the process industry is a highly demanding task. Having access to optimal production schedules at short notice, for instance, after spontaneous changes, offers numerous advantages in terms of robustness, economics, and ultimately customer satisfaction, as delays are minimized. In this work, we describe our initial efforts to apply and evaluate deep reinforcement learning (DRL) for optimized scheduling in a typical fill‐and‐finish batch production plant in the chemical industry. Our pilot study demonstrates how DRL can be implemented using an approach based on discrete event simulation. We discuss the results and benefits of DRL, compare it to mathematical programming approaches, and outline a potential path forward. Our study suggests that the application of DRL in the chemical industry is a promising research direction and that DRL can complement established methods such as process simulation and mathematical programming.

Application of Deep Reinforcement Learning in Quantum Control

Machine learning technology based on artificial neural network has been successfully applied to solve many scientific problems. One of the most interesting areas of machine learning is reinforcement learning, which has natural applicability to optimization problems in physics. In the quantum control task, it is necessary to find a set of optimal control functions to transfer a quantum system from the initial state to the target state with the highest fidelity possible, which is essentially an optimization task. In this paper, we use Deep Deterministic Policy Gradient algorithm (DDPG) to study the quantum control tasks. We use the algorithm to control the transfer of several quantum systems from one state to another. The results show that DDPG algorithm can find a control strategy to make the fidelity of the final state and the target state of the quantum system be maximum value 1. The results show the potential of DDPG in quantum control.

A general motion control framework for an autonomous underwater vehicle through deep reinforcement learning and disturbance observers

This paper investigates the application of deep reinforcement learning (RL) in the motion control for an autonomous underwater vehicle (AUV), and proposes a novel general motion control framework which separates training and deployment. Firstly, the state space, action space, and reward function are customized under the condition of ensuring generality for various motion control tasks. Next, in order to efficiently learn the optimal motion control policy in the case that the AUV model is imprecise and there are unknown external disturbances, a virtual AUV model composed of the known and determined items of an actual AUV is put forward and a simulation training method is developed on this basis. Then, in the given deployment method, three independent extended state observers (ESOs) are designed to deal with the unknown items in different directions, and the final controller is obtained by compensating the estimated value of ESOs into the output of the optimal motion control policy obtained through simulation training. Finally, soft actor-critic is chosen as deep RL algorithm of the framework, and the generality and effectiveness of the proposed method are verified in four different AUV motion control tasks.

Deep Reinforcement Learn In Robotics Challenges and Applications

Deep reinforcement learning (DRL) has emerged as a strong paradigm in the field of robotics, offering promising solutions to complex decision-making problems. This paper provides an overview of the challenges and applications of DRL in robotics. We discuss the basic concepts of DRL, how it integrates with robotic systems, and the challenges associated with this integration. We also explore various applications of DRL in robotics, highlighting the impact of this technology in areas such as automatic communication, flexibility, and control. By addressing these challenges and demonstrating real-world applications, this paper demonstrates the potential of DRL to shape robotic systems in the future.

Application of Deep Reinforcement Learning to Major Solar Flare Forecasting

In this study, we present the application of deep reinforcement learning to the forecasting of major solar flares. For this, we consider full-disk magnetograms at 00:00 UT from the Solar and Heliospheric Observatory/Michelson Doppler Imager (1996–2010) and the Solar Dynamics Observatory/Helioseismic and Magnetic Imager (2011–2019), as well as Geostationary Operational Environmental Satellite X-ray flare data. We apply Deep Q-Network (DQN) and Double DQN, which are popular deep reinforcement learning methods, to predict “Yes or No” for daily M- and X-class flare occurrence. The reward functions, consisting of four rewards for true positive, false positive, false negative, and true negative, are used for our models. The major results of this study are as follows. First, our deep-learning models successfully predict major solar flares with good skill scores, such as HSS, F1, TSS, and ApSS. Second, the performance of our models depends on the reward function, learning method, and target agent update time. Third, the performance of our deep-learning models is noticeably better than that of a convolutional neural network (CNN) model with the same structure: 0.38 (CNN) to 0.44 (ours) for HSS, 0.47 to 0.52 for F1, 0.53 to 0.59 for TSS, and 0.09 to 0.12 for ApSS.

Emergency load shedding strategy for high renewable energy penetrated power systems based on deep reinforcement learning

Traditional event-driven emergency load shedding determines the quantitative strategy by simulation of a specific set of expected faults, which requires high model accuracy and operation mode matching. However, due to the model complexity of renewable power generators and fluctuating power generation, traditional event-driven load shedding strategy faces the risk of mismatching in high renewable energy penetrated power systems. To address these challenges, this paper proposes an emergency load shedding method based on data-driven strategies and deep reinforcement learning (RL). Firstly, the reason for the possible mismatch of the event-driven load shedding strategy in the renewable power system is analyzed, and a typical mismatch scenario is constructed. Then, the emergency load shedding strategy is transformed into a Markov Decision Process (MDP), and the decision process’s action space, state space, and reward function are designed. On this basis, an emergency control strategy platform based on the Gym framework is established for application of deep reinforcement learning in the power system emergency control strategy. In order to enhance the adaptability and efficiency of the RL intelligence agent to multi-fault scenarios, the Proximal Policy Optimization (PPO) is adopted to optimize the constructed MDP. Finally, the proposed reinforcement learning-based emergency load shedding strategy is trained and verified through a modified IEEE 39-bus system. The results show that the proposed strategy can effectively make correct strategies to restore system frequency in the event-driven load shedding mismatch scenario, and have good adaptability for different faults and operation scenarios.

Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions

Deep reinforcement learning (DRL) has been applied to a variety of problems during the past decade and has provided effective control strategies in high-dimensional and non-linear situations that are challenging to traditional methods. Flourishing applications now spread out into the field of fluid dynamics and specifically active flow control (AFC). In the community of AFC, the encouraging results obtained in two-dimensional and chaotic conditions have raised the interest to study increasingly complex flows. In this review, we first provide a general overview of the reinforcement-learning and DRL frameworks, as well as their recent advances. We then focus on the application of DRL to AFC, highlighting the current limitations of the DRL algorithms in this field, and suggesting some of the potential upcoming milestones to reach, as well as open questions that are likely to attract the attention of the fluid mechanics community.

Aero-Engine Modeling and Control Method with Model-Based Deep Reinforcement Learning

Due to the strong representation ability and capability of learning from data measurements, deep reinforcement learning has emerged as a powerful control method, especially for nonlinear systems, such as the aero-engine control system. In this paper, a novel application of deep reinforcement learning (DRL) is presented for aero-engine control. In addition, transition dynamic characteristic information of the aero-engine is extracted from the replay buffer of deep reinforcement learning to train a neural-network dynamic prediction model for the aero-engine. In turn, the dynamic prediction model is used to improve the learning efficiency of reinforcement learning. The practical applicability of the proposed control system is demonstrated by the numerical simulations. Compared with the traditional control system, this novel aero-engine control system has faster response speed, stronger self-learning ability, and avoids the complicated manual parameter adjustment without sacrificing the control performance. Moreover, the dynamic prediction model has satisfactory prediction accuracy, and the model-based method can achieve higher learning efficiency than the model-free method.

Deep reinforcement learning in recommender systems: A survey and new perspectives

In light of the emergence of deep reinforcement learning (DRL) in recommender systems research and several fruitful results in recent years, this survey aims to provide a timely and comprehensive overview of recent trends of deep reinforcement learning in recommender systems. We start by motivating the application of DRL in recommender systems, followed by a taxonomy of current DRL-based recommender systems and a summary of existing methods. We discuss emerging topics, open issues, and provide our perspective on advancing the domain. The survey serves as introductory material for readers from academia and industry to the topic and identifies notable opportunities for further research.

A data-driven approach to assist offensive and defensive players in optimal decision making

Among all the popular sports, soccer is a relatively long-lasting game with a small number of goals per game. This renders the decision-making cumbersome, since it is not straightforward to evaluate the impact of in-game actions apart from goal scoring. Although several action valuation metrics and counterfactual reasoning have been proposed by researchers in recent years, assisting coaches in discovering the optimal actions in different situations of a soccer game has received little attention of soccer analytics. This work proposes the application of deep reinforcement learning on the event and tracking data of soccer matches to discover the most impactful actions at the interrupting point of a possession. Our optimization framework assists players and coaches in inspecting the optimal action, and on a higher level, we provide for the adjustment required for the teams in terms of their action frequencies in different pitch zones. The optimization results have different suggestions for offensive and defensive teams. For the offensive team, the optimal policy suggests more shots in half-spaces (i.e. long-distance shots). For the defending team, the optimal policy suggests that when locating in wings, defensive players should increase the frequency of fouls and ball outs rather than clearances, and when located in the centre, players should increase the frequency of clearances rather than fouls and ball outs.

Process design and optimization of single mixed-refrigerant processes with the application of deep reinforcement learning

Deep reinforcement learning approach is considered for the design and optimization of SMR (Single Mixed Refrigerant) cycles with which the refrigeration power is minimized. Deep Q-Network (DQN) agent is programmed with Python®, which is interacted with a process simulator UniSim Design®, as an environment, through the delivery of decision with MATLAB®. The optimization goal is achieved by designing a reward function such that specific power requirement is minimized, subject to the constraint of minimum temperature difference for the heat transfer. The case study focuses on the design of the SMR cycle for the liquefaction of lean and rich natural gas feeds. GA (Generic Algorithm) optimization framework is also constructed and applied for the case study. Almost the same or similar computational performance between the DQN and the GA methods is observed for finding optimal solutions. For the case study, the specific power required for the liquefaction process is reduced by 15.7% and 13.4% through DQN optimization for the cases of lean and rich feeds, respectively, compared to base cases. The case study clearly demonstrates the applicability of the reinforcement learning which can effectively deal with the optimization problem having complex interactions and high nonlinearities.

Service Management and Energy Scheduling Toward Low-Carbon Edge Computing

Edge computing has become an alternative low-latency provision of cloud computing thanks to its close-proximity to the users, and the geo-distribution nature of edge servers enables the utilization green energy from the environment on-site. To pursue the goal of low-carbon edge computing, it is desirable to minimize the operational expenditure by scheduling the computing resource and green energy according to the spatially and temporally varying user demands. In this article, inspired by the successful application of deep reinforcement learning (DRL) in diverse domains, we propose a DRL-based edge computing management strategy which continuously explores the states and adaptively makes decisions on service management and energy scheduling, towards long-term cost minimization. Different from model-based solutions, our proposal is a model-free method, without any assumption on statistical knowledge as a priori, and therefore is practical in implementation. To speedup the agent training procedure, we further design a prioritized replay memory by utilizing the model-based solution as a guideline to set the transition priority. Extensive experiment results based on real-world traces validate that our proposed DRL-based strategy can make considerably progress compared to the one-shot greedy strategy, and it can learn the system dynamically to manage the edge computing services at runtime.

Constructing autonomous, personalized, and private working management of smart home products based on deep reinforcement learning

Smart home products have been increasingly popular in markets and design research. Although modern smart home products are interactive and aware, they are not competently autonomous, personalized, and private to satisfy users’ experience in convenience and trustworthiness. As a branch of machine learning, deep reinforcement learning is characterized by its learning through interaction with environments without data exchanges and thus has the potential to resolve this problem. This paper proposes a method based on deep reinforcement learning to construct autonomous, personalized, and private working management of smart home products. This method generally splits into users’ manual management, training in deep reinforcement learning, and autonomous working management. Besides, this paper illustrates this method with a case study of robot vacuum cleaners. Overall, the contribution of this paper lies in the innovative application of deep reinforcement learning, which dynamically interacts with users’ contexts and working conditions, to realize autonomy, personalization, and data privacy of smart home products.