Deep Reinforcement Learning Method Research Articles

Optimal operation of reverse osmosis desalination process with deep reinforcement learning methods

AbstractThe reverse osmosis (RO) process is a well-established desalination technology, wherein energy-efficient techniques and advanced process control methods significantly reduce production costs. This study proposes an optimal real-time management method to minimize the total daily operation cost of an RO desalination plant, integrating a storage tank system to meet varying daily freshwater demand. Utilizing the dynamic model of the RO process, a cascade structure with two reinforcement learning (RL) agents, namely the deep deterministic policy gradient (DDPG) and deep Q-Network (DQN), is developed to optimize the operation of the RO plant. The DDPG agent, manipulating the high-pressure pump, controls the permeate flow rate to track a reference setpoint value. Simultaneously, the DQN agent selects the optimal setpoint value and communicates it to the DDPG controller to minimize the plant’s operation cost. Monitoring storage tanks, permeate flow rates, and water demand enables the DQN agent to determine the required amount of permeate water, optimizing water quality and energy consumption. Additionally, the DQN agent monitors the storage tank’s water level to prevent overflow or underflow of permeate water. Simulation results demonstrate the effectiveness and practicality of the designed RL agents.

Learning vision-based robotic manipulation tasks sequentially in offline reinforcement learning settings

Abstract With the rise of deep reinforcement learning (RL) methods, many complex robotic manipulation tasks are being solved. However, harnessing the full power of deep learning requires large datasets. Online RL does not suit itself readily into this paradigm due to costly and time-consuming agent-environment interaction. Therefore, many offline RL algorithms have recently been proposed to learn robotic tasks. But mainly, all such methods focus on a single-task or multitask learning, which requires retraining whenever we need to learn a new task. Continuously learning tasks without forgetting previous knowledge combined with the power of offline deep RL would allow us to scale the number of tasks by adding them one after another. This paper investigates the effectiveness of regularisation-based methods like synaptic intelligence for sequentially learning image-based robotic manipulation tasks in an offline-RL setup. We evaluate the performance of this combined framework against common challenges of sequential learning: catastrophic forgetting and forward knowledge transfer. We performed experiments with different task combinations to analyse the effect of task ordering. We also investigated the effect of the number of object configurations and the density of robot trajectories. We found that learning tasks sequentially helps in the retention of knowledge from previous tasks, thereby reducing the time required to learn a new task. Regularisation-based approaches for continuous learning, like the synaptic intelligence method, help mitigate catastrophic forgetting but have shown only limited transfer of knowledge from previous tasks.

Deep reinforcement learning for pairs trading: Evidence from China black series futures

Pair trading is one of the main methods of statistical arbitrage, mainly by taking advantage of the temporary price anomalies between related financial products with long-term equilibrium relationships to obtain arbitrage opportunities. In this paper, based on the co-integration method for the selection of allotment, the deep reinforcement learning method is integrated. The deep reinforcement learning method establishes an autonomous learning model and develops paired trading rules under different cycles. At the same time, the deep neural network model is used to implement the learning and training algorithm. This paper uses the data of Shanghai Commodity Exchange and Dalian Commodity Exchange on black futures, and the period is from January 2, 2014 to December 31, 2021. Three different periods were set for five models, namely simple threshold method (ST), simple threshold method based on pairwise cointegration (CA-ST), simple threshold method based on tripartite Cointegration (CA-ST-ALL), deep reinforcement learning method (DRL), and Deep reinforcement learning method based on pairwise cointegration (CA-DRL), and economic indicators were compared. The final results show that, combining the results of the three time periods, the deep reinforcement learning method based on pairwise cointegration performs better regarding return and risk control, validating the feasibility of deep reinforcement learning in China's future market.

Prism Blockchain Enabled Internet of Things with Deep Reinforcement Learning

This paper presents a Deep Reinforcement Learning based IoT enabled Prism BLockchain. The recent advancements in the field of Internet of Things (IoT) motivate the development of a secure infrastructure for storing and sharing vast amounts of data. Blockchain, a distributed and immutable ledger, is best known as a potential solution to data security and privacy for IoT. The scalability of blockchain, which should optimize the throughput and handle the dynamics of the IoT environment, becomes a challenge due to the enormous amount of IoT data. The critical challenge in scaling blockchain is to guarantee decentralization, latency, and security of the system while optimizing the transaction throughput. This paper presents a deep reinforcement learning (DRL)-based performance optimization for blockchain-enabled IoT. We consider one of the recent promising blockchains, Prism as the underlying blockchain system because of its good performance guarantees. We integrate the IoT data to Prism Blockchain and optimize the performance of the system by leveraging Proximal Policy Optimization (PPO) method. The DRL method helps to optimize the blockchain parameters like mining rate and mined blocks to adapt to the environment dynamics of the IoT system. Our results show that the proposed method can improve the throughput of Prism blockchain based IoT systems while preserving Prism performance guarantees. Our scheme can achieve 1.5 times more system rewards than IoT integrated Prism. In our experimental setup, the proposed scheme could improve the average throughput of the system by about 6,000 transactions per sec compared to Prism.

Knowledge-network-embedded deep reinforcement learning: an innovative way to high-efficiently develop an energy management strategy for the integrated energy system with renewable energy sources and multiple energy storage systems

To achieve efficient energy management in complex integrated energy systems (IESs) with renewable energy sources (RESs) and multiple energy storage systems (ESSs), the study aims to propose a novel approach. Evolutionary-based methods are difficult to find the optimal scheme, while deep reinforcement learning (DRL)-based methods face problems with low training efficiency. To address the limitations, a knowledge-network-embedded DRL method is proposed, which combines the fast-searching capability of genetic algorithm (GA) with the environment-aware decision-making capability of DRL, with the aim of developing a highly efficient energy management strategy in improving user comfort and reducing operating costs. GA is used to solve the IES optimization model for obtaining a solution set that serves as domain knowledge, and a knowledge network is constructed using deep learning (DL). Then, DRL uses this network as an actor network to explore the optimal strategy. The use of GA helps the DRL to enhance training efficiency and find an optimal strategy. Simulations show significant improvements in cost reduction (5.7%) and user comfort (4.2%), along with improved training efficiency (58.3%) compared to baseline DRLs.

A deep reinforcement learning model for dynamic job-shop scheduling problem with uncertain processing time

The dynamic job-shop scheduling problem (DJSP) is a type of scheduling tasks where rescheduling is performed when encountering the uncertainties such as the uncertain operation processing time. However, the current deep reinforcement learning (DRL) scheduling approaches are hard to train convergent scheduling policies as the problem scale increases, which is very important for rescheduling under uncertainty. In this paper, we propose a DRL scheduling method for DJSP based on the proximal policy optimization (PPO) with hybrid prioritized experience replay. The job shop scheduling problem is formulated as a sequential decision-making problem based on Markov Decision Process (MDP) where a novel state representation is designed based on the feasible solution matrix which depicts the scheduling order of a scheduling task, a set of paired priority dispatching rules (PDR) are used as the actions and a new intuitive reward function is established based on the machine idle time. Moreover, a new hybrid prioritized experience replay method for PPO is proposed to reduce the training time where samples with positive temporal-difference (TD) error are replayed. Static experiments on classic benchmark instances show that the make-span obtained by our scheduling agent has been reduced by 1.59% on average than the best known DRL methods. In addition, dynamic experiments demonstrate that the training time of the reused scheduling policy is reduced by 27% compared with the retrained policy when encountering uncertainties such as uncertain operation processing time.

Integrated operations strategies for shared and privately-owned autonomous vehicles: A deep reinforcement learning framework

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%).

Knowledge-Based Effective Dispatch for Job Shop Scheduling

Although Deep Reinforcement Learning (DRL) is promising in solving Job Shop Scheduling Problems (JSPs), existing DRL-based methods still have large optimality gaps when learning job-to-machine solutions. This is mainly because: (i) existing state representations autonomously learned from graph-structured data cannot fully capture node information to support agents in making optimal decisions; and (ii) existing reward functions cannot accurately reflect some actions that will seriously worsen the current state. Aiming to address these issues, we propose a knowledge-based DRL method that selects nine well-known priority dispatching rules (PDRs) as state features, which can achieve effective model training. To avoid feature over-redundancy, we discard significantly correlated features based on the Pearson correlation relationship analysis, which can help to identify the key factors that affect the agents’ decision-making. Furthermore, since it is difficult to design a reward function that can accurately distinguish actions, we mask poor-performing actions based on problem-specific knowledge to prevent them from being selected at the current decision point. Comprehensive experimental results demonstrate the superiority of our approach over four PDRs and four state-of-the-art methods on various benchmarks.

Progressive Hierarchical Deep Reinforcement Learning for defect wafer test

Wafer testing plays a critical role in the wafer fabrication process. It involves using a probe to inspect all the wafers on a map, which presents a Coverage Path Planning (CPP) problem. Deep Reinforcement Learning (DRL) has emerged as a state-of-the-art (SOTA) solution for automating regular wafer testing. However, in the presence of production defects, the wafer maps become irregular, requiring more advanced probe strategies. Unfortunately, SOTA DRL methods cannot be directly applied to defect wafer testing due to high computational complexity, low sample efficiency, and limited transferability between tasks. To address these challenges, we propose a novel DRL solution called the Progressive Hierarchical DRL method for automated defect wafer testing. Specifically, our method utilizes a Visual Memory Attention Model (VMAM) to create a high-level strategy, referred to as a manager, which generates a goal specifying the most crucial image region for the task. Subsequently, a low-level strategy, referred to as a worker, generates precise cruising actions to navigate to the designated area. Additionally, we employ transfer learning to generalize the method to handle more complex defect wafer maps. Our approach outperforms seven SOTA DRL baselines in the domains of CPP and wafer testing. It achieves significant improvements in task duration and wafer visit overlap, ranging from 6.0% to 30.5%. Furthermore, we explore the application of our method to a dual-probe defect wafer testing system.

Intelligent agent for hyperspectral image classification with noisy labels: a deep reinforcement learning framework

ABSTRACT Label accuracy plays an important role in supervised hyperspectral image classification problems. However, labelling can be prone to various types of noise, mainly including human errors and incorrect pseudo-label predictions. Learning from datasets with noisy labels is a critical challenge in practice. To address this issue, we first propose a deep reinforcement learning method named HyperRefine to tackle the label noise problem for hyperspectral image classification. The method utilizes an agent to learn the accuracy of each sample label and determine its suitability to be used in training the classifier model. The agent is trained using a reinforcement signal based on the reward obtained from a small validation set that reflects performance on the target task, where the validation set is generated from the training set by a momentum-based recruiter. Compared to complex hand-crafted methods of label noise cleaning, HyperRefine is easy to implement, yet highly effective. The robustness of the model to different types of label noises is significantly improved by further combining a novel label refinement approach. Experiments on artificially added label noise and pseudo-label noise hyperspectral datasets show that the proposed method can achieve promising performance.

A dual grid-based deep reinforcement learning and computational fluid dynamics method for flow control and its application to nucleate pool boiling

This paper presents a proposition of a dual grid (DG)-based coupled deep reinforcement learning (DRL) and computational fluid dynamics (CFD) method for active flow control. The DG-DRL-CFD method uses a dual-resolution grid, with a coarser grid for the training phase and a finer grid for the testing phase of the DRL-CFD method. Further, after a validation study for our DRL-CFD-based parallelized in-house solvers, this paper presents a performance study for a periodic suction/ejection-based drag reduction for a cylinder in a channel-confined flow. Using an immersed boundary method for CFD on a Cartesian grid, this novel DG-DRL-CFD method is shown to result in almost same accuracy (within 1%) in a substantially reduced computational time as compared to the traditional DRL-CFD method (on the finer uniform grid). Finally, using a sharp interface level set method for an axisymmetric two-phase flow, this paper presents an application of the DR-DRL-CFD method for an oscillating base plate-based enhancement of heat transfer during nucleate pool boiling. As compared to our recent CFD study-based parametric study for the nucleate pool boiling problem, the present DG-DRL-CFD method leads to a profile and frequency of plate-oscillation that results in a larger value of average Nusselt number Nuavg for the plate. The coupled DRL-CFD method leads to plate oscillation profile giving higher enhancement in Nuavg, compared to an open-loop control strategy that involves a parametric sweep involving multiple simulations.

An effective deep actor-critic reinforcement learning method for solving the flexible job shop scheduling problem

An effective deep actor-critic reinforcement learning method for solving the flexible job shop scheduling problem

Off-policy actor-critic deep reinforcement learning methods for alert prioritization in intrusion detection systems

Alert prioritization plays a very important role in network security as it helps security teams manage and respond to the overwhelming volume of alerts generated by intrusion detection systems (IDSs). Although current deep reinforcement learning (DRL) based deep deterministic policy gradient (DDPG) has achieved good performance for alert prioritization, there are still limitations in dealing with DDPG's instability and limited exploration capabilities. In this paper, we propose TD3-AP and SAC-AP, two DRL empowered off-policy actor-critic alert prioritization methods based on twin delayed deep deterministic policy gradient (TD3) and soft actor-critic (SAC), respectively. We model the interaction between an adversary and a defender as a zero-sum game and use a double oracle framework to obtain mixed strategy Nash equilibrium (MSNE). Our objective is to minimize the defender's loss which represents the inability of the defender to investigate alerts generated due to the attacks. To demonstrate the benefits and scalability of the proposed approaches, we conduct extensive experiments on three datasets: MQTT-IoT-IDS2020, DARPA 2000 LLDOS 1.0 and CSE-CIC-IDS2018. The results depict that TD3-AP and SAC-AP exhibit a decrease in the defender's loss by 50% and 14.28%, respectively, in comparison to the alert prioritization method based on DDPG. The improvements are significantly higher when the proposed approaches are compared with traditional alert prioritization methods including Uniform, Snort, Fuzzy and RAP. Additionally, we also provide an analysis of the interpretability of our results through the utilization of SHapley Additive exPlanations (SHAP). We assess the sensitivity of our proposed approaches to different hyperparameters and evaluate the training time and computational resources necessary for their implementation.

Autonomous spacecraft collision avoidance with a variable number of space debris based on safe reinforcement learning

The avoidance of multiple space debris collisions by autonomous spacecraft has garnered significant interests worldwide. Applying deep reinforcement learning (DRL) to autonomous spacecraft collision avoidance problems is still difficult because of limitations on constraint satisfaction and environment state perception, even if DRL is a suitable model-free and data-driven framework. In this research, a state-of-the-art penalized proximal policy optimization (P3O) method is applied to address the spacecraft's autonomous collision avoidance problem, which is formalized as a constrained Markov decision process (CMDP). In contrast with traditional DRL methods, P3O promises to satisfy multiple constraints in actual spaceship operations while also facilitating efficient learning in multi-dimensional, continuous state and action spaces. The scalability of the P3O algorithm is enhanced by combining the feature extraction capabilities and variable-length input capabilities of the long short-term memory (LSTM) networks, enabling the P3O to adapt to a variable number of space debris without the need for network retraining. The performance of the proposed method is compared with other five methods through simulation cases, which verifies the superior performance of the proposed method in terms of scalability, energy consumption, collision probability and constraint satisfaction.

Deep reinforcement learning for PMSG wind turbine control via twin delayed deep deterministic policy gradient (TD3)

AbstractThis article proposes the use of a deep reinforcement learning method—and precisely a variant of the deep deterministic policy gradient (DDPG) method known as twin delayed DDPG, or TD3—for maximum power point tracking in wind energy conversion systems that use permanent magnet synchronous generators (PMSGs). An overview of the TD3 algorithm is provided, together with a detailed description of its implementation and training for the considered application. Simulation results are provided, also including a comparison with a model‐based control method based on feedback linearization and linear‐quadratic regulation. The proposed TD3‐based controller achieves a satisfactory control performance and is more robust to PMSG parameter variations as compared to the presented model‐based method. To the best of the authors' knowledge, this article presents for the first time an approach for generating both speed and current control loops using DRL for wind energy conversion systems.

UISA: User Information Separating Architecture for Commodity Recommendation Policy with Deep Reinforcement Learning

Commodity recommendation contributes an important part of individuals’ daily life. In this context, deep reinforcement learning methods have demonstrated substantial efficacy in enhancing recommender systems’ performance. Nevertheless, several recommender systems directly utilize original feature information as a foundational element for decision-making, which seems simplistic and low efficient. Furthermore, the incorporation of sequential decision-making adds complexity to the task of recommendation. In pursuit of maximizing the long-term sequential returns of recommender systems, our study introduces a novel architecture, named User Information Separating Architecture (UISA). This framework is tailored to align with classic reinforcement learning algorithms and aims to extract the user’s interest value through the discrete processing of both static and dynamic user information. Through integration with deep reinforcement learning, the architecture is oriented towards the maximization of long-term profit and is applicable in sequential recommendation scenarios. We conduct experimental assessments by combining the proposed architecture with proximal policy optimization (PPO) and deep deterministic policy gradient (DDPG) algorithms. The outcomes illustrate marked improvements in commodity recommendation, showcasing enhancements ranging from approximately 5% to 40% in both reward and click-through rate metrics across a self-constructed JDEnv environment and the Virtual Taobao environment. Through comparison experiments, the UISA models demonstrate comparable performance.

Dynamic scheduling mechanism for intelligent workshop with deep reinforcement learning method based on multi-agent system architecture

With the development and changes of industry and market demand, the personalized customization production mode with small batch and multiple batches has gradually become a new production mode. This makes production environment become more complex and dynamic. However, traditional production workshops cannot effectively adapt to this environment. Combining with new technologies, transforming traditional workshops into intelligent workshop to cope with new production mode become an urgent problem. Therefore, this paper proposes a multi-agent manufacturing system based on IoT for intelligent workshop. Meanwhile, this paper takes flexible job shop scheduling problem (FJSP) as a specific production scenario and establishes relevant mathematics model. To build the agent in intelligent workshop, this paper proposes a data-based with combination of virtual and physical agent (DB-VPA) which has information layer, software layer and physical layer. Then, based on the manufacturing system, this paper designs a dynamic scheduling mechanism with deep reinforcement learning (DRL) for intelligent workshop. This method contains three aspects: (1) Modeling production process based on Markov decision process (MDP). (2) Designing communication mechanism for DB-VPAs. (3) Designing scheduling model combining with improved genetic programming and proximal policy optimization (IGP-PPO) which is a DRL method. Finally, relevant experiments are executed in a prototype experiment platform. The experiments indicate that the proposed method has superiority and generality in solving scheduling problem with dynamic events.

Stochastic Economic Lot Scheduling via Self-Attention Based Deep Reinforcement Learning

The Stochastic Economic Lot Scheduling Problem (SELSP) is a difficult dynamic optimization problem with wide industrial applications. Traditional methods such as hyper-heuristics are manually designed based on substantial expert knowledge, which may limit their optimization performance. Recently, Deep Reinforcement Learning (DRL) is shown to be promising in automatically learning scheduling policies for SELSP. However, its performance is still quite far from that of hyper-heuristics, due to the lack of suitable deep models. In this paper, we propose a novel DRL method to learn dynamic scheduling policies for SELSP in an end-to-end fashion. Based on self-attention, our method can effectively extract useful features from raw state information, and is flexible in handling different numbers of products, which is not viable for previous methods. Experiments on a complex biopharmaceutical manufacturing process show that our method outperforms a recent DRL method and state-of-the-art hyper-heuristics. Moreover, the trained policy performs better in environments different from training with demand forecast errors and varying number of products, showing its strong robustness and generalization ability. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The Stochastic Economic Lot Scheduling Problem (SELSP) is an important problem for manufacturing enterprises, which is to optimally balance the production and inventory so as to minimize the total cost. However, SELSP is very challenging to solve due to the involvement of uncertain factors such as customer demands and machine failures. Traditional methods for solving SELSP, such as heuristic policies and hyper-heuristics, heavily rely on human experiences to design and hence the performance could be limited. This paper proposes a Deep Reinforcement Learning (DRL) based method to automatically learn scheduling policy for solving SELSP, which could alleviate the above limitation through a self-attention based feature extraction mechanism and reward based training. Experimental results on a realistic manufacturing process show that our method can deliver higher revenue than conventional manual policy and an existing DRL based method.

Resources Scheduling for Ambient Backscatter Communication-Based Intelligent IIoT: A Collective Deep Reinforcement Learning Method

Resources Scheduling for Ambient Backscatter Communication-Based Intelligent IIoT: A Collective Deep Reinforcement Learning Method

Dynamic Incentive Pricing on Charging Stations for Real-Time Congestion Management in Distribution Network: An Adaptive Model-Based Safe Deep Reinforcement Learning Method

Dynamic Incentive Pricing on Charging Stations for Real-Time Congestion Management in Distribution Network: An Adaptive Model-Based Safe Deep Reinforcement Learning Method