Model-free Deep Reinforcement Learning Research Articles

Model-Free Deep Recurrent Q-Network Reinforcement Learning for Quantum Circuit Architectures Design

Artificial intelligence (AI) technology leads to new insights into the manipulation of quantum systems in the Noisy Intermediate-Scale Quantum (NISQ) era. Classical agent-based artificial intelligence algorithms provide a framework for the design or control of quantum systems. Traditional reinforcement learning methods are designed for the Markov Decision Process (MDP) and, hence, have difficulty in dealing with partially observable or quantum observable decision processes. Due to the difficulty of building or inferring a model of a specified quantum system, a model-free-based control approach is more practical and feasible than its counterpart of a model-based approach. In this work, we apply a model-free deep recurrent Q-network (DRQN) reinforcement learning method for qubit-based quantum circuit architecture design problems. This paper is the first attempt to solve the quantum circuit design problem from the recurrent reinforcement learning algorithm, while using discrete policy. Simulation results suggest that our long short-term memory (LSTM)-based DRQN method is able to learn quantum circuits for entangled Bell–Greenberger–Horne–Zeilinger (Bell–GHZ) states. However, since we also observe unstable learning curves in experiments, suggesting that the DRQN could be a promising method for AI-based quantum circuit design application, more investigation on the stability issue would be required.

Constrained large-scale real-time EV scheduling based on recurrent deep reinforcement learning

The rapid growth of electric vehicles (EVs) is an unstoppable worldwide development trend. An optimal charging strategy for large-scale EVs is able to deal with the randomness of EVs charging and satisfy charging demands of users while ensuring safe and economic operation of the power system. The current centralized and model-based methods failed to overcome the randomness charging problem of the large-scale EVs. Thus, the paper proposes a decentralized approach based on model-free deep reinforcement learning (DRL) to determine the optimal strategy for reducing EVs charging cost considering power limit of the charging station (CS), users' charging demands and fair charging fees. First, a decentralized framework and a dynamic energy boundary (DEB) model of single EV which discretizes the charging demand are proposed. Second, the problem as a Markov Decision Process (MDP) with unknown transition probability is formulated. Moreover, the recurrent deep deterministic policy gradient (RDDPG) based approach is proposed to determine the charging strategy for all charging piles in the CS. Finally, digital simulation studies are conducted to demonstrate the effectiveness of the proposed approach in charging cost reduction and fair charging fees. In addition, the RDDPG-based approach has great scalability which can apply a small-scale model to solve a large-scale problem without being retrained.

Predictive Maintenance Model for IIoT-Based Manufacturing: A Transferable Deep Reinforcement Learning Approach

The Industrial Internet of Things (IIoT) is crucial for accurately assessing the state of complex equipment in order to perform predictive maintenance (PdM) successfully. However, existing IIoT-based PdM frameworks do not consider the influence of various practical yet complex system factors, such as the real-time production states, machine health, and maintenance manpower resources. For this reason, we propose a generic PdM optimization framework to assist maintenance teams in prioritizing and resolving maintenance task conflicts under real-world manufacturing conditions. Specifically, the PdM framework aims to jointly optimize the edge-based machine network uptime and the allocation of manpower resources in a stochastic IIoT-enabled manufacturing environment using the model-free deep reinforcement learning (DRL) methods. Since DRL requires a significant amount of training data, we propose and demonstrate the use of the transfer learning (TL) method to assist DRL in learning more efficiently by incorporating expert demonstrations, termed TL with demonstrations (TLDs). TLD reduces training wall time by 58% compared to baseline methods, and we conduct numerous experiments to illustrate the performance, robustness, and scalability of TLD. Finally, we discuss the general benefits and limitations of the proposed TL method, which are not well addressed in the existing literature but could be beneficial to both researchers and industry practitioners.

Deep Dyna-Reinforcement Learning Based on Random Access Control in LEO Satellite IoT Networks

Random access schemes in satellite Internet-of-Things (IoT) networks are being considered a key technology of new-type machine-to-machine (M2M) communications. However, the complicated situations and long-distance transmission can make the current random access schemes not suitable for the satellite IoT networks. The random access problem in the satellite IoT networks is studied in this article. A novel random access scheme for machine-type-communication devices (MTCDs) is proposed, to maximize the efficiency of random access for contention-based and contention-free random access. Under the set of random access opportunities (RAOs) and limited delay, the random access control model is designed via maximizing efficiency of random access. The model-free deep reinforcement learning (DRL) algorithm is proposed to tackle the problem based on the random access model. Subsequently, the deep Dyna- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> learning algorithm is introduced to deal with the proposed random access control model. In this proposed scheme, the random access model-free DRL algorithm is developed using simulated experience. The proposed algorithms’ performances are discussed, and simulation results show the desirable performance of the proposed DRL methods on different system parameters.

Self- Driving With Deep Reinforcement ( Survey )

We will focus concentrating on the scientific and technological challenges that the development of autonomous vehicles brings to a number of different manufacturers and research organizations (AVs). It is anticipated that automobiles operated by humans will continue to be a common sight on the roads for the foreseeable future; nevertheless, it is possible that autonomous vehicles may coexist in the same traffic environments. In order to keep traffic moving smoothly throughout the various sorts of mixed-use zones, autonomous vehicles (AVs) need to have driving rules and negotiation abilities that are analogous to those used by humans. In order to develop driving abilities that are comparable to those of humans, model-free deep reinforcement learning is used to simulate the actions of a skilled human driver. In this simulation, the difficulty of avoiding static obstacles on a two-lane roadway is investigated .

Semi-Decentralized Network Slicing for Reliable V2V Service Provisioning: A Model-Free Deep Reinforcement Learning Approach

Applying of network slicing in vehicular networks becomes a promising paradigm to support emerging Vehicle-to-Vehicle (V2V) applications with diverse quality of service (QoS) requirements. However, achieving effective network slicing in dynamic vehicular communications still faces many challenges, particularly time-varying traffic of Vehicle-to-Vehicle (V2V) services and the fast-changing network topology. By leveraging the widely deployed LTE infrastructures, we propose a semi-decentralized network slicing framework in this paper based on the C-V2X Mode-4 standard to provide customized network slices for diverse V2V services. With only the long-term and partial information of vehicular networks, eNodeB (eNB) can infer the underlying network situation and then intelligently adjust the configuration for each slice to ensure the long-term QoS performance. Under the coordination of eNB, each vehicle can autonomously select radio resources for its V2V transmission in a decentralized manner. Specifically, the slicing control at the eNB is realized by a model-free deep reinforcement learning (DRL) algorithm, which is a convergence of Long Short Term Memory (LSTM) and actor-critic DRL. Compared to the existing DRL algorithms, the proposed DRL neither requires any prior knowledge nor assumes any statistical model of vehicular networks. Furthermore, simulation results show the effectiveness of our proposed intelligent network slicing scheme.

Data-Driven Coordinated Charging for Electric Vehicles With Continuous Charging Rates: A Deep Policy Gradient Approach

In this article, we consider a parking lot that manages the charging processes of its parked electric vehicles (EVs). Upon arrival, each EV requests a certain amount of energy. This request should be fulfilled before the EV’s departure. It is of critical importance to coordinate the EVs’ charging rates to smooth out the load profile of the parking lot because inappropriate charging rates can lead to sharp spikes and fluctuations on the load profile, imposing negative effects on the power grid. Meanwhile, empirical studies show that many parking lots exhibit statistical patterns on EV dynamics. For example, the bulk of EVs arrives during rush hours. Therefore, in this article, we incorporate such patterns into charging rate coordination. Although the statistical patterns can be summarized from historical data, they are difficult to be analytically modeled. As a result, we adopt a model-free deep reinforcement learning approach. We also take the latest continuous charging rate control technology into consideration. The decision variables are thus continuous and a policy gradient algorithm is needed to perform reinforcement learning. Technically, we first formulate the problem as a Markov decision process (MDP) with unknown state transition probabilities. To further derive a deep policy gradient algorithm, the challenge lies in the inconsistent and state-dependent action space of the MDP model, due to the constraint to satisfy EVs’ energy demands before their scheduled departure. To tackle the challenge, we design a customized model for neural network training by extending the action space to be consistent and state independent, and revise the reward function to penalize the neural network output if it is beyond the action space of the original MDP model. With this customized model, we then develop a deep policy gradient algorithm based on the proximal policy gradient framework. Numerical results show that our algorithm outperforms the benchmarks.

Learning to Drive Like Human Beings: A Method Based on Deep Reinforcement Learning

In this paper, a new framework for path tracking is proposed through learning to drive like human beings. Firstly, the imitation algorithm (behavior cloning) is adopted to initialize the deep reinforcement learning (DRL) algorithm through learning the professional drivers’ experience. Secondly, a continuous, deterministic, model free deep reinforcement learning algorithm is adopted to optimize our DRL model on line through trial and error. By combining behavior cloning and deep reinforcement learning algorithms, the DRL model can learn an effective policy quickly for path tracking using some easy-to-measure vehicle state parameters and environment information as inputs. Actor-Critic structure is adopted in the DRL algorithm. In order to speed up the convergence rate of the DRL model and improve the learning effect, we propose a dual actor networks structure for the two different action outputs (steering wheel angle and vehicle speed), and a chief critic network is built to guide the updating process of dual actor networks at the same time. Based on this dual actor networks structure, we can pick out some more important state information as state inputs for different action outputs. Besides, a kind of reward mechanism is also presented for autonomous driving. Finally, simulation training and experiment test are carried out, and the results confirm that the framework proposed in this paper is more than data efficient than the original algorithm, and the trained DRL model can track the reference path with accuracy and has the generalization ability for different roads.

Three-Stage Inverter-Based Peak Shaving and Volt-VAR Control in Active Distribution Networks Using Online Safe Deep Reinforcement Learning

This paper presents a three-stage inverter-based peak shaving and Volt-VAR control (VVC) framework in active distribution systems using the online safe deep reinforcement learning (DRL) method. The proposed framework aims to reduce the peak load, voltage violations, and real power loss by coordinating three stages with different control timescales. In the first stage, a day-ahead charging/discharging scheduling of energy storage systems (ESSs) with a 30 min resolution is performed via their inverters for peak shaving. In the second stage, the discharging power of ESSs is adjusted through measurements with a 1 min resolution to completely shave peak loads. A model-free DRL algorithm integrated with a safety module is also implemented in the second stage. Using this algorithm, the reactive powers of photovoltaic (PV) systems and ESSs are controlled by the DRL agent to reduce the voltage violation and real power loss, whereas no voltage violation occurs during the online training process. In the third stage, a proportional-integral controller with real-power compensation is integrated into inverters of PV systems and ESSs to rapidly mitigate local voltage violations with a 0.1 s resolution. The high efficiency and safety of the proposed method were validated on the IEEE 33-bus and IEEE 123-bus systems.

Rapid Autonomous Vehicle Drifting with Deep Reinforcement Learning

Abstract: Now days all types of vehicles run on road are upgrading to automated self-driving techniques. Drifting is a complicated procedure for autonomous vehicle control which is used for high-speed sideslip cornering. It is difficult to drift a vehicle at a high speed of 80-120 km/hr by using traditional techniques like two-state single-track model and three-state singletrack models, which depend on the knowledge of tire or road forces that are not to be known accurately due to the real-world environmental complexity. Because of these inaccuracies in these parameters there will be poor control performance and may lead to road accidents. In this scenario, this project presents a strong drift control algorithm, which is based on the most recent model-free deep reinforcement learning algorithm called soft actor-critic (SAC). SAC is used to control front-wheel drive (FWD) vehicles to drive at high speed (80– 120 km/h), and to drift through sharp corners quickly and stability which in turn helps to prevent road accidents at the time of drifting. The efficiency of this algorithm is evaluated by conducting experiments on Trajectories dataset taken from Git-Hub repository, which consists of some road maps with reference drift trajectories. The SAC algorithm can also deal with vehicle types with various actual properties, like mass, tyre friction, and so on to show its notable generalization ability.

Dynamic Topology Design of NFV-Enabled Services Using Deep Reinforcement Learning

Next-generation networks are endowed with enhanced capabilities thanks to software-defined networking and network function virtualization (NFV). There is a radical shift from device-centric to experience-driven environments of which data is the primary driver behind its running engines. In this paper, we consider joint topology design, traffic routing and NF placement for unicast NFV-enabled services. We develop an end-to-end model-free deep reinforcement learning (RL) framework to dynamically allocate processing and transmission resources, while considering time-varying network traffic patterns. First, we provide a flexible pre-processing technique that represents and reduces the state space and action space of the considered joint problem for the deep RL algorithm. Second, we present a deep deterministic policy gradient (DDPG) algorithm that is enhanced with a model-assisted exploration procedure. Due to the multiple resource types with strongly adverse effects, the existing vanilla DDPG algorithm cannot achieve consistent performance. The model-assisted exploration procedure, which utilizes a perturbed step-wise sub-optimal integer linear program, bootstraps and stabilizes the vanilla DDPG algorithm and finds optimal solutions efficiently.

Volt-VAR control for distribution networks with high penetration of DGs: An overview

Continuously improving the ability to accept distributed renewable energies is the trend of future grid development, and a large number of papers have been published in recent years to study the problem of Volt-VAR control (VVC) for distribution networks with high penetration of distributed generations. This paper summarizes the relevant modeling and solution methods for VVC problems, mainly including VVC based on multiple time scales, hierarchical partitioning, multi-stage and network reconstruction, in conjunction with the operational characteristics of distribution networks containing distributed renewable energies; meanwhile, it analyzes the advantages and disadvantages of traditional optimization methods, heuristic intelligent algorithms and random variable processing methods used to solve VVC problems, and then introduces the application of model-free deep reinforcement learning as a latest decision method in VVC of distribution networks. Most of the models and methods compiled in this article are from the research results of the last three years and have some reference value.

Wind Farm Power Generation Control Via Double-Network-Based Deep Reinforcement Learning

A model-free deep reinforcement learning (DRL) method is proposed in this article to maximize the total power generation of wind farms through the combination of induction control and yaw control. Specifically, a novel double-network (DN)-based DRL approach is designed to generate control policies for thrust coefficients and yaw angles simultaneously and separately. Two sets of critic-actor networks are constructed to this end. They are linked by a central power-related reward, providing a coordinated control structure while inheriting the critic-actor mechanism's advantages. Compared with conventional DRL methods, the proposed DN-based DRL strategy can adapt to the distinctive and incompatible features of different control inputs, guaranteeing a reliable training process and ensuring superior performance. Also, the prioritized experience replay strategy is utilized to improve the training efficiency of deep neural networks. Simulation tests based on a dynamic wind farm simulator show that the proposed method can significantly increase the power generation for wind farms with different layouts.

Deep-Reinforcement-Learning-Based Predictive Maintenance Model for Effective Resource Management in Industrial IoT

Unplanned breakdown of critical equipment interrupts production throughput in Industrial IoT (IIoT), and data-driven Predictive Maintenance (PdM) becomes increasingly important for companies seeking a competitive business advantage. Manufacturers, however, are constantly faced with the onerous challenge of manually allocating suitably competent manpower resources in the event of an unexpected machine breakdown. Furthermore, human error has a negative rippling impact on both overall equipment downtime and production schedules. In this paper, we formulate the complex resource management problem as a resource optimisation problem to determine if a model-free Deep Reinforcement Learning (DRL) based PdM framework can be used to automatically learn an optimal decision-policy from a stochastic environment. Unlike the existing PdM frameworks, our approach considers PdM sensor information and the resources of both physical equipment and human as part of the optimisation problem. The proposed DRL-based framework and Proximal Policy Optimisation Long Short Term Memory (PPO-LSTM) model are evaluated alongside baselines results from human participants using a maintenance repair simulator. Empirical results indicate that our PPO-LSTM efficiently learns the optimal decision-policy for the resource management problem, outperforming comparable DRL methods and human participants by 53% and 65% respectively. Overall, the simulation results corroborate the proposed DRL-based PdM framework’s superiority in terms of convergence efficiency, simulation performance and flexibility.

Voltage Regulation of DC-DC Buck Converters Feeding CPLs via Deep Reinforcement Learning

Modeling accuracy of DC-DC converters may deviate largely in the presence of different variation levels of constant power loads (CPLs), hence is well acknowledged as a main hurdle for the design of advanced model-driven control strategies in the literature. Aiming to enhance the bus voltage regulation performance of DC-DC buck converters, a model-free deep reinforcement learning (DRL) control strategy is proposed in this brief. Firstly, a Markov Decision Process (MDP) model and a deep Q network (DQN) algorithm are utilized for the stabilization issue of the converter. Secondly, through a subgoal reward/penalty mechanism, the control objective and prescribed performance of the system are therefore guaranteed. Moreover, a specified action space is designed to match the switch speed of the switching element. As a distinguishable feature, the settling time under the proposed control scheme is significantly reduced in the occurrence of disturbance, resulting from the fast adaption ability of DRL. The simulation comparison results in reference to PI and fuzzy PI controllers demonstrate the efficacy and superiorities under large signal perturbation conditions.

Autonomous robotic additive manufacturing through distributed model‐free deep reinforcement learning in computational design environments

The objective of autonomous robotic additive manufacturing for construction in the architectural scale is currently being investigated in parts both within the research communities of computational design and robotic fabrication (CDRF) and deep reinforcement learning (DRL) in robotics. The presented study summarizes the relevant state of the art in both research areas and lays out how their respective accomplishments can be combined to achieve higher degrees of autonomy in robotic construction within the Architecture, Engineering and Construction (AEC) industry. A distributed control and communication infrastructure for agent training and task execution is presented, that leverages the potentials of combining tools, standards and algorithms of both fields. It is geared towards industrial CDRF applications. Using this framework, a robotic agent is trained to autonomously plan and build structures using two model-free DRL algorithms (TD3, SAC) in two case studies: robotic block stacking and sensor-adaptive 3D printing. The first case study serves to demonstrate the general applicability of computational design environments for DRL training and the comparative learning success of the utilized algorithms. Case study two highlights the benefit of our setup in terms of tool path planning, geometric state reconstruction, the incorporation of fabrication constraints and action evaluation as part of the training and execution process through parametric modeling routines. The study benefits from highly efficient geometry compression based on convolutional autoencoders (CAE) and signed distance fields (SDF), real-time physics simulation in CAD, industry-grade hardware control and distinct action complementation through geometric scripting. Most of the developed code is provided open source.

Proactive and AoI-Aware Failure Recovery for Stateful NFV-Enabled Zero-Touch 6G Networks: Model-Free DRL Approach

In this paper, we propose a Zero-Touch, deep reinforcement learning (DRL)-based Proactive Failure Recovery framework called ZT-PFR for stateful network function virtualization (NFV)-enabled networks. To this end, we formulate a resource-efficient optimization problem minimizing the network cost function including resource cost and wrong decision penalty. As a solution, we propose state-of-the-art DRL-based methods such as soft-actor-critic (SAC) and proximal-policy-optimization (PPO). In addition, to train and test our DRL agents, we propose a novel impending-failure model. Moreover, to keep network status information at an acceptable freshness level for appropriate decision-making, we apply the concept of age of information to strike a balance between the event and scheduling based monitoring. Several key systems and DRL algorithm design insights for ZT-PFR are drawn from our analysis and simulation results. For example, we use a hybrid neural network, consisting long short-term memory layers in the DRL agents structure, to capture impending-failures time dependency.

Training effective deep reinforcement learning agents for real-time life-cycle production optimization

Life-cycle production optimization aims to obtain the optimal well control scheme at each time control step to maximize financial profit and hydrocarbon production. However, searching for the optimal policy under the limited number of simulation evaluations is a challenging task. In this paper, a novel production optimization method is presented, which maximizes the net present value (NPV) over the entire life-cycle and achieves real-time well control scheme adjustment. The proposed method models the life-cycle production optimization problem as a finite-horizon Markov decision process (MDP), where the well control scheme can be viewed as sequence decisions. Soft actor-critic, known as the state-of-the-art model-free deep reinforcement learning (DRL) algorithm, is subsequently utilized to train DRL agents that can solve the above MDP. The DRL agent strives to maximize long-term NPV rewards as well as the control scheme randomness by training a stochastic policy that maps reservoir states to well control variables and an action-value function that estimates the objective value of the current policy. Since the trained policy is an explicit function structure, the DRL agent can adjust the well control scheme in real-time under different reservoir states. Different from most existing methods that introduce task-specific sensitive parameters or construct complex supplementary structures, the DRL agent learns adaptively by executing goal-directed interactions with an uncertain reservoir environment and making use of accumulated well control experience, which is similar to the actual field well control mode. The key insight here is that the DRL method's ability to utilize gradients information (well-control experience) for higher sample efficiency. The simulation results based on two reservoir models indicate that compared to other optimization methods, the proposed method can attain higher NPV and access excellent performance in terms of oil displacement.

Learning Robotic Manipulation Tasks via Task Progress Based Gaussian Reward and Loss Adjusted Exploration

Multi-step manipulation tasks in unstructured environments are extremely challenging for a robot to learn. Such tasks interlace high-level reasoning that consists of the expected states that can be attained to achieve an overall task and low-level reasoning that decides what actions will yield these states. We propose a model-free deep reinforcement learning method to learn multi-step manipulation tasks. We introduce a Robotic Manipulation Network (RoManNet) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , which is a vision-based model architecture, to learn the action-value functions and predict manipulation action candidates. We define a Task Progress based Gaussian (TPG) reward function that computes the reward based on actions that lead to successful motion primitives and progress towards the overall task goal. To balance the ratio of exploration/exploitation, we introduce a Loss Adjusted Exploration (LAE) policy that determines actions from the action candidates according to the Boltzmann distribution of loss estimates. We demonstrate the effectiveness of our approach by training RoManNet to learn several challenging multi-step robotic manipulation tasks in both simulation and real-world. Experimental results show that our method outperforms the existing methods and achieves state-of-the-art performance in terms of success rate and action efficiency. The ablation studies show that TPG and LAE are especially beneficial for tasks like multiple block stacking.

Deep Reinforcement Learning With Communication Transformer for Adaptive Live Streaming in Wireless Edge Networks

The emerging mobile edge computing (MEC) technology has been recently applied to improve the Quality of Experience (QoE) of network services, such as live video streaming. In this paper, we study an energy-aware adaptive live streaming scheme in wireless edge networks. In particular, we aim to design a joint uplink transmission and edge transcoding algorithm maximizing the video followers’ QoE, while minimizing the energy consumption of the video streamer. We formulate the problem as a Markov decision process (MDP), and propose a deep reinforcement learning (DRL) based framework, named SACCT, to determine the streamer’s encoding bitrate, the uploading power as well as the edge transcoding bitrates and frequency. We decompose the MDP problem into inter-frame and intra-frame problems to address the key design challenges that arise from continuous-discrete hybrid action space, time-varying state and action spaces, and unknown network variation. By doing so, SACCT integrates model-based optimization and model-free DRL to determine the intra-frame continuous resource allocation decisions and the inter-frame discrete bitrate adaptation decisions, respectively. To integrate both the numerical features (e.g., channel gain) and the categorical features (e.g., bitrate), we propose a communication Transformer (CT) as a backbone of SACCT by representing network states as communication tokens and running Transformers to model multi-scale dependencies. Extensive simulations manifest that compared with state-of-the-art approaches, SACCT can provide 128.23% (on average) extra reward. As such, by leveraging joint uplink adaption and edge transcoding, the proposed scheme enables an intelligent wireless network edge with QoE-assured and energy-aware live streaming services.