Deep Q-Network Agent Research Articles

Deep reinforcement learning using deep-Q-network for Global Maximum Power Point tracking: Design and experiments in real photovoltaic systems

This paper presents a methodology for integrating Deep Reinforcement Learning (DRL) using a Deep-Q-Network (DQN) agent into real-time experiments to achieve the Global Maximum Power Point (GMPP) of Photovoltaic (PV) systems under various environmental conditions. Conventional methods, such as the Perturb and Observe (P&O) algorithm, often become stuck at the Local Maximum Power Point (LMPP) and fail to reach the GMPP under Partial Shading Conditions (PSC). The main contribution of this work is the experimental validation of the DQN agent's implementation in a synchronous DC-DC Buck converter (step-down converter) un-der both uniform and PSC conditions. Additionally, we establish a testing pipeline for DRL models. The DQN agent's performance is evaluated alongside the P&O algorithm. Results consistently indicate the DQN agent's superiority over the P&O algorithm in all simulated scenarios. Although this trend does not entirely replicate in real-world test setups, significant results are observed. Specifically, in PSC sce-narios where the P&O algorithm becomes trapped at an LMPP, the DQN algorithm extracts up to 63.5 % more power than the P&O algorithm. An open repository is available, containing PCB schematic designs and layouts, along with the code used for model training and deployment.

Multi-Objective Deep Q-Network Control for Actively Lubricated Bearings

This paper aims to study and demonstrate the possibilities of using reinforcement learning for the synthesis of multi-objective controllers for radial actively lubricated hybrid fluid film bearings (ALHBs), which are considered to be complex multi-physical systems. In addition to the rotor displacement control problem being typically solved for active bearings, the proposed approach also includes power losses due to friction and lubricant pumping in ALHBs among the control objectives to be minimized by optimizing the lubrication modes. The multi-objective controller was synthesized using the deep Q-network (DQN) learning technique. An optimal control policy was determined by the DQN agent during its repetitive interaction with the simulation model of the rotor system with ALHBs. The calculations were sped up by replacing the numerical model of an ALHB with its surrogate ANN-based counterpart and by predicting the shaft displacements in response to operation of two independent control loops. The controller synthesized considering the formulated reward function for DQN agent is able to find a stable shaft position that reduces power losses by almost half compared to the losses observed when using a passive system. It also is able to prevent the established limit of the minimum fluid film thickness being exceeded to avoid possible system damage, for example, when the rotor is unbalanced during the operation. Analysis of the development process and the results obtained allowed us to draw conclusions about the main advantages and disadvantages of the considered approach, and also allowed us to identify some important directions for further research.

A Study for Comparative Analysis of Dueling DQN and Centralized Critic Approaches in Multi-Agent Reinforcement Learning

In this study, we introduce a deep Q-network agent utilizing a dueling architecture to refine the valuation of actions through separate estimations of the state-value and action-value functions, adapted to facilitate concurrent multi-agent operations within a shared environment. Inspired by the self-organized, decentralized cooperation observed in natural swarms, this study uniquely integrates a centralized mechanism, or a centralized critic. This enhances performance and coherence in decision-making within the multi-agent system. This hybrid approach enables agents to execute informed and optimized decisions by considering the actions of their counterparts while maintaining an element of collective and flexible task-information sharing, thereby presenting a groundbreaking framework for cooperation and information sharing in swarm robot systems. To augment the communication capabilities, we employ low-power wide-area networks, or Long Range (LoRa), which are characterized by their low power consumption and long-range communication abilities, facilitating the sharing of task information and reducing the load on individual robots. The aim is to leverage LoRa as a communication platform to construct a cooperative algorithm that enables efficient task-information sharing among groups. This can provide innovative solutions and promote effective cooperation and communication within multi-agent systems, with significant implications for industrial and exploratory robots. In conclusion, by integrating a centralized system into the proposed model, this approach successfully enhances the performance of multi-agent systems in real-world applications, offering a balanced synergy between decentralized flexibility and centralized control.

Indoor Positioning by Double Deep Q-Network in VLC-Based Empty Office Environment

Recently, artificial intelligence (AI) has been applied in various industries. One such application is indoor user positioning using Big Data. The traditional method for positioning is the global positioning system (GPS). However, the performance of GPS is limited indoors due to propagation loss. Hence, radio frequency (RF)-based communication methods such as WiFi and Bluetooth have been proposed as indoor positioning solutions. However, positioning performance inaccuracies arise due to signal interference caused by RF band saturation. Therefore, this study proposes indoor user positioning based on visible light communication (VLC). The proposed method involves the sequential application of fingerprinting and double deep Q-Network. Fingerprinting is utilized to define the action and state of the double deep Q-Network agent. The agent is designed to learn and locate the reference point (RP) closest to the user’s position in a shorter search time. The core idea of the proposed system is to converge a Cell-ID scheme and fingerprinting. Through this, the initial state of the double deep Q-Network agent can be limited. A limited initial state can increase the positioning speed. Simulation results show that the proposed scheme attains a positioning resolution of less than 13 cm and achieves a processing time of less than 0.03 s to obtain the final position in VLC-based office environments.

Optimal operation of reverse osmosis desalination process with deep reinforcement learning methods

The reverse osmosis (RO) process is a well-established desalination technology, wherein energy-efficient techniques and advanced process control methods significantly reduce production costs. This study proposes an optimal real-time management method to minimize the total daily operation cost of an RO desalination plant, integrating a storage tank system to meet varying daily freshwater demand. Utilizing the dynamic model of the RO process, a cascade structure with two reinforcement learning (RL) agents, namely the deep deterministic policy gradient (DDPG) and deep Q-Network (DQN), is developed to optimize the operation of the RO plant. The DDPG agent, manipulating the high-pressure pump, controls the permeate flow rate to track a reference setpoint value. Simultaneously, the DQN agent selects the optimal setpoint value and communicates it to the DDPG controller to minimize the plant’s operation cost. Monitoring storage tanks, permeate flow rates, and water demand enables the DQN agent to determine the required amount of permeate water, optimizing water quality and energy consumption. Additionally, the DQN agent monitors the storage tank’s water level to prevent overflow or underflow of permeate water. Simulation results demonstrate the effectiveness and practicality of the designed RL agents.

Sampling Efficient Deep Reinforcement Learning Through Preference-Guided Stochastic Exploration.

Stochastic exploration is the key to the success of the deep Q -network (DQN) algorithm. However, most existing stochastic exploration approaches either explore actions heuristically regardless of their Q values or couple the sampling with Q values, which inevitably introduce bias into the learning process. In this article, we propose a novel preference-guided ϵ -greedy exploration algorithm that can efficiently facilitate exploration for DQN without introducing additional bias. Specifically, we design a dual architecture consisting of two branches, one of which is a copy of DQN, namely, the Q branch. The other branch, which we call the preference branch, learns the action preference that the DQN implicitly follows. We theoretically prove that the policy improvement theorem holds for the preference-guided ϵ -greedy policy and experimentally show that the inferred action preference distribution aligns with the landscape of corresponding Q values. Intuitively, the preference-guided ϵ -greedy exploration motivates the DQN agent to take diverse actions, so that actions with larger Q values can be sampled more frequently, and those with smaller Q values still have a chance to be explored, thus encouraging the exploration. We comprehensively evaluate the proposed method by benchmarking it with well-known DQN variants in nine different environments. Extensive results confirm the superiority of our proposed method in terms of performance and convergence speed.

Task-Based Visual Attention for Continually Improving the Performance of Autonomous Game Agents

Deep Reinforcement Learning (DRL) has been effectively performed in various complex environments, such as playing video games. In many game environments, DeepMind’s baseline Deep Q-Network (DQN) game agents performed at a level comparable to that of humans. However, these DRL models require many experience samples to learn and lack the adaptability to changes in the environment and handling complexity. In this study, we propose Attention-Augmented Deep Q-Network (AADQN) by incorporating a combined top-down and bottom-up attention mechanism into the DQN game agent to highlight task-relevant features of input. Our AADQN model uses a particle-filter -based top-down attention that dynamically teaches an agent how to play a game by focusing on the most task-related information. In the evaluation of our agent’s performance across eight games in the Atari 2600 domain, which vary in complexity, we demonstrate that our model surpasses the baseline DQN agent. Notably, our model can achieve greater flexibility and higher scores at a reduced number of time steps.Across eight game environments, AADQN achieved an average relative improvement of 134.93%. Pong and Breakout games both experienced improvements of 9.32% and 56.06%, respectively. Meanwhile, SpaceInvaders and Seaquest, which are more intricate games, demonstrated even higher percentage improvements, with 130.84% and 149.95%, respectively. This study reveals that AADQN is productive for complex environments and produces slightly better results in elementary contexts.

Performance Assessment and Comparative Analysis of Photovoltaic-Battery System Scheduling in an Existing Zero-Energy House Based on Reinforcement Learning Control

The development of distributed renewable energy resources and smart energy management are efficient approaches to decarbonizing building energy systems. Reinforcement learning (RL) is a data-driven control algorithm that trains a large amount of data to learn control policy. However, this learning process generally presents low learning efficiency using real-world stochastic data. To address this challenge, this study proposes a model-based RL approach to optimize the operation of existing zero-energy houses considering PV generation consumption and energy costs. The model-based approach takes advantage of the inner understanding of the system dynamics; this knowledge improves the learning efficiency. A reward function is designed considering the physical constraints of battery storage, photovoltaic (PV) production feed-in profit, and energy cost. Measured data of a zero-energy house are used to train and test the proposed RL agent control, including Q-learning, deep Q network (DQN), and deep deterministic policy gradient (DDPG) agents. The results show that the proposed RL agents can achieve fast convergence during the training process. In comparison with the rule-based strategy, test cases verify the cost-effectiveness performances of proposed RL approaches in scheduling operations of the hybrid energy system under different scenarios. The comparative analysis of test periods shows that the DQN agent presents better energy cost-saving performances than Q-learning while the Q-learning agent presents more flexible action control of the battery with the fluctuation of real-time electricity prices. The DDPG algorithm can achieve the highest PV self-consumption ratio, 49.4%, and the self-sufficiency ratio reaches 36.7%. The DDPG algorithm outperforms rule-based operation by 7.2% for energy cost during test periods.

Worst-Case Analysis of Complex Nonlinear Flight Control Designs Using Deep Q-Learning

With the objective of exposing hidden design deficiencies in complex nonlinear systems, this paper presents the use of reinforcement learning techniques for application in flight control law development and testing. Following the idea of worst-case testing, a deep Q-network agent is trained to identify input sequences that lead to detrimental system behavior. Because the analysis is based directly on the repeated interaction between the agent and the investigated system, no model simplifications are required, making the presented method applicable to highly complex systems. The capability of the learning-based worst-case analysis is demonstrated for the speed protection function of the hover flight control law of an electric vertical takeoff and landing (eVTOL) aircraft. The analysis discovers possible piloted maneuvers that violate the implemented protection algorithm. A root cause analysis of the emerging behavior reveals the neglect of an important flight mechanical coupling term in the design of the protection algorithm and ultimately leads to the revision and improvement of the controller. This demonstrates the benefits of the presented testing method for the design, verification, and validation of complex systems. The application to a high-fidelity system used for control law development of an actual eVTOL prototype currently under construction demonstrates the relevance of the method beyond academia.

Deep-Reinforcement-Learning-Based IoT Sensor Data Cleaning Framework for Enhanced Data Analytics.

The Internet of things (IoT) combines different sources of collected data which are processed and analyzed to support smart city applications. Machine learning and deep learning algorithms play a vital role in edge intelligence by minimizing the amount of irrelevant data collected from multiple sources to facilitate these smart city applications. However, the data collected by IoT sensors can often be noisy, redundant, and even empty, which can negatively impact the performance of these algorithms. To address this issue, it is essential to develop effective methods for detecting and eliminating irrelevant data to improve the performance of intelligent IoT applications. One approach to achieving this goal is using data cleaning techniques, which can help identify and remove noisy, redundant, or empty data from the collected sensor data. This paper proposes a deep reinforcement learning (deep RL) framework for IoT sensor data cleaning. The proposed system utilizes a deep Q-network (DQN) agent to classify sensor data into three categories: empty, garbage, and normal. The DQN agent receives input from three received signal strength (RSS) values, indicating the current and two previous sensor data points, and receives reward feedback based on its predicted actions. Our experiments demonstrate that the proposed system outperforms a common time-series-based fully connected neural network (FCDQN) solution, with an accuracy of around 96% after the exploration mode. The use of deep RL for IoT sensor data cleaning is significant because it has the potential to improve the performance of intelligent IoT applications by eliminating irrelevant and harmful data.

A New Efficient Method for Refining the Reinforcement Learning Algorithm to Train Deep Q Agents for Reaching a Consensus in P2P Networks

The usage of distributed Peer-to-Peer (P2P) networks has been growing steadily for a reasonable period. Various applications rely on the infrastructure of P2P networks, where nodes communicate to accomplish a task without the need for a central authority. One of the significant challenges in P2P networks is the ability of the network nodes to reach a consensus on a shared item; the challenge increases as time passes. Thus, this work proposes a new effective method for tweaking the Deep Reinforcement Learning (DRL) algorithm to train Deep Q Network (DQN) learning agents to reach a consensus among the P2P nodes. We propose various hierarchies of deep agents to address this crucial challenge in P2P networks. DRL is utilized to build and train agents; more precisely, DQN learning agents are constructed and trained. Two scenarios are proposed and evaluated. In the first scenario, one DQN agent is trained to find the consensus between the network nodes. In the second scenario, three hierarchies with different numbers of layers of agents are proposed and evaluated. In both scenarios, the P2P network used is a blockchain network. The best result was obtained using the third hierarchy of the second scenario; the overall accuracy of the model is 87.8%.

Sensing-Communication Bandwidth Allocation in Vehicular Links Based on Reinforcement Learning

Influenced by the randomnesses in the realistic vehicle-to-infrastructure (V2I) scenarios, traditional deterministic optimization algorithms cannot be adopted directly to promote sensing-communication performance. In this letter, we consider a vehicle served by several base stations (BSs), model the integrated performance based on information theory, and formulate an optimization problem to enlarge the total information amount. A reinforcement-learning-based algorithm of jointly optimizing the bandwidth allocation and BS selection is proposed, where a double deep Q-network agent is adopted. Simulation results verify the system performance, which shows performing better than baselines.

Deep Reinforcement Learning based dynamic optimization of bus timetable

Bus timetable optimization is a key issue to reduce operational cost of bus company and improve the transit service quality. Existing methods optimize the timetable offline. However, in practice, the short-term passenger flow may change dramatically from time to time. Timetables generated offline cannot be adjusted in real time to handle the changed passenger flow. In this paper, we propose a Deep Reinforcement Learning based bus Timetable dynamic Optimization method (DRL-TO). In DRL-TO, the problem of bus timetable optimization is formulated as a Markov Decision Process (MDP). A Deep Q-Network (DQN) is applied as the agent to decide whether a bus departs at each minute during the service period. Therefore, departure intervals of bus services are determined in real time in accordance with passenger demand. We identify several new and useful state features for the DQN agent, including the load factor, the carrying capacity utilization rate, passengers’ waiting time and the number of stranded passengers. Considering the interests of both the bus company and passengers, a reward function is designed, which includes metrics of full load rate, empty load rate, passengers’ waiting time, and the number of stranded passengers. Experiments demonstrate that, in comparison to the timetable generated by offline optimization approaches and the manual method, DRL-TO can dynamically determine the departure intervals based on the real-time passenger flow, and generate a timetable with less departure time points (i.e., operational cost) and shorter passengers’ waiting time (i.e., higher quality of service).

Cooperative Traffic Signal Control of n-intersections Using a Double Deep QNetwork Agent

Cooperative Traffic Signal Control of n-intersections Using a Double Deep QNetwork Agent

Adversarial Attacks on Heterogeneous Multi-Agent Deep Reinforcement Learning System with Time-Delayed Data Transmission

This paper studies the gradient-based adversarial attacks on cluster-based, heterogeneous, multi-agent, deep reinforcement learning (MADRL) systems with time-delayed data transmission. The structure of the MADRL system consists of various clusters of agents. The deep Q-network (DQN) architecture presents the first cluster’s agent structure. The other clusters are considered as the environment of the first cluster’s DQN agent. We introduce two novel observations in data transmission, termed on-time and time-delay observations. The proposed observations are considered when the data transmission channel is idle, and the data is transmitted on time or delayed. By considering the distance between the neighboring agents, we present a novel immediate reward function by appending a distance-based reward to the previously utilized reward to improve the MADRL system performance. We consider three types of gradient-based attacks to investigate the robustness of the proposed system data transmission. Two defense methods are proposed to reduce the effects of the discussed malicious attacks. We have rigorously shown the system performance based on the DQN loss and the team reward for the entire team of agents. Moreover, the effects of the various attacks before and after using defense algorithms are demonstrated. The theoretical results are illustrated and verified with simulation examples.

Reward criteria impact on the performance of reinforcement learning agent for autonomous navigation

In reinforcement learning, an agent takes action at every time step (follows a policy) in an environment to maximize the expected cumulative reward. Therefore, the shaping of a reward function plays a crucial role in an agent’s learning. Designing an optimal reward function is not a trivial task. In this article, we propose a reward criterion using which we develop different reward functions. The reward criterion chosen is based on the percentage of positive and negative rewards received by an agent. This reward criteria further gives rise to three different classes, ‘Balanced Class,’ ‘Skewed Positive Class,’ and ‘Skewed Negative Class.’ We train a Deep Q-Network agent on a point-goal based navigation task using the different reward classes. We also compare the performance of the proposed classes with a benchmark class. Based on the experiments, the skewed negative class outperforms the benchmark class by achieving very less variance. On the other hand, the benchmark class converges relatively faster than the skewed negative class.

A multi‐objective multi‐agent deep reinforcement learning approach to residential appliance scheduling

Residential buildings are large consumers of energy. They contribute significantly to the demand placed on the grid, particularly during hours of peak demand. Demand-side management is crucial to reducing this demand placed on the grid and increasing renewable utilisation. This research study presents a multi-objective tunable deep reinforcement learning algorithm for demand-side management of household appliances. The proposed tunable Deep Q-Network (DQN) algorithm learns a single policy that accounts for different preferences for multiple objectives present when scheduling appliances. These include electricity cost, peak demand, and punctuality. The tunable Deep Q-Network algorithm is compared to two rule-based approaches for appliance scheduling. When comparing the 1-month simulation results for the tunable DQN with an electricity cost rule-based benchmark method, the tunable DQN agent provides a statistically significant improvement of 30%, 18.2%, and 37.3% for the cost, peak power, and punctuality objectives. Moreover, the tunable Deep Q-Network can produce a range of appliance scheduling policies for different objective preferences without requiring any computationally intensive retraining. This is the key advantage of the proposed tunable Deep Q-Network algorithm for appliance scheduling.

Visual servoing with deep reinforcement learning for rotor unmanned helicopter

Visual servoing is a key approach to achieve visual control for the rotor unmanned helicopter. The challenges of the inaccurate matrix estimation and the target loss restrict the performance of the visual servoing control systems. This work proposes a novel visual servoing controller using the deep Q-network to achieve an efficient matrix estimation. A deep Q-network learning agent learns a policy estimating the interaction matrix for visual servoing of a rotor unmanned helicopter using continuous observation. The observation includes a combination of feature errors. The current matrix and the desired matrix constitute the action space. A well-designed reward guides the deep Q-network agent to get a policy to generate a time-varying linear combination between the current matrix and the desired matrix. Then, the interaction matrix is calculated by the linear combination. The potential mapping between the observation and the interaction matrix is learned by cascading the deep neural network layers. Experimental results show that the proposed method achieves faster convergence and lower target loss probability in tracking than the visual servoing methods with the fixed parameter.

Active Fluid-Film Bearing With Deep Q-Network Agent-Based Control System

Abstract Despite the fact that the hydrodynamic lubrication is a self-controlled process, we designed control systems based on proportional integral (PI) controller, adaptive PI controller, and Deep Q network (DQN)-agent to minimize the friction torque in a conical fluid-film bearing. The bearing construction allows the shaft axial displacement and, as the result, the regulation of the bearing average clearance due to the controlled supply pressure. The main challenge is that the friction torque minimization may lead to the loss of load-carrying capacity and the contact in the shaft-bearing tribocouple. The other challenge is that random events may have an influence on the hydrodynamic lubrication parameters, therefore, changing the load-carrying capacity and the friction torque. So, the proposed control systems were designed and tested under the conditions of limited lateral shaft displacements and the action of a random external force. The tests were performed using simulation models of a controlled rotating machine in matlab software. The rotating machine simulation model includes modules of the rigid shaft, the coupling with linear axial reaction, and the conical bearing. The bearing module is based on the numerical solution of the generalized Reynolds equation and its nonlinear approximation with fully connected neural networks. The obtained results demonstrated that the application of an adaptive PI controller or a DQN agent allows decreasing friction torque in a bearing under the conditions of a random external force. The goal of a DQN agent is self-learning in contrast to an adaptive PI controller that needs to be tuned.

Deep reinforcement learning for a color-batching resequencing problem

In automotive paint shops, changes of colors between consecutive production orders cause costs for cleaning the painting robots. It is a significant task to re-sequence orders and group orders with identical color as a color batch to minimize the color changeover costs. In this paper, a Color-batching Resequencing Problem (CRP) with mix bank buffer systems is considered. We propose a Color-Histogram (CH) model to describe the CRP as a Markov decision process and a Deep Q-Network (DQN) algorithm to solve the CRP integrated with the virtual car resequencing technique. The CH model significantly reduces the number of possible actions of the DQN agent, so that the DQN algorithm can be applied to the CRP at a practical scale. A DQN agent is trained in a deep reinforcement learning environment to minimize the costs of color changeovers for the CRP. Two experiments with different assumptions on the order attribute distributions and cost metrics were conducted and evaluated. Experimental results show that the proposed approach outperformed conventional algorithms under both conditions. The proposed agent can run in real time on a regular personal computer with a GPU. Hence, the proposed approach can be readily applied in the production control of automotive paint shops to resolve order-resequencing problems.