Dueling Deep Q-network Research Articles

Optimization of Bulk Cargo Terminal Unloading and Outbound Operations Based on a Deep Reinforcement Learning Framework

This study addresses the integrated scheduling problem of dry bulk cargo terminal yards, which includes three components: transportation planning, yard selection optimization, and equipment scheduling. Additionally, the research integrates safety considerations and addresses the complexities of dynamic transportation planning. This work presents two innovations. Firstly, this study develops a sophisticated modeling framework that integrates graph structures for precise yard mapping with mixed-integer programming to enforce operational constraints. This integrated approach facilitates a more accurate and comprehensive representation of yard operations, capturing diverse operational aspects while maintaining model clarity and computational efficiency. Secondly, this study proposes an advanced solution methodology that employs a reinforcement learning technique integrating a Dueling Deep Q-Network and Double Deep Q-Network. This hybrid algorithm significantly enhances optimization performance and accelerates the learning process, thereby improving the efficiency of the solutions. The experimental results demonstrate that the proposed model effectively manages the integrated scheduling of bulk material ingress, storage, and egress within the yard. The operational plans generated by the approach outperform traditional first-come, first-served strategies, showcasing substantial improvements in port operational efficiency and reliability. This comprehensive solution underscores the potential for significant advancements in the overall management and performance of dry bulk cargo ports.

Enhancing intrusion detection in next-generation networks based on a multi-agent game-theoretic framework

<span lang="EN-US">With cyber threats becoming increasingly sophisticated, existing intrusion detection systems (IDS) in next generation networks (NGNs) are subjected to more false-positives and struggles to offer robust security feature, highlighting a critical need for more adaptive and reliable threat detection mechanisms. This research introduces a novel IDS that leverages a dueling deep Q-network (DQN) a reinforcement learning algorithm within game-theoretic framework simulating a multi-agent adversarial learning scenario to address these challenges. By employing a customized OpenAI Gym environment for realistic threat simulation and advanced dueling DQN mechanisms for reduced overestimation bias, the proposed scheme significantly enhances the adaptability and accuracy of intrusion detection. Comparative analysis against current state-of-the-art methods reveals that the proposed system achieves superior performance, with accuracy and F1-score improvements to 95.02% and 94.68%, respectively. These results highlight the potential scope of the proposed adaptive IDS to provide a robust defense against the dynamic threat landscape in NGNs.</span>

Optimizing smart city planning: A deep reinforcement learning framework

We introduce a deep reinforcement learning-based approach for smart city planning, designed to determine the optimal timing for constructing various smart city components such as apartments, base stations, and hospitals over a specified development period. Utilizing the Dueling Deep Q-Network (DQN), the proposed method aims to maximize the city’s population while maintaining a predetermined happiness level of residents in the smart city. This optimization is achieved through strategic construction of smart city components, considering that both the total population and happiness levels are influenced by the interplay between housing, communication, transportation, and healthcare infrastructures, as well as the population ratio. Specifically, we present two distinct formulations of the Markov Decision Process (MDP) for smart city planning to illustrate the practicality of applying reinforcement learning across different scenarios.

Towards various occupants with different thermal comfort requirements: A deep reinforcement learning approach combined with a dynamic PMV model for HVAC control in buildings

Reinforcement learning (RL) has great potential in achieving energy-efficient, comfortable and intelligent control of heating, ventilation and air conditioning (HVAC) systems. Although research on RL-based HVAC control has attracted increasing interest, current studies generally use simple building simulation as the environment to train agents, and the definition of thermal comfort is limited to a wide temperature range, which cannot meet the different thermal comfort requirements of various occupants. This study proposes a deep reinforcement learning (DRL) control framework based on the Dueling Deep Q-network (DQN) algorithm, combined with a self-designed environmental model and reward function, for HVAC control meeting different thermal comfort requirements. Specifically, based on the theory of building thermal dynamics, a nonlinear equation modified by experimental data is used for the environmental model that reflects the actual thermal change of building. Different thermal comfort requirements are considered and analysed through a dynamic predicted mean vote (PMV) model that focuses on the metabolic rate and clothing level of occupants. By systematically exploring different heating modes for occupants and control time intervals, the proposed framework demonstrates that heating energy consumption can be reduced by 4.8%-39.58% under various conditions compared to rule-based control. In addition, the study found that the HVAC control based on DRL has greater potential in saving energy when the heating demand of building is higher. Our study is helpful for researchers to make HVAC control more energy-efficient and user-friendly with the help of artificial intelligence.

Brain Emotion Perception Inspired EEG Emotion Recognition With Deep Reinforcement Learning.

Inspired by the well-known Papez circuit theory and neuroscience knowledge of reinforcement learning, a double dueling deep Q network (DQN) is built incorporating the electroencephalogram (EEG) signals of the frontal lobe as prior information, which is named frontal lobe double dueling DQN (FLD3QN). The framework of FLD3QN is constructed in accord with the brain emotion mechanism which takes the frontal lobe and the thalamus as the core, in which the part of the Papez circuit is simulated by the bifrontal lobe residual convolution neural network (BiFRCNN). Moreover, a step penalty factor is designed to constrain the number of mistakes of the agent. The ablation studies results on the public EEG emotion dataset DEAP verified the important roles of the frontal lobe and the Papez circuit in modeling the procedure of learning rewards during the perception of emotions, with a great increase in the average accuracies by 25.24% and 23.31% in valence and arousal dimensions.

Spectrum-efficient user grouping and resource allocation based on deep reinforcement learning for mmWave massive MIMO-NOMA systems

Millimeter-wave (mmWave) massive multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) is proven to be a primary technique for sixth-generation (6G) wireless communication networks. However, the great increase in users and antennas brings challenges for interference suppression and resource allocation for mmWave massive MIMO-NOMA systems. This study proposes a spectrum-efficient and fast convergence deep reinforcement learning (DRL)-based resource allocation framework to optimize user grouping and allocation of subchannel and power. First, an enhanced K-means grouping algorithm is proposed to reduce the multi-user interference and accelerate the convergence. Then, a dueling deep Q-network (DQN) structure is proposed to perform subchannel allocation, which further improves the convergence speed. Moreover, a deep deterministic policy gradient (DDPG)-based power resource allocation algorithm is designed to avoid the performance loss caused by power quantization and improve the system’s achievable sum-rate. The simulation results demonstrate that our proposed scheme outperforms other neural network-based algorithms in terms of convergence performance, and can achieve higher system capacity compared with the greedy algorithm, the random algorithm, the RNN algorithm, and the DoubleDQN algorithm.

Deep reinforcement learning for dynamic distributed job shop scheduling problem with transfers

Dynamic events and transportation constraints would significantly affect the full utilization of resources and the reduction of production costs in distributed job shops. Therefore, in this paper, a deep reinforcement learning algorithm (DRL)-based real-time scheduling method is developed to minimize the mean tardiness of the dynamic distributed job shop scheduling problem with transfers (DDJSPT) considering random job arrivals. Firstly, the proposed DDJSPT is modeled as a Markov decision process (MDP). Then, ten problem-oriented state features covering four aspects of factories, machines, jobs, and operations are elaborately extracted from the dynamic distributed job shop. After that, eleven composite rules considering the uniqueness of DDJSPT are constructed as a pool of actions to intelligently prioritize unfinished jobs and allocate the selected job to an appropriate factory. Moreover, a justified reward function adapted from the objective is designed for better convergence of DRLs. Subsequently, five DRLs are employed to address the DDJSPT, encompassing deep Q-network (DQN), double DQN (DDQN), dueling DQN (DlDQN), trust region policy optimization (TRPO), and proximal policy optimization (PPO). Finally, grounded in numerical comparison experiments under 243 production configurations of the DDJSPT, the effectiveness and generalization of DRL-based scheduling methods are credibly verified and confirmed.

Adaptive operator selection with dueling deep Q-network for evolutionary multi-objective optimization

Adaptive operator selection is an online method that automatically adjusts the application rate of different operators based on their actual performance. This paper proposes an adaptive operator selection paradigm based on dueling deep Q-network (DDQN), aiming to improve the training efficiency of the Q-network for solving multi-objective optimization problems. The Q-network is decomposed into state value and action advantage networks, allowing the agent to learn state values more frequently and accurately. A novel state space and reward mechanism are designed to adapt to the characteristics of evolutionary multi-objective optimization. Combining adaptive operator selection with a multi-objective evolutionary algorithm with adaptive weights (AdaW), an AdaW-DDQN algorithm with high adaptability is proposed. Experimental results on three complex benchmark suites demonstrate that the proposed adaptive operator selection paradigm significantly improves the performance of multi-objective optimization algorithms. The AdaW-DDQN algorithm provides an effective and efficient solution for solving multi-objective optimization problems.

Dynamic pricing on maximum concurrency for heterogeneous instances using hyperparameter optimization in dueling deep reinforcement learning in a multi-cloud scenario

Users possess the option to rent instances of various sorts, in a variety of regions, and a variety of availability zones, thanks to cloud service carriers like AWS, GCP, and Azure. In the cloud business right now, fixed price models are king when it comes to pricing. However, as the diversity of cloud providers and users grows, this approach is unable to accurately reflect the market’s current needs for cost savings. As a consequence, a dynamic pricing strategy has become a desirable tactic to better handle the erratic cloud demand. In this study, a deep learning model was used to propose a dynamic pricing structure that ensures service providers are treated fairly in a multi-cloud context. The computational optimization of DL approaches can be severely hampered by the requirement for human hyperparameter selection. Traditional automated solutions to this issue have inadequate durability or fail in specific circumstances. To choose the hyper-parameters in the Dueling Deep Q-Network (DDQN), the hybrid DL approach in this study uses the concept-based wild horse optimization (WHO) method. A community of untamed horses is evolved, and the fitness of the population is evaluated concurrently to estimate the optimum hyper-parameters. The plan changes the price appropriately to promote the use of underutilized resources and discourage the use of overutilized resources. The evaluation’s findings demonstrated that the suggested strategy can lower end-user costs while conducting compute- and data-intensive activities in a multi-cloud environment. The research was concluded by comparing current models after the results were analyzed using various performance indicators.

GA-Dueling DQN Jamming Decision-Making Method for Intra-Pulse Frequency Agile Radar.

Optimizing jamming strategies is crucial for enhancing the performance of cognitive jamming systems in dynamic electromagnetic environments. The emergence of frequency-agile radars, capable of changing the carrier frequency within or between pulses, poses significant challenges for the jammer to make intelligent decisions and adapt to the dynamic environment. This paper focuses on researching intelligent jamming decision-making algorithms for Intra-Pulse Frequency Agile Radar using deep reinforcement learning. Intra-Pulse Frequency Agile Radar achieves frequency agility at the sub-pulse level, creating a significant frequency agility space. This presents challenges for traditional jamming decision-making methods to rapidly learn its changing patterns through interactions. By employing Gated Recurrent Units (GRU) to capture long-term dependencies in sequence data, together with the attention mechanism, this paper proposes a GA-Dueling DQN (GRU-Attention-based Dueling Deep Q Network) method for jamming frequency selection. Simulation results indicate that the proposed method outperforms traditional Q-learning, DQN, and Dueling DQN methods in terms of jamming effectiveness. It exhibits the fastest convergence speed and reduced reliance on prior knowledge, highlighting its significant advantages in jamming the subpulse-level frequency-agile radar.

Rough knowledge enhanced dueling deep Q-network for household integrated demand response optimization

Implementing a household integrated demand response (HIDR) can be an effective solution to save energy and reduce carbon emissions in household multi-energy system (HMES). However, the HMES are subject to uncertainties, and HIDR optimization needs to automatically adapt to the uncertainty and make decisions within 15 min. Therefore, we propose an HIDR optimization method combining rough knowledge and a dueling deep Q-network (DDQN). Firstly, we propose an HIDR optimization framework of rough knowledge and DDQN fusion, in which knowledge is utilized to generate good DDQN samples and serves as action guidance value input to DDQN. Secondly, we formulate the household equipment models and knowledge rules. Subsequently, we design the HIDR optimization algorithm that incorporates rough knowledge into DDQN, focusing on dynamic probability of knowledge sample participation in DDQN learning, knowledge serving as DDQN action adviser, knowledge diversification and dynamic adjustment of DDQN’s random exploration probability. Simulation results show that our method saves energy costs by 5.6% and 0.9% compared to rule-based methods and DDQN, respectively. Additionally, based on an i5-10300H CPU, our method’s convergence time is no more than 13 min even for scheduling interval of 15 min. The average convergence time of our method is 29% of DDQN.

UrbanEnQoSPlace: A Deep Reinforcement Learning Model for Service Placement of Real-Time Smart City IoT Applications

Multi-access Edge Computing (MEC) enables IoT applications to place their services in the edge servers of mobile networks, balancing Quality-of-Service (QoS) and energy-efficiency. Previous works consider compute requirements, while the IoT and latency/bandwidth per-flow communicate requirements are largely ignored. Moreover, the Smart City domain presents unique challenges - modeling the Urban Smart Things (USTs - urban IoT clients), their connectivity with MEC network, diverse resource requirements (compute, communicate, and IoT) of application services, modeling the federation of multiple MEC providers in a city, which we consider in this paper. To address these research gaps, we propose: i) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">UrbanEnQoSMDP</i> - formulation for energy and QoS (latency) optimized service placement for a set of applications in the ‘Urban IoT-Federated MEC-Cloud’ architecture to satisfy applications’ compute, per-flow communicate, and IoT requirements; ii) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">‘’<inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula>-greedy with mask”</i> policy for apriori satisfaction of IoT requirements by shortlisting suitable USTs; iii) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">UrbanEnQoSPlace</i> - proposed multi-action Deep Reinforcement Learning (DRL) model, designed from Dueling Deep-Q Network, that uses the proposed policy to solve the UrbanEnQoSMDP for simultaneously placing all services of an application. Extensive simulation results illustrate efficacy and scalability of proposed model against state-of-the-art DRL algorithms (better convergence, higher rewards, lesser runtime; proposed policy w.r.t fewer violations).

DRL-based max-min fair RIS discrete phase shift optimization for MISO-OFDM systems

In this paper, we investigate a reconfigurable intelligent surface (RIS) assisted downlink orthogonal frequency division multiplexing (OFDM) transmission system. Taking into account hardware constraint, the RIS is considered to be organized into several blocks, and each block of RIS share the same phase shift, which has only 1-bit resolution. With multiple antennas at the base station (BS) serving multiple single-antenna users, we try to design the BS precoder and the RIS reflection phase shifts to maximize the minimum user spectral efficiency, so as to ensure fairness. A deep reinforcement learning (DRL) based algorithm is proposed, in which maximum ratio transmission (MRT) precoding is utilized at the BS and the dueling deep Q-network (DQN) framework is utilized for RIS phase shift optimization. Simulation results demonstrate that the proposed DRL-based algorithm can achieve almost optimal performance, while has much less computation consumption.

Energy-Efficient Collaborative Multi-Access Edge Computing via Deep Reinforcement Learning

The joint problem of task offloading, collaborative computing and resource allocation for Multi-access Edge Computing (MEC) is a challenging issue. In this paper, splitting computing tasks at MEC servers through collaboration among MEC servers and a cloud server, we investigate the joint problem of collaborative task offloading and resource allocation. A collaborative task offloading, computing resource allocation, and subcarrier and power allocation problem in MEC is formulated. The goal is to minimize the total energy consumption of the MEC system while satisfying a delay constraint. The formulated problem is a non-convex mixed-integer optimization problem. In order to solve the problem, we propose a Deep Reinforcement Learning (DRL) based bi-level optimization framework. The task offloading decision, computing collaboration decision, and power and subcarriers allocation subproblems are solved at the upper-level, while the computing resource allocation subproblem is solved at the lower-level. We combine Dueling-Deep-Q-Network (DQN), Double-DQN and add adaptive parameter space Noise (D <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> N-DQN) to improve DRL performance in MEC. Simulation results demonstrate that the proposed algorithm achieves near-optimal performance in Energy Efficiency (EE) and task completion rate compared to other DRL-based approaches and other benchmark schemes under various network parameter settings.

Blockchain-Enabled M2M Communications for UAV-Assisted Data Transmission

Internet of Things (IoT) technology has uncovered a wide range of possibilities in several industrial sectors where smart devices are capable of exchanging real-time data. Machine-to-machine (M2M) data exchange provides a new method for connecting and exchanging data among machine-oriented communication entities (MOCE). Conspicuously, network services will be severely affected if the underneath IoT infrastructure is disrupted. Moreover, it is difficult for MOCEs to re-establish connectivity automatically. Conspicuously, in the current paper, an analysis is performed regarding potential technologies including unmanned aerial vehicles, blockchain, and mobile edge computing (MEC) that can enable the secure establishment of M2M communications networks that have been compromised to maintain the secure transmissible data. Furthermore, a Markov decision process-based joint optimization approach is proposed for blockchain systems that aims to elevate computational power and performance. Additionally, the dueling deep Q-network (DDQ) is incorporated to address the dynamic and complex optimization issue so that UAV selection is ensured to maximize performance. The results of experimental simulation with several statistical attributes suggest that the proposed framework can increase throughput optimally in comparison to state-of-the-art techniques. Additionally, a performance measure of reliability and stability depicts significant enhancement for the proposed framework.

Compressed Feedback Using Autoencoder Based on Deep Learning for D2D Communication Networks

In this letter, we propose a feedback reduction scheme based on AutoEncoder and deep reinforcement learning for frequency division duplex (FDD) overlay device-to-device (D2D) communication networks. All D2D receivers and transmitters are equipped with an Encoder and Decoder, respectively, of the trained AutoEncoder. The D2D receivers compress the feedback information using the Encoder before transmitting while the transmitters decompress the received feedback information. We also employ a dueling deep Q network (DQN) to allow each D2D transmitter to autonomously determine whether to transmit data based on the decompressed feedback information. The performance of the proposed feedback reduction scheme is analyzed in terms of the average MSE of the AutoEncoder and the average sum-rate of a D2D communication network. Our numerical results show that the proposed feedback reduction scheme using the AutoEncoder can achieve 100%, 89%, and 86% of the average sum-rate of the perfect feedback scheme with no compression when the signal-to-noise ratio is 10dB, −5dB, and −20dB, respectively, while reducing the feedback by 50%.

Stackelberg game-based dynamic resource trading for network slicing in 5G networks

Network slicing (NS) has been envisioned as the main technology in 5G to accommodate different virtualized operators with versatile service requirements, on a common substrate infrastructure. NS has come to radically reform the interaction among different entities in the resource sharing market, where resource is exchanged for monetary gains as a way of ensuring efficient resource utilization. However, resource exchange is only attractive to entities if they are able to achieve their respective benefits in a win–win situation. In this paper, we design an economic model that analyzes the pricing and purchasing interaction between multiple mobile virtual network operators (MVNOs) and their respective users for resource trading in NS. We formulate the pricing and purchasing problem as a two-stage multi-leader multi-follower (MLMF) Stackelberg game where the MVNOs as leaders set their unit price first, and then users as followers respond by determining their purchasing volume. Then, we theoretically prove the existence and uniqueness of a Nash equilibrium (NE). To obtain an optimal dynamic pricing solution for the MVNOs, we transform the game-based optimization problem into a stochastic Markov decision process (MDP) problem and propose a method based on multi-agent dueling deep Q-Network (DQN) algorithm. Simulation results show that the proposed algorithm achieves convergence under competitive pricing scheme (CPS) and independent pricing scheme (IPS), while ensuring high MVNOs’ profit and users’ utility at acceptable levels. In terms of cumulative profits, the proposed dueling DQN-based pricing algorithm achieves percentage gains of 15.3%, 30.4%, and 71.4% against Q-learning-based pricing, premium pricing, and undercut pricing algorithms as benchmarks.

Deep reinforcement learning‐based joint optimization of computation offloading and resource allocation in F‐RAN

AbstractThe fog radio access network (F‐RAN) has been regarded as a promising wireless access network architecture in the fifth generation (5G) and beyond systems to satisfy the increasing requirements for low‐latency and high‐throughput services by providing fog computing. However, because the cloud computing centre and fog computing‐enabled access points (F‐APs) in the F‐RAN have different computation and communication capabilities, it is crucial to make an efficient computation offloading and resource allocation strategy that can fully exploit the potential of the F‐RAN system. In this paper, the authors investigate a decentralized low‐complexity deep reinforcement learning (DRL)‐based framework for joint computation task offloading and resource allocation in the F‐RAN, which supports assistive computing‐enabled tasks offloading between F‐APs. Considering the constraints of task latency, wireless transmission rate, transmission power, and computational resource capacity, the authors formulate the system processing efficiency maximization problem by jointly optimizing offloading mode selection, channel allocation, power control, and computation resource allocation in the F‐RAN. To solve this non‐linear and non‐convex problem, the authors propose a federated DRL‐based computation offloading and resource allocation algorithm to improve the task processing efficiency and ensure privacy in the system, which can significantly reduce the computing complexity and signalling overhead of the training process compared with the centralized learning‐based method. Specifically, each local F‐AP agent consists of dueling deep Q‐network (DDQN) and deep deterministic policy gradient (DDPG) networks to appropriately deal with discrete and continuous valuable action spaces, respectively. Finally, the simulation results show that the proposed federated DRL algorithm can achieve significant performance improvements in terms of system processing efficiency and task latency compared with other benchmarks.

Real-time scheduling for distributed permutation flowshops with dynamic job arrivals using deep reinforcement learning

Distributed manufacturing plays an important role for large-scale companies to reduce production and transportation costs for globalized orders. However, how to real-timely and properly assign dynamic orders to distributed workshops is a challenging problem. To provide real-time and intelligent decision-making of scheduling for distributed flowshops, we studied the distributed permutation flowshop scheduling problem (DPFSP) with dynamic job arrivals using deep reinforcement learning (DRL). The objective is to minimize the total tardiness cost of all jobs. We provided the training and execution procedures of intelligent scheduling based on DRL for the dynamic DPFSP. In addition, we established a DRL-based scheduling model for distributed flowshops by designing suitable reward function, scheduling actions, and state features. A novel reward function is designed to directly relate to the objective. Various problem-specific dispatching rules are introduced to provide efficient actions for different production states. Furthermore, four efficient DRL algorithms, including deep Q-network (DQN), double DQN (DbDQN), dueling DQN (DlDQN), and advantage actor-critic (A2C), are adapted to train the scheduling agent. The training curves show that the agent learned to generate better solutions effectively and validate that the system design is reasonable. After training, all DRL algorithms outperform traditional meta-heuristics and well-known priority dispatching rules (PDRs) by a large margin in terms of solution quality and computation efficiency. This work shows the effectiveness of DRL for the real-time scheduling of dynamic DPFSP.

Resource allocation for network slicing in dynamic multi-tenant networks: A deep reinforcement learning approach

To support the wide range of 5G use cases in a cost-efficient way, network slicing has been considered a promising solution, which makes it possible to serve multiple customized and isolated network services on a common physical infrastructure. In this paper, we investigate the resource allocation problem of network slicing in multi-tenant networks where network resources can be used by low-priority tenants change dynamically due to the preemption of high-priority tenants. We formulate the problem as an energy-minimizing mathematical optimization problem considering practical constraints. Due to the dynamic characteristics of the problem, the complexity of the optimization problem is exceptionally high, making it impossible to solve the problem in real-time using traditional optimization approaches. With discovering the special structure of the problem, we propose a Dueling-Deep Q Network (DQN)-based algorithm to solve the problem efficiently. The experimental results show that the proposed algorithm outperforms compared algorithms in terms of total energy cost, runtime, and robustness.