DQN Algorithm Research Articles

A PDDQNLP Algorithm for Energy Efficient Computation Offloading in UAV-Assisted MEC

Unmanned aerial vehicle (UAV)-assisted mobile edge computing (MEC) is a promising technology to provide computational services to ground terminal devices (TDs) in remote areas or for emergency incidents with its flexibility and mobility. This paper aims to maximize the UAV’s energy efficiency while considering the fairness of offloading. We formulate this optimization problem by jointly considering the UAV flight time, the UAV 3D trajectory, the TD binary offloading decisions, and the time allocated to TDs, which is a mixed-integer nonlinear programming problem. The problem is transformed into two sub-problems and we propose a PDDQNLP (parametrized dueling deep Q-network and linear programming) algorithm based on the combination of deep reinforcement learning (DRL) and linear programming (LP) to address them. For the first sub-problem, a DRL-based algorithm is used to optimize the TD offloading decisions, the UAV trajectory, and the UAV flight time. The action space is hybrid that contains discrete actions (e.g., binary offloading) and continuous actions (e.g., UAV flight time). Therefore, we parameterize the action space and propose the PDDQN algorithm, which combines the DDPG algorithm for handling the continuous action space and the Dueling DQN algorithm for handling the discrete action space. According to the solution obtained from the first sub-problem, the LP is used to adjust the time allocated to TDs in the second sub-problem. The numerical results show that the PDDQNLP algorithm outperforms its counterparts.

Multiple Container Terminal Berth Allocation and Joint Operation Based on Dueling Double Deep Q-Network

In response to the evolving challenges of the integration and combination of multiple container terminal operations under berth water depth constraints, the multi-terminal dynamic and continuous berth allocation problem emerges as a critical issue. Based on computational logistics, the MDC-BAP is formulated to be a unique variant of the classical resource-constrained project scheduling problem, and modeled as a mixed-integer programming model. The modeling objective is to minimize the total dwelling time of linerships in ports. To address this, a Dueling Double DQN-based reinforcement learning algorithm is designed for the multi-terminal dynamic and continuous berth allocation problem A series of computational experiments are executed to validate the algorithm’s effectiveness and its aptitude for multiple terminal joint operation. Specifically, the Dueling Double DQN algorithm boosts the average solution quality by nearly 3.7%, compared to the classical algorithm such as Proximal Policy Optimization, Deep Q Net and Dueling Deep Q Net also have better results in terms of solution quality when benchmarked against the commercial solver CPLEX. Moreover, the performance advantage escalates as the number of ships increases. In addition, the approach enhances the service level at the terminals and slashes operation costs. On the whole, the Dueling Double DQN algorithm shows marked superiority in tackling complicated and large-scale scheduling problems, and provides an efficient, practical solution to MDC-BAP for port operators.

An Optimization Method for Non-IID Federated Learning Based on Deep Reinforcement Learning.

Federated learning (FL) is a distributed machine learning paradigm that enables a large number of clients to collaboratively train models without sharing data. However, when the private dataset between clients is not independent and identically distributed (non-IID), the local training objective is inconsistent with the global training objective, which possibly causes the convergence speed of FL to slow down, or even not converge. In this paper, we design a novel FL framework based on deep reinforcement learning (DRL), named FedRLCS. In FedRLCS, we primarily improved the greedy strategy and action space of the double DQN (DDQN) algorithm, enabling the server to select the optimal subset of clients from a non-IID dataset to participate in training, thereby accelerating model convergence and reaching the target accuracy in fewer communication epochs. In simulation experiments, we partition multiple datasets with different strategies to simulate non-IID on local clients. We adopt four models (LeNet-5, MobileNetV2, ResNet-18, ResNet-34) on the four datasets (CIFAR-10, CIFAR-100, NICO, Tiny ImageNet), respectively, and conduct comparative experiments with five state-of-the-art non-IID FL methods. Experimental results show that FedRLCS reduces the number of communication rounds required by 10-70% with the same target accuracy without increasing the computation and storage costs for all clients.

Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-Aided and Wireless-Powered Wireless Networks

In this paper, we focus on a wireless-powered sensor network coordinated by a multi-antenna access point (AP). Each node can generate sensing information and report the latest information to the AP using the energy harvested from the AP’s signal beamforming. We aim to minimize the average age-of-information (AoI) by adapting the nodes’ scheduling and the transmission control strategies jointly. To reduce the transmission delay, an intelligent reflecting surface (IRS) is used to enhance the channel conditions by controlling the AP’s beamforming strategy and the IRS’s phase shifting matrix. Considering dynamic data arrivals at different sensing nodes, we propose a hierarchical deep reinforcement learning (DRL) framework to for AoI minimization in two steps. The users’ transmission scheduling is firstly determined by the outer-loop DRL approach, e.g. the DQN or PPO algorithm, and then the inner-loop optimization is used to adapt either the uplink information transmission or downlink energy transfer to all nodes. A simple and efficient approximation is also proposed to reduce the inner-loop rum time overhead. Numerical results verify that the hierarchical learning framework outperforms typical baselines in terms of the average AoI and proportional fairness among different nodes.

SBFT: A BFT consensus mechanism based on DQN algorithm for industrial Internet of Thing

SBFT: A BFT consensus mechanism based on DQN algorithm for industrial Internet of Thing

UAV Path Planning Based on Deep Reinforcement Learning

Abstract Path planning is one of the very important aspects of UAV navigation control, which refers to the UAV searching for an optimal or near-optimal route from the starting point to the end point according to the performance indexes such as time, distance, et al. The path planning problem has a long history and has more abundant algorithms. The path planning problem has a long history and a rich set of algorithms, but most of the current algorithms require a known environment, however, in most cases, the environment model is difficult to describe and obtain, and the algorithms perform less satisfactorily. To address the above problems, this paper proposes a UAV path planning method based on deep reinforcement learning algorithm. Based on the OpenAI-GYM architecture, a 3D map environment model is constructed, with the map grid as the state set and 26 actions as the action set, which does not need an environment model and relies on its own interaction with the environment to complete the path planning task. The algorithm is based on stochastic process theory, modeling the path planning problem as a Markov Decision Process (MDP), fitting the UAV path planning decision function and state-action function, and designing the DQN algorithm model according to the state space, action space and network structure. The algorithm enables the intelligences to carry out strategy iteration efficiently. Through simulation, the DQN algorithm is verified to avoid obstacles and complete the path planning task in only about 160 rounds, which validates the effectiveness of the proposed path planning algorithm.

AUV Path Planning Considering Ocean Current Disturbance Based on Cloud Desktop Technology.

In the field of ocean energy detection, Autonomous Underwater Vehicles (AUVs) offer significant advantages in terms of manpower, resource, and energy efficiency. However, the unpredictable nature of the ocean environment, particularly the real-time changes in ocean currents, poses navigational risks for AUVs. Therefore, effective path planning in dynamic environments is crucial for AUVs to perform specific tasks. This paper addresses the static path planning problem and proposes a model called the noise net double DQN network with prioritized experience replay (N-DDQNP). The N-DDQNP model combines a noise network and a prioritized experience replay mechanism to address the limited exploration and slow convergence speed issues of the DQN algorithm, which are caused by the greedy strategy and uniform sampling mechanism. The proposed approach involves constructing a double DQN network with a priority experience replay and an exploration mechanism using the noise network. Second, a compound reward function is formulated to take into account ocean current, distance, and safety factors, ensuring prompt feedback during the training process. Regarding the ocean current, the reward function is designed based on the angle between the current direction and the AUV's heading direction, considering its impact on the AUV's speed. As for the distance factor, the reward is determined by the Euclidean distance between the current position and the target point. Furthermore, the safety factor considers whether the AUV may collide with obstacles. By incorporating these three factors, the compound reward function is established. To evaluate the performance of the N-DDQNP model, experiments were conducted using real ocean data in various complex ocean environments. The results demonstrate that the path planning time of the N-DDQNP model outperforms other algorithms in different ocean current scenarios and obstacle environments. Furthermore, a user console-AUV connection has been established using spice cloud desktop technology. The cloud desktop architecture enables intuitive observation of the AUV's navigation posture and the surrounding marine environment, facilitating safer and more efficient underwater exploration and marine resource detection tasks.

Data-intensive workflow scheduling strategy based on deep reinforcement learning in multi-clouds

With the increase development of Internet of Things devices, the data-intensive workflow has emerged as a new kinds of representation for IoT applications. Because most IoT systems are structured in multi-clouds environment and the data-intensive workflow has the characteristics of scattered data sources and distributed execution requirements at the cloud center and edge clouds, it brings many challenges to the scheduling of such workflow, such as data flow control management, data transmission scheduling, etc. Aiming at the execution constraints of business and technology and data transmission optimization of data-intensive workflow, a data-intensive workflow scheduling method based on deep reinforcement learning in multi-clouds is proposed. First, the execution constraints, edge node load and data transmission volume of IoT data workflow are modeled; then the data-intensive workflow is segmented with the consideration of business constraints and the first optimization goal of data transmission; besides, taking the workflow execution time and average load balancing as the secondary optimization goal, the improved DQN algorithm is used to schedule the workflow. Based on the DQN algorithm, the model reward function and action selection are redesigned and improved. The simulation results based on WorkflowSim show that, compared with MOPSO, NSGA-II, GTBGA and DQN, the algorithm proposed in this paper can effectively reduce the execution time of IoT data workflow under the condition of ensuring the execution constraints and load balancing of multi-clouds.

Deep reinforcement learning in NOMA-assisted UAV networks for path selection and resource offloading

This paper constructs a NOMA-based UAV-assisted Cellular Offloading (UACO) framework and designs a UAV path selection and resource offloading algorithm (UPRA) based on deep reinforcement learning. This paper focuses on the coupling relationship between path selection and resource offloading during the movement of UAVs. A joint optimization problem between UAV three-dimensional path design and resource offloading is proposed, considering the UAV’s autonomous obstacle avoidance in complex environments and the influence of obstacles in 3D space on the channel model. In particular, a constrained clustering-assignment algorithm is designed by improving the K-means algorithm and combining it with the assignment algorithm in order to achieve periodic clustering of random motion users and UAV task assignment. In addition, a semi-fixed hierarchical power allocation algorithm is embedded in the designed DQN algorithm to improve the convergence performance of the learning algorithm in this paper. Simulation results show that: the NOMA-based design is able to improve the spectrum utilization efficiency and communication throughput of the UAV network system. Compared with the artificial potential field, the proposed algorithm is able to solve the problem of falling into suboptimal solutions in path selection and improve the communication throughput. In addition, this paper explores the impact of the reward function on the training convergence and the results in deep reinforcement learning. The excellent adaptability of the designed algorithm in dynamic networks as well as in complex environments is demonstrated by random deployment of users and varying the maximum user movement speed.

Energy‐efficient resource allocation over wireless communication systems through deep reinforcement learning

SummaryAs the popularity of the Internet of Things (IoT) increases, so do the energy requirements of IoT terminal equipment. To address the energy shortage problem of equipment and ensure continuous and stable operation in light of renewable energy and an uncertain environment, a rational and efficient energy allocation strategy is required. This paper proposes a deep reinforcement learning energy allocation strategy that uses the DQN algorithm to directly interact with the unknown environment. The best energy allocation method is independent of environmental knowledge, and a pretraining algorithm is proposed to maximise the initialization state of the strategy. Experiments of comparison and simulation are conducted under various channel data circumstances. Results indicate that the proposed energy allocation strategy outperforms the current strategy in multiple channel conditions and has a high capacity for adaptation to changing conditions.

Intelligent multimedia content delivery in 5G/6G networks: A reinforcement learning approach

AbstractMultimedia content in 5G/6G networks makes safe, confidential, and efficient content delivery difficult. Intelligent systems that adapt to the ever‐changing network environment are needed to distribute multimedia content in these networks. Reinforcement learning (RL) can optimize multimedia content distribution based on network congestion, capacity, and user preferences. This study proposes RL‐based intelligent multimedia content distribution. RL algorithms learn from the network environment and generate optimum judgments incorporating several aspects of the suggested framework. The framework delivers multimedia material securely and privately with great quality. This study provides an intelligent multimedia content delivery architecture that uses RL approaches to solve 5G/6G content delivery problems. This research presents an RL system optimized with the double DQN algorithm having a reward of 51604.93 in 7000 episodes for efficient video file sharing on intracity buses. The RL agent balances network congestion and bandwidth by leveraging multiple sources such as bus and intersection caches and base stations, improving secure multimedia content delivery in 5G/6G networks and enhancing the passenger experience. The study confirms the system's effectiveness using reward and loss metrics and identifies potential future research directions. Future work could explore additional RL algorithms, scalability for larger networks, complex delivery scenarios, and integration with blockchain and edge computing for improved security and efficiency in multimedia content delivery.

Incentive-based demand response strategies for natural gas considering carbon emissions and load volatility

With the continuous development of the global economy, the demand for natural gas is increasing year by year. At the same time, the widespread adoption of emission reduction measures around the world prompts many countries to switch to cleaner energy sources, which is driving the growth in demand for natural gas. In order to regulate the balance of natural gas supply and demand, maintain the stable operation of natural gas systems and reduce environmental pollution, this paper proposes an incentive-based demand respond (DR) model for natural gas, which takes carbon emissions and load volatility into account, and in the end, it concludes an effective response strategy. In order to get the multi-objective incentive-based DR, first of all, the reward function is constructed by adopting the TOPSIS method, and then it is solved by using the DQN algorithm, in the end, a multi-objective reinforcement learning method based on DQN is proposed. The DR model is evaluated by adopting a real load data from Xi'an city and DR strategies are proposed for different gas users in two periods: the heating and non-heating periods. The DR strategy proposed in this paper can effectively increase gas users motivation, reduce peak loads, maximize the revenue of gas suppliers and gas users, mitigate load volatility and reduce carbon emissions.

Deep reinforcement learning enables adaptive-image augmentation for automated optical inspection of plant rust.

This study proposes an adaptive image augmentation scheme using deep reinforcement learning (DRL) to improve the performance of a deep learning-based automated optical inspection system. The study addresses the challenge of inconsistency in the performance of single image augmentation methods. It introduces a DRL algorithm, DQN, to select the most suitable augmentation method for each image. The proposed approach extracts geometric and pixel indicators to form states, and uses DeepLab-v3+ model to verify the augmented images and generate rewards. Image augmentation methods are treated as actions, and the DQN algorithm selects the best methods based on the images and segmentation model. The study demonstrates that the proposed framework outperforms any single image augmentation method and achieves better segmentation performance than other semantic segmentation models. The framework has practical implications for developing more accurate and robust automated optical inspection systems, critical for ensuring product quality in various industries. Future research can explore the generalizability and scalability of the proposed framework to other domains and applications. The code for this application is uploaded at https://github.com/lynnkobe/Adaptive-Image-Augmentation.git.

Research on energy management of hydrogen electric coupling system based on deep reinforcement learning

In this paper, a deep reinforcement learning-based energy optimization management method for hydrogen-electric coupling system is proposed for the conversion and utilization and joint optimization operation of hydrogen, wind and solar energy forms considering information uncertainty on the demand side of smart grid. Based on the wind energy, photovoltaic energy generation and load forecast information, the method uses deep Q network to simulate the energy management strategy set of the hydrogen-electric coupling system, and obtains the optimal strategy through reinforcement learning to finally realize the optimal operation of the hydrogen-electric coupling system based on the demand response. Firstly, based on the energy management model, a research framework and equipment model for integrated energy systems is established. On the basis of fundamental theories of reinforcement learning framework, Q-learning algorithm and DQN algorithm, the empirical replay mechanism and freezing parameter mechanism to improve the performance of DQN are analyzed, and the energy management and optimization of integrated energy system is completed with the objective of economy. By comparing the performance of DQN algorithms with different parameters in integrated energy system energy management, the simulation results demonstrate the improvement of algorithm performance after inheriting the set of strategies, and verify the feasibility and superiority of deep reinforcement learning compared to genetic algorithm in integrated energy system energy management applications.

Distributed Dynamic Pricing Strategy Based on Deep Reinforcement Learning Approach in a Presale Mechanism

Despite the emergence of a presale mechanism that reduces manufacturing and ordering risks for retailers, optimizing the real-time pricing strategy in this mechanism and unknown demand environment remains an unsolved issue. Consequently, we propose an automatic real-time pricing system for e-retailers under the inventory backlog impact in the presale mode, using deep reinforcement learning technology based on the Dueling DQN algorithm. This system models the multicycle pricing problem with a finite sales horizon as a Markov decision process (MDP) to cope with the uncertain environment. We train and evaluate the proposed environment and agent in a simulation environment and compare it with two tabular reinforcement learning algorithms (Q-learning and SARSA). The computational results demonstrate that our proposed real-time pricing learning framework for joint inventory impact can effectively maximize retailers’ profits and has universal applicability to a wide range of presale models. Furthermore, according to a series of experiments, we find that retailers should not neglect the impact of the presale or previous prices on consumers’ purchase behavior. If consumers pay more attention to past prices, the retailer must decrease the current price. When the cost of inventory backlog increases, they need to offer deeper discounts in the early selling period. Additionally, introducing blockchain technology can improve the transparency of commodity traceability information, thus increasing consumer demand for purchase.

Energy-Efficient Coverage and Capacity Enhancement With Intelligent UAV-BSs Deployment in 6G Edge Networks

With the development of 5G/6G networks, the number of wireless users is growing exponentially, and the application scenarios are increasingly diversified. Using unmanned aerial vehicles as base stations (UAV-BSs) to serve ground users has become a trend for wide area coverage and capacity enhancement for rapid access of service in 6G networks. However, as UAV-BSs have limited energy or battery storage, solutions to optimize energy efficiency while providing high-quality services are necessary. Therefore, this paper mainly concentrates on the energy-efficient deployment of coverage-aimed UAV-BSs (Co-UAV-BSs) and capacity-aimed UAV-BSs (Ca-UAV-BSs) for the coverage and capacity enhancement of ground communication under disaster areas or burst data traffic. First, Co-UAV-BSs are deployed with DQN algorithm to to get the UAV-BSs’ optimal flight paths, which mainly adopted to detect out of service users in such areas. Then the users are completely clustered based on the detection results. After that, Co-UAV-BSs and Ca-UAV-BSs are deployed hierarchically based on the user distribution and sought to optimize the energy efficiency with acceptable user services. Still, DQN algorithm and the A3C algorithm are used for obtaining all the UAV-BSs’ location deployment and users’ best connections. The simulation results show that the dynamic flying path requires less energy than the fixed path for user detecting. For the coverage and capacity enhancement, it reveals the solution we proposed could provide high-quality service for users with high energy efficiency comparing to traditional algorithms.

Research on international logistics supply chain management strategy based on deep reinforcement learning

Abstract The use of deep reinforcement learning algorithms for strategy formulation in supply chain management enables the nodes in the supply chain to better improve their management strategies. In this paper, a supply chain model is constructed as a starting point, and deep reinforcement learning algorithms are introduced on this basis. Firstly, the decision problem of uncertainty is handled by the reinforcement learning method of functions, and the DQN algorithm (deep neural network algorithm) is divided into two parts for iterative rules. Then the target network is established to make the iterative process more stable, to improve the convergence of the algorithm, evaluate the loss function in the training process of the network, and to determine its influence factor. Then the neural network is used to improve the iteration rule, improve the output layer, select the final action, and define the model expectation reward. Finally, the Bellman equation is fitted to the function by a deep neural network to calculate the final result. The experimental results show that by analyzing and constructing the cost of international logistics under supply chain management, the capacity utilization rate of ocean freight link is 57% The unloading link is 74% and the total capacity utilization rate is calculated as 76%. It shows that using deep reinforcement learning algorithms under international logistics supply chain management is feasible and necessary for improving the management strategy research of supply chains.

Power Battery Scheduling Optimization Based on Double DQN Algorithm with Constraints

Power battery scheduling optimization can improve the service life of the battery, but the existing heuristic algorithm has poor adaptability, and the capacity fluctuates significantly in the cycle aging process, which makes it easy to fall into the local optimal. To overcome these problems, we take the battery cycle life maximization as the goal, propose a reinforcement learning scheduling optimization model with temperature and internal resistance difference constraints, so as to determine whether to charge or discharge during battery cycle aging. We do this using the deep−learning−based battery capacity estimation model as the learning environment for the agent, using the Double DQN algorithm to train the agent, and proposing the principal component analysis method to reduce the dimension of the state space. These experiments, using multiple publicly available battery aging data sets, show that the principal component analysis method and the constraint functions reduce the computational time to find the optimal solution, providing the possibility of obtaining larger reward values. Meanwhile, the trained model effectively extends the cycle life of the battery, and has good adaptivity. It can automatically adjust parameters with the battery aging process to develop optimal charging and discharging protocols for power batteries with different chemical compositions.

Dynamic Selection Slicing-Based Offloading Algorithm for In-Vehicle Tasks in Mobile Edge Computing

With the surge in tasks for in-vehicle terminals, the resulting network congestion and time delay cannot meet the service needs of users. Offloading algorithms are introduced to handle vehicular tasks, which will greatly improve the above problems. In this paper, the dependencies of vehicular tasks are represented as directed acyclic graphs, and network slices are integrated within the edge server. The Dynamic Selection Slicing-based Offloading Algorithm for in-vehicle tasks in MEC (DSSO) is proposed. First, a computational offloading model for vehicular tasks is established based on available resources, wireless channel state, and vehicle loading level. Second, the solution of the model is transformed into a Markov decision process, and the combination of the DQN algorithm and Dueling Network from deep reinforcement learning is used to select the appropriate slices and dynamically update the optimal offloading strategy for in-vehicle tasks in the effective interval. Finally, an experimental environment is set up to compare the DSSO algorithm with LOCAL, MINCO, and DJROM, the results show that the system energy consumption of DSSO algorithm resources is reduced by 10.31%, the time latency is decreased by 22.75%, and the ratio of dropped tasks is decreased by 28.71%.

Research on the Control Method of Urban Expressway Weaving Area

With the rapid development of urban expressway construction, there are some problems in the actual operation process, which leads to the traffic efficiency not reaching the expected level. Due to the long-term and large amount of actual data is difficult to obtain, this paper uses the method of combining actual data and simulation to explore the traffic operation law of urban expressway weaving area, analyzes the factors affecting the traffic efficiency of weaving area, and applies the variable speed limit guidance control method based on DQN algorithm, which can effectively improve the traffic efficiency of urban expressway weaving area.