Deep Q-learning Algorithm Research Articles

Enhancing Helpdesk Efficiency through Management Information System – A Resource Allocation Study in Technology Firms

Purpose: This paper aims to enhance helpdesk efficiency by integrating deep reinforcement learning for resource allocation and customer satisfaction improvement. Methodology/Approach: The study implemented a deep Q-learning algorithm within a helpdesk management system. The dataset was divided into training and testing sets in an 80:20 ratio. The architectural and computational parameters of the model were optimised, focusing on resource utilisation and workload distribution. Findings: The proposed method reduced the normal resolution time from 3.5 hours to 2.65 hours, representing a 24.3% improvement. Customer satisfaction improved, averaging a score of 3.85. The allocation of support staff workloads was enhanced, leading to a more balanced distribution across different locations. Research Limitation/Implication: Various parameter patterns for the proposed method were tested, revealing the approach's computational expense. Originality/Value of paper: The study proposes a novel use of deep Q-learning for helpdesk management, significantly improving classification accuracy and workload distribution over conventional methods.

Learning and fine-tuning a generic value-selection heuristic inside a constraint programming solver

Constraint programming is known for being an efficient approach to solving combinatorial problems. Important design choices in a solver are the branching heuristics, designed to lead the search to the best solutions in a minimum amount of time. However, developing these heuristics is a time-consuming process that requires problem-specific expertise. This observation has motivated many efforts to use machine learning to automatically learn efficient heuristics without expert intervention. Although several generic variable-selection heuristics are available in the literature, the options for value-selection heuristics are more scarce. We propose to tackle this issue by introducing a generic learning procedure that can be used to obtain a value-selection heuristic inside a constraint programming solver. This has been achieved thanks to the combination of a deep Q-learning algorithm, a tailored reward signal, and a heterogeneous graph neural network. Experiments on graph coloring, maximum independent set, maximum cut, and minimum vertex cover problems show that this framework competes with the well-known impact-based and activity-based search heuristics and can find solutions close to optimality without requiring a large number of backtracks. Additionally, we observe that fine-tuning a model with a different problem class can accelerate the learning process.

Research on robot path planning and obstacle avoidance algorithm in dynamic environment based on deep reinforcement learning

Abstract. In dynamic environments, robot path planning and obstacle avoidance are critical tasks, especially in applications such as autonomous driving, industrial automation, and mobile robotics. These tasks are inherently challenging due to the unpredictability of the environment and the need for real-time decision-making. This paper seeks to address these challenges by developing and analyzing both traditional and optimized models for robot navigation. The initial model utilizes a basic Q-learning algorithm, which provides a straightforward approach to learning from the environment but often struggles with the complexity of dynamic scenarios. To this end, an optimized model is developed that combines the Double Deep Q-Learning algorithm (Double DQN) in conjunction with heuristic strategies. The research employs the MATLAB Reinforcement Learning Toolbox to implement and train these models, and utilizes a simulated environment with dynamic obstacles as a testing site. The simulation generates the necessary data to allow for comprehensive testing and evaluation of the models performance. The results show that the optimized model greatly exceeds the initial model in terms of path planning efficiency and obstacle avoidance capabilities, and that the combination of advanced reinforcement learning techniques and heuristic strategies is extremely important for enhancing the performance and reliability of robotic systems in complex, dynamic environments, offering valuable insights for future applications in various fields of robotics.

Sea-Based UAV Network Resource Allocation Method Based on an Attention Mechanism

As humans continue to exploit the ocean, the number of UAV nodes at sea and the demand for their services are increasing. Given the dynamic nature of marine environments, traditional resource allocation methods lead to inefficient service transmission and ping-pong effects. This study enhances the alignment between network resources and node services by introducing an attention mechanism and double deep Q-learning (DDQN) algorithm that optimizes the service-access strategy, curbs action outputs, and improves service-node compatibility, thereby constituting a novel method for UAV network resource allocation in marine environments. A selective suppression module minimizes the variability in action outputs, effectively mitigating the ping-pong effect, and an attention-aware module is designed to strengthen node-service compatibility, thereby significantly enhancing service transmission efficiency. Simulation results indicate that the proposed method boosts the number of completed services compared with the DDQN, soft actor–critic (SAC), and deep deterministic policy gradient (DDPG) algorithms and increases the total value of completed services.

A comparative study of DQN and D3QN for HVAC system optimization control

Ensuring the optimal performance of Heating, Ventilation, and Air Conditioning (HVAC) systems is paramount for achieving energy efficiency. This paper investigates the application of deep reinforcement learning algorithms in HVAC system control, aiming to identify the most suitable Q-network structure for optimizing HVAC systems and comparing the performance of Deep Q-learning (DQN) and Double Dueling Deep Q-learning (D3QN) algorithms. Initially, this paper evaluates and analyses existing literature to perform a normalization treatment on the state space. Through systematic simulation and rigorous data analysis, the impact of the Q-network structure on the efficacy of the DQN and D3QN algorithms is evaluated, resulting in the proposal of specific values for the Q-network structures within these two algorithms. Subsequently, comparisons are drawn on the optimization effectiveness, stability and reliability of these algorithms across diverse engineering projects. Results highlight the superiority of the D3QN algorithm over the DQN algorithm regarding both optimization effectiveness and stability across all evaluated projects. The proposed efficient Q-network structure comprises two hidden layers, with 64 and 12 neurons respectively in each layer. The findings of this paper are crucial in providing insights for HVAC systems control optimization using reinforcement learning and pave the way for advanced research and applications in the future.

Achieving accurate trajectory predicting and tracking for autonomous vehicles via reinforcement learning-assisted control approaches

In complex urban traffic scenarios, autonomous vehicles face significant challenges in adapting to diverse and dynamic traffic conditions. Reward-based reinforcement learning has emerged as an effective approach to tackle these challenges. This paper presents a novel method that combines deep reinforcement learning with automotive dynamics systems. Building upon the Double Deep Q-learning algorithm, our approach integrates a Recurrent Neural Network with Gated Recurrent Units to enhance the environmental exploration capabilities of autonomous vehicles. To obtain more precise reward values, we introduce a trajectory tracking algorithm based on a combination of proportional-integral-derivative control and feedforward control within the automotive dynamics system. The proportional-integral-derivative controller is utilized for longitudinal control, while the Error-Optimized feedforward controller enhances lateral control, thereby improving trajectory tracking accuracy. Finally, extensive simulation experiments are conducted to evaluate the proposed method, comparing it against other baseline methods in terms of vehicle following and lane-changing scenarios. The results demonstrate that our approach significantly improves both the reward values and control performance of the algorithm.

Multi-agent deep reinforcement learning for hyperspectral band selection with hybrid teacher guide

Due to the presence of noisy and highly redundant bands in hyperspectral images (HSIs), band selection serves as a key preprocessing for downstream classification tasks. Recently, deep reinforcement learning (DRL) has been developed as a new trend for band selection of HSIs. Existing DRL-based methods often adopt single-agent, which are prone to fall into local optima due to an excessive action space. The multi-agent methods provide a feasible solution, but often require too much computation. To address these problems, a novel multi-agent DRL method with hybrid teacher guide (MH-DRL) is proposed for band selection of HSIs. In MH-DRL, each agent corresponding to a spectral band decides whether this band is selected. Moreover, a presentation-evaluation network (PE-Net) is constructed to design the reward by evaluating the candidate band subsets without any fine-tuning and represent the state by extracting the spatial-spectral features of HSIs. Then, three kinds of experienced band selection models are regarded as the teachers and designed to participate in the band exploration of DRL, which can improve the learning effectiveness and efficiency by accumulating the external knowledge from diverse teacher models. Finally, deep Q-learning algorithm is designed to update the agents and improve their self-learning ability from continuous exploration. Experimental results on three widely-used HSI data verify the performance of the proposed method better than some advanced band selection algorithms of HSIs. Our source code is available at https://github.com/jiefeng0109/MH-DRL.

UAV path planning algorithm based on Deep Q-Learning to search for a floating lost target in the ocean

In the context of real world application, Search and Rescue Missions on the ocean surface remain a complex task due to the large-scale area and the forces of the ocean currents, spreading lost targets and debris in an unpredictable way. In this work, we present a Path Planning Approach to search for a lost target on ocean surface using a swarm of UAVs. The combination of GlobCurrent dataset and a Lagrangian simulator is used to determine where the particles are moved by the ocean currents forces while Deep Q-learning algorithm is applied to learn from their dynamics. The evaluation results of the trained models show that our search strategy is effective and efficient. Over a total search area (red Sea zone), surface of 453422 Km2, we have shown that our strategy Search Success Rate is 98.61%, the maximum Search Time to detection is 15 days and the average Search Time to detection is almost 15 h.

Enhanced LSTM‐DQN algorithm for a two‐player zero‐sum game in three‐dimensional space

AbstractTo tackle the challenges presented by the two‐player zero sum game (TZSG) in three‐dimensional space, this study introduces an enhanced deep Q‐learning (DQN) algorithm that utilizes long short term memory (LSTM) network. The primary objective of this algorithm is to enhance the temporal correlation of the TZSG in three‐dimensional space. Additionally, it incorporates the hindsight experience replay (HER) mechanism to improve the learning efficiency of the network and mitigate the issue of the “sparse reward” that arises from prolonged training of intelligence in solving the TZSG in the three‐dimensional. Furthermore, this method enhances the convergence and stability of the overall solution.An intelligent training environment centred around an airborne agent and its mutual pursuit interaction scenario was designed to proposed approach's effectiveness. The algorithm training and comparison results show that the LSTM‐DQN‐HER algorithm outperforms similar algorithm in solving the TZSG in three‐dimensional space. In conclusion, this paper presents an improved DQN algorithm based on LSTM and incorporates the HER mechanism to address the challenges posed by the TZSG in three‐dimensional space. The proposed algorithm enhances the solution's temporal correlation, learning efficiency, convergence, and stability. The simulation results confirm its superior performance in solving the TZSG in three‐dimensional space.

Federated deep reinforcement learning for mobile robot navigation

Navigation of a mobile robot in an unknown environment ensuring the safety of the robot and its surroundings is of utmost importance. Traditional methods, such as pathplanning algorithms, simultaneous localization and mapping, computer vision, and fuzzy techniques, have been employed to address this challenge. However, to achieve better generalization and self-improvement capabilities, reinforcement learning has gained significant attention. The concern of privacy issues in sharing data is also rising in various domains. In this study, a deep reinforcement learning strategy is applied to the mobile robot to move from its initial position to a destination. Specifically, the Deep Q-Learning algorithm has been used for this purpose. This strategy is trained using a federated learning approach to overcome privacy issues and to set a foundation for further analysis of distributed learning. The application scenario considered in this work involves the navigation of a mobile robot to a charging point within a greenhouse environment. The results obtained indicate that both the traditional deep reinforcement learning and federated deep reinforcement learning frameworks are providing 100% success rate. However federated deep reinforcement learning could be a better alternate since it overcomes the privacy issue along with other advantages discussed in this paper.

Deadline-aware task offloading in vehicular networks using deep reinforcement learning

Smart vehicles have a rising demand for computation resources, and recently vehicular edge computing has been recognized as an effective solution. Edge servers deployed in roadside units are capable of accomplishing tasks beyond the capacity which is embedded inside the vehicles. However, the main challenge is to carefully select the tasks to be offloaded considering the deadlines, and in order to reduce energy consumption, while delivering a good performance. In this paper, we consider a vehicular edge computing network in which multiple cars are moving at non-constant speed and produce tasks at each time slot. Then, we propose a task offloading algorithm, aware of the vehicle’s direction, based on Rainbow, a deep Q-learning algorithm combining several independent improvements to the deep Q-network algorithm. This is to overcome the conventional limits and to reach an optimal offloading policy, by effectively incorporating the computation resources of edge servers to jointly minimize average delay and energy consumption. Real-world traffic data is used to evaluate the performance of the proposed approach compared to other algorithms, in particular deep Q-network, double deep Q-network, and deep recurrent Q-network. Results of the experiments show an average reduction of 18% and 15% in energy consumption and delay, respectively, when using the proposed Rainbow deep Q-network based algorithm in comparison to the state-of-the-art. Moreover, the stability and convergence of the learning process have significantly improved by adopting the Rainbow algorithm.

Shovel allocation and scheduling for open-pit mining using deep reinforcement learning

ABSTRACT The open-pit production system is a highly dynamic and uncertain environment with complex interactions between haulage and loading equipment on a shared road network. One of the key decisions in open-pit short-term planning is the allocation sequence of shovels to mining faces to meet the production targets established by long- and medium-term strategic plans. Deep Reinforcement Learning(DRL) techniques are commonly used in dynamic production environments. In this approach, an agent is trained on a simulation of the production system to learn the optimal decisions based on the system’s current state. This paper proposes a DRL approach based on the Deep Q-Learning algorithm to obtain a robust shovel allocation plan for open-pit short-term planning. First, a discrete-event simulation of the mining production system incorporating trucks, shovels, crushers, waste dumps, and the road network is created. This simulation models the uncertainties of each component’s operating cycle based on historical activity records, and it is used to train the DRL agent. The goal is to learn a robust shovel allocation strategy for the next production quarter, 3 months, to meet the tonnes per hour (TPH) production target to be delivered to the crusher feeds by interacting with the production simulator. The framework is tested in an iron ore open-pit mine case study where the shovel allocation agent successfully learns a strategy that consistently delivers the desired production target.

Optimizing warfarin dosing for patients with atrial fibrillation using machine learning

While novel oral anticoagulants are increasingly used to reduce risk of stroke in patients with atrial fibrillation, vitamin K antagonists such as warfarin continue to be used extensively for stroke prevention across the world. While effective in reducing the risk of strokes, the complex pharmacodynamics of warfarin make it difficult to use clinically, with many patients experiencing under- and/or over- anticoagulation. In this study we employed a novel implementation of deep reinforcement learning to provide clinical decision support to optimize time in therapeutic International Normalized Ratio (INR) range. We used a novel semi-Markov decision process formulation of the Batch-Constrained deep Q-learning algorithm to develop a reinforcement learning model to dynamically recommend optimal warfarin dosing to achieve INR of 2.0–3.0 for patients with atrial fibrillation. The model was developed using data from 22,502 patients in the warfarin treated groups of the pivotal randomized clinical trials of edoxaban (ENGAGE AF-TIMI 48), apixaban (ARISTOTLE) and rivaroxaban (ROCKET AF). The model was externally validated on data from 5730 warfarin-treated patients in a fourth trial of dabigatran (RE-LY) using multilevel regression models to estimate the relationship between center-level algorithm consistent dosing, time in therapeutic INR range (TTR), and a composite clinical outcome of stroke, systemic embolism or major hemorrhage. External validation showed a positive association between center-level algorithm-consistent dosing and TTR (R2 = 0.56). Each 10% increase in algorithm-consistent dosing at the center level independently predicted a 6.78% improvement in TTR (95% CI 6.29, 7.28; p < 0.001) and a 11% decrease in the composite clinical outcome (HR 0.89; 95% CI 0.81, 1.00; p = 0.015). These results were comparable to those of a rules-based clinical algorithm used for benchmarking, for which each 10% increase in algorithm-consistent dosing independently predicted a 6.10% increase in TTR (95% CI 5.67, 6.54, p < 0.001) and a 10% decrease in the composite outcome (HR 0.90; 95% CI 0.83, 0.98, p = 0.018). Our findings suggest that a deep reinforcement learning algorithm can optimize time in therapeutic range for patients taking warfarin. A digital clinical decision support system to promote algorithm-consistent warfarin dosing could optimize time in therapeutic range and improve clinical outcomes in atrial fibrillation globally.

Reinforcement learning-based motion control for snake robots in complex environments

AbstractSnake robots can move flexibly due to their special bodies and gaits. However, it is difficult to plan their motion in multi-obstacle environments due to their complex models. To solve this problem, this work investigates a reinforcement learning-based motion planning method. To plan feasible paths, together with a modified deep Q-learning algorithm, a Floyd-moving average algorithm is proposed to ensure smoothness and adaptability of paths for snake robots’ passing. An improved path integral algorithm is used to work out gait parameters to control snake robots to move along the planned paths. To speed up the training of parameters, a strategy combining serial training, parallel training, and experience replaying modules is designed. Moreover, we have designed a motion planning framework consists of path planning, path smoothing, and motion planning. Various simulations are conducted to validate the effectiveness of the proposed algorithms.

An improved deep Q-learning algorithm for a trade-off between energy consumption and productivity in batch scheduling

The single-batch machine, commonly found in industrial manufacturing, can concurrently process a group of jobs in variable-speed batches, leading to fluctuating levels of both energy consumption and processing time. Identifying an optimal balance between total energy consumption and makespan presents a challenge due to the intricate nature of their relationship in real-world scenarios. This study introduces a mixed-integer programming model designed to determine optimal machine operating states for diverse workloads, meeting customer requirements while concurrently achieving energy savings for sustainability. An energy-aware batch scheduling deep Q-learning network (EBSDQN) framework has been created, encompassing sequencing policies, batching rules, and speed adjustment policies. This framework is explicitly crafted to tackle the NP-hard problem, a challenge that commonly perplexes commercial solvers like Gurobi when seeking optimal solutions within a 360-second time frame for small-scale instances. The EBSDQN design is fortified with efficient action decoding policies, refined reward assessments, exploitative neighborhood rules, and a streamlined buffer training process. This approach not only conserves training time but also adapts dynamically in accordance with the principles of the Markov Decision Process (MDP) and deep neural network. The developed algorithm exhibits impressive robustness, reaching convergence after 600 episodes of training. In separate comparisons with a commercial solver, two single-objective algorithms, and two multi-objective algorithms, our algorithm consistently demonstrates superior overall performance across the same instances. In the worst-case analysis, further examination delves into the influences of job features. In summary, this study underscores the high sensitivity of both TEC and Cmax to job processing time distribution, suggesting the prioritization of TEC over Cmax in optimization for sustainable planning and operations. Furthermore, this research has the potential to accelerate the integration of artificial intelligence into the manufacturing sector.

A deep Q-learning approach to optimize ordering and dynamic pricing decisions in the presence of strategic customers

In this paper, we present an optimization method to analyze the simultaneous decisions on dynamic pricing and ordering quantities for seasonal products, by a retailer in monopolistic condition. Customers are assumed to be strategic and may postpone their purchase to get a lower price in future. The problem has been investigated in the context of multiple substitute products. We have developed a model based on deep neural networks to estimate customers’ demand. The problem is complex and cannot be solved using classical optimization methods. Therefore, we have developed a reinforcement learning algorithm called deep Q-learning algorithm (DQL) to solve the problem. The proposed algorithm is a combination of a Q-learning algorithm and two deep neural networks for the primary and discount sales periods, which uses the neural network to estimate the Q-values in a large space of states and actions. The performances of the demand model and the proposed optimization algorithm have been tested using a real-world dataset taken from the clothing industry. The results of our experiments demonstrate that the proposed demand model performs better than a fully connected neural network-based model and a latent class model tested in this paper. Furthermore, the performance of the DQL algorithm is significantly superior to those of two simulated annealing and genetic algorithms. In addition, the results of a comparison between the DQL algorithm and another reinforcement learning algorithm called State-Action-Reward-State-Action (SARSA) indicate that the proposed algorithm results in higher revenues and takes less time to converge. Consequently, the proposed algorithm has a high potential for solving such a large scale integrated pricing and ordering optimization problem.

Experimental Research on Avoidance Obstacle Control for Mobile Robots Using Q-Learning (QL) and Deep Q-Learning (DQL) Algorithms in Dynamic Environments

This study provides simulation and experimental results on techniques for avoiding static and dynamic obstacles using a deep Q-learning (DQL) reinforcement learning algorithm for a two-wheel mobile robot with independent control. This method integrates the Q-learning (QL) algorithm with a neural network, where the neural networks in the DQL algorithm act as approximators for the Q matrix table for each pair (state–action). The effectiveness of the proposed solution was confirmed through simulations, programming, and practical experimentation. A comparison was drawn between the DQL algorithm and the QL algorithm. Initially, the mobile robot was connected to the control script using the Robot Operating System (ROS). The mobile robot was programmed in Python within the ROS operating system, and the DQL controller was programmed in Gazebo software. The mobile robot underwent testing in a workshop with various experimental scenarios considered. The DQL controller displayed improvements in computation time, convergence time, trajectory planning accuracy, and obstacle avoidance. As a result, the DQL controller surpassed the QL algorithm in terms of performance.

A Search and Matching Model of Intersectoral Labor Mobility With an Extension of a Deep Q-learning Algorithm

A Search and Matching Model of Intersectoral Labor Mobility With an Extension of a Deep Q-learning Algorithm

Dynamic adaptation of contention window boundaries using deep Q networks in UAV swarms

In flying ad hoc networks (FANET), medium access layer (MAC) protocols play an essential role in ensuring better network performance. The effective utilization of network resources and providing access fairness are key research issues in this area, and the contention window size directly influences these factors. This paper mainly focuses on addressing the upper and lower boundaries of the contention window in Carrier Sense Multiple Access/Collision Avoidance (CSMA/CA) with a binary exponential back-off algorithm. Existing solutions include static and dynamic window adjustment techniques only to update the upper boundary of the contention window size. However, the static window adjustment technique is unsuitable for changing network conditions, while the dynamic window adjustment technique may need to be more efficient in dense network environments. The proposed solution uses a Deep Q-learning algorithm(DQN) framework to adjust the contention window adaptively based on the reward function, resulting in efficient and dynamic adaptation to changing network conditions among unmanned aerial vehicles (UAV). The proposed methodology is evaluated through simulations, and performance is measured using channel utilization, efficiency, delay, and collision rate. The simulation findings show that, in comparison to generic MAC protocols, the proposed algorithm is 5% more efficient in throughput and incurs a four-fold reduction in delay.

General deep learning framework for emissivity engineering

Wavelength-selective thermal emitters (WS-TEs) have been frequently designed to achieve desired target emissivity spectra, as a typical emissivity engineering, for broad applications such as thermal camouflage, radiative cooling, and gas sensing, etc. However, previous designs require prior knowledge of materials or structures for different applications and the designed WS-TEs usually vary from applications to applications in terms of materials and structures, thus lacking of a general design framework for emissivity engineering across different applications. Moreover, previous designs fail to tackle the simultaneous design of both materials and structures, as they either fix materials to design structures or fix structures to select suitable materials. Herein, we employ the deep Q-learning network algorithm, a reinforcement learning method based on deep learning framework, to design multilayer WS-TEs. To demonstrate the general validity, three WS-TEs are designed for various applications, including thermal camouflage, radiative cooling and gas sensing, which are then fabricated and measured. The merits of the deep Q-learning algorithm include that it can (1) offer a general design framework for WS-TEs beyond one-dimensional multilayer structures; (2) autonomously select suitable materials from a self-built material library and (3) autonomously optimize structural parameters for the target emissivity spectra. The present framework is demonstrated to be feasible and efficient in designing WS-TEs across different applications, and the design parameters are highly scalable in materials, structures, dimensions, and the target functions, offering a general framework for emissivity engineering and paving the way for efficient design of nonlinear optimization problems beyond thermal metamaterials.