Deep Q-learning Approach Research Articles

A scalable Deep Q-Learning approach for hot stamping process under dynamic control environment

Despite the significant advancements in Artificial Intelligence and its applications, traditional control techniques still dominate industrial and manufacturing processes. These techniques often rely on static setups and default values, which make them resistant to environmental changes. However, in dynamic and uncertain contexts, real-time process adaptation offers significant advantages in decision-making and control. In this research, we present a novel approach utilising Deep Reinforcement Learning for real-time dynamic parameter adaptation and autonomous control of processes in unpredictable environments. The effectiveness of our proposal is illustrated through the development and training of a Deep Q-Learning control system tailored for optimising a hot stamping process. The control model aims to minimise defect rates and enhance efficiency by reducing forming times during batch productions. Comparative analysis against conventional industry practices, characterised by static and non-adaptive time settings, validates the efficacy of our solution, with improvements of over 20% in the times, depending on the scenario considered. Experimental evaluation in a semi-industrial pilot setting not only demonstrates the adaptability of our approach but also highlights its potential scalability for diverse industrial applications.

Heat Conduction Control Using Deep Q-Learning Approach with Physics-Informed Neural Networks

As modern systems become more complex, their control strategy no longer relies only on measurement data from probes; it also requires information from mathematical models for non-measurable places. On the other hand, those mathematical models can lead to unbearable computation times due to their own complexity, making the control process non-viable. To overcome this problem, it is possible to implement any kind of surrogate model that enables the computation of such estimates within an acceptable time frame, which allows for making decisions. Using a Physics-Informed Neural Network as a surrogate model, it is possible to compute the temperature distribution at each time step, replacing the need for running direct numerical simulations. This approach enables the use of a Deep Reinforcement Learning algorithm to train a control strategy. On this work, we considered a one-dimensional heat conduction problem, in which temperature distribution feeds a control system. Such control system has the objective of reacing and maintaining constant temperature value at a specific location of the 1D problem by activating a heat source; the desired location somehow cannot be directly measured so, the PINN approach allows to estimate its temperature with a minimum computational workload. With this approach, the control training becomes much faster without the need of performing numerical simulations or laboratory measurements.

A deep Q-learning approach to optimize ordering and dynamic pricing decisions in the presence of strategic customers

In this paper, we present an optimization method to analyze the simultaneous decisions on dynamic pricing and ordering quantities for seasonal products, by a retailer in monopolistic condition. Customers are assumed to be strategic and may postpone their purchase to get a lower price in future. The problem has been investigated in the context of multiple substitute products. We have developed a model based on deep neural networks to estimate customers’ demand. The problem is complex and cannot be solved using classical optimization methods. Therefore, we have developed a reinforcement learning algorithm called deep Q-learning algorithm (DQL) to solve the problem. The proposed algorithm is a combination of a Q-learning algorithm and two deep neural networks for the primary and discount sales periods, which uses the neural network to estimate the Q-values in a large space of states and actions. The performances of the demand model and the proposed optimization algorithm have been tested using a real-world dataset taken from the clothing industry. The results of our experiments demonstrate that the proposed demand model performs better than a fully connected neural network-based model and a latent class model tested in this paper. Furthermore, the performance of the DQL algorithm is significantly superior to those of two simulated annealing and genetic algorithms. In addition, the results of a comparison between the DQL algorithm and another reinforcement learning algorithm called State-Action-Reward-State-Action (SARSA) indicate that the proposed algorithm results in higher revenues and takes less time to converge. Consequently, the proposed algorithm has a high potential for solving such a large scale integrated pricing and ordering optimization problem.

Influence Maximization in Dynamic Networks Using Reinforcement Learning

Influence maximization (IM) has been widely studied in recent decades, aiming to maximize the spread of influence over networks. Despite many works for static networks, fewer research studies have been dedicated to the IM problem for dynamic networks, which creates many challenges. An IM method for such an environment, should consider its dynamics and perform well under different network structures. To fulfill this objective, more computations are required. Hence, an IM approach should be efficient enough to be applicable for the ever-changing structure of a network. In this research, an IM method for dynamic networks has been proposed which uses a deep Q-learning (DQL) approach. To learn dynamic features from the network and retain previously learned information, incremental and transfer learning methods have been applied. Experiments substantiate the good performance of the DQL methods and their superiority over compared methods on larger sizes of tested synthetic and real-world networks. These experiments illustrate better performance for incremental and transfer learning methods on real-world networks.

Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care.

Medical treatments often involve a sequence of decisions, each informed by previous outcomes. This process closely aligns with reinforcement learning (RL), a framework for optimizing sequential decisions to maximize cumulative rewards under unknown dynamics. While RL shows promise for creating data-driven treatment plans, its application in medical contexts is challenging due to the frequent need to use sparse rewards, primarily defined based on mortality outcomes. This sparsity can reduce the stability of offline estimates, posing a significant hurdle in fully utilizing RL for medical decision-making. We introduce a deep Q-learning approach to obtain more reliable critical care policies by integrating relevant but noisy frequently measured biomarker signals into the reward specification without compromising the optimization of the main outcome. Our method prunes the action space based on all available rewards before training a final model on the sparse main reward. This approach minimizes potential distortions of the main objective while extracting valuable information from intermediate signals to guide learning. We evaluate our method in off-policy and offline settings using simulated environments and real health records from intensive care units. Our empirical results demonstrate that our method outperforms common offline RL methods such as conservative Q-learning and batch-constrained deep Q-learning. By disentangling sparse rewards and frequently measured reward proxies through action pruning, our work represents a step towards developing reliable policies that effectively harness the wealth of available information in data-intensive critical care environments.

Joint Content Caching, Recommendation, and Transmission for Layered Scalable Videos Over Dynamic Cellular Networks: A Dueling Deep Q-Learning Approach

Joint Content Caching, Recommendation, and Transmission for Layered Scalable Videos Over Dynamic Cellular Networks: A Dueling Deep Q-Learning Approach

A Collaborative Computation and Offloading for Compute-Intensive and Latency-Sensitive Dependency-Aware Tasks in Dew-Enabled Vehicular Fog Computing: A Federated Deep Q-Learning Approach

A Collaborative Computation and Offloading for Compute-Intensive and Latency-Sensitive Dependency-Aware Tasks in Dew-Enabled Vehicular Fog Computing: A Federated Deep Q-Learning Approach

Development of improved reinforcement learning smart charging strategy for electric vehicle fleet

Due to its environmental and energy sustainability, electric vehicles (EV) have emerged as the preferred option in the current transportation system. Uncontrolled EV charging, however, can raise consumers; charging costs and overwhelm the grid. Smart charging coordination systems are required to prevent the grid overload caused by charging too many electric vehicles at once. In light of the baseload that is present in the power grid, this research suggests an improved reinforcement learning charging management system. An optimization method, however, requires some knowledge in advance, such as the time the vehicle departs and how much energy it will need when it arrives at the charging station. Therefore, under realistic operating conditions, our improved Reinforcement Learning method with Double Deep Q-learning approach provides an adjustable, scalable, and flexible strategy for an electric car fleet. Our proposed approach provides fair value which solves the overestimation action value problem in deep Q-learning. Then, a number of different charging strategies are compared to the Reinforcement Learning algorithm. The proposed Reinforcement Learning technique minimizes the variance of the overall load by 68 % when compared to an uncontrolled charging strategy.

Deep reinforcement learning based controller for ship navigation

A majority of marine accidents that occur can be attributed to errors in human decisions. Through automation, the occurrence of such incidents can be minimized. Therefore, automation in the marine industry has been receiving increased attention in the recent years. This paper investigates the automation of the path following action of a ship. A deep Q-learning approach is proposed to solve the path-following problem of a ship. This method comes under the broader area of deep reinforcement learning (DRL) and is well suited for such tasks, as it can learn to take optimal decisions through sufficient experience. This algorithm also balances the exploration and the exploitation schemes of an agent operating in an environment. A three-degree-of-freedom (3-DOF) dynamic model is adopted to describe the ship’s motion. The Krisco container ship (KCS) is chosen for this study as it is a benchmark hull that is used in several studies and its hydrodynamic coefficients are readily available for numerical modeling. Numerical simulations for the turning circle and zig-zag maneuver tests are performed to verify the accuracy of the proposed dynamic model. A reinforcement learning (RL) agent is trained to interact with this numerical model to achieve waypoint tracking. Finally, the proposed approach is investigated not only by numerical simulations but also by model experiments using 1:75.5 scaled model.

The Wisdom of the Crowd: Reliable Deep Reinforcement Learning Through Ensembles of Q-Functions.

Reinforcement learning (RL) agents learn by exploring the environment and then exploiting what they have learned. This frees the human trainers from having to know the preferred action or intrinsic value of each encountered state. The cost of this freedom is that RL is slower and more unstable than supervised learning. We explore the possibility that ensemble methods can remedy these shortcomings by investigating a novel technique which harnesses the wisdom of crowds by combining Q-function approximator estimates utilizing a simple combination scheme similar to the supervised learning approach known as bagging. Bagging approaches have not yet found widespread adoption in the RL literature nor has a comprehensive look at its performance been performed. Our results show that the proposed approach improves all three tasks and RL approaches attempted. The primary contribution of this work is a demonstration that the improvement is a direct result of the increased stability of the action portion of the state-action-value function. Subsequent experimentation demonstrates that the stability in learning allows an actor-critic method to find more efficient solutions. Finally we show that this approach can be used to decrease the amount of time necessary to solve problems which require a deep Q-learning (DQN) approach.

Developing a Deep Q-Learning and Neural Network Framework for Trajectory Planning

With the recent expansion in Self-Driving and Autonomy field, every vehicle is occupied with some kind or alter driver assist features in order to compensate driver comfort. Expansion further to fully Autonomy is extremely complicated since it requires planning safe paths in unstable and dynamic environments. Impression learning and other path learning techniques lack generalization and safety assurances. Selecting the model and avoiding obstacles are two difficult issues in the research of autonomous vehicles. Q-learning has evolved into a potent learning framework that can now acquire complicated strategies in high-dimensional contexts to the advent of deep feature representation. A deep Q-learning approach is proposed in this study by using experienced replay and contextual expertise to address these issues. A path planning strategy utilizing deep Q-learning on the network edge node is proposed to enhance the driving performance of autonomous vehicles in terms of energy consumption. When linked vehicles maintain the recommended speed, the suggested approach simulates the trajectory using a proportional-integral-derivative (PID) concept controller. Smooth trajectory and reduced jerk are ensured when employing the PID controller to monitor the terminals. The computational findings demonstrate that, in contrast to traditional techniques, the approach could investigate a path in an unknown situation with small iterations and a higher average payoff. It can also more quickly converge to an ideal strategic plan.

Deep Q-Learning Technique for Offloading Offline/Online Computation in Blockchain-Enabled Green IoT-Edge Scenarios

The number of Internet of Things (IoT)-related innovations has recently increased exponentially, with numerous IoT objects being invented one after the other. Where and how many resources can be transferred to carry out tasks or applications is known as computation offloading. Transferring resource-intensive computational tasks to a different external device in the network, such as a cloud, fog, or edge platform, is the strategy used in the IoT environment. Besides, offloading is one of the key technological enablers of the IoT, as it helps overcome the resource limitations of individual objects. One of the major shortcomings of previous research is the lack of an integrated offloading framework that can operate in an offline/online environment while preserving security. This paper offers a new deep Q-learning approach to address the IoT-edge offloading enabled blockchain problem using the Markov Decision Process (MDP). There is a substantial gap in the secure online/offline offloading systems in terms of security, and no work has been published in this arena thus far. This system can be used online and offline while maintaining privacy and security. The proposed method employs the Post Decision State (PDS) mechanism in online mode. Additionally, we integrate edge/cloud platforms into IoT blockchain-enabled networks to encourage the computational potential of IoT devices. This system can enable safe and secure cloud/edge/IoT offloading by employing blockchain. In this system, the master controller, offloading decision, block size, and processing nodes may be dynamically chosen and changed to reduce device energy consumption and cost. TensorFlow and Cooja’s simulation results demonstrated that the method could dramatically boost system efficiency relative to existing schemes. The findings showed that the method beats four benchmarks in terms of cost by 6.6%, computational overhead by 7.1%, energy use by 7.9%, task failure rate by 6.2%, and latency by 5.5% on average.

A graph neural networks-based deep Q-learning approach for job shop scheduling problems in traffic management

A key problem in traffic management is to schedule the movements of vehicles to reduce unnecessary costs; this issue has arisen in applications such as train schedule management, air-traffic control, and urban traffic management. This problem can be modeled as a job shop scheduling problem (JSSP), which is an important combinatorial optimization problem that is widely applied in real-world scenarios. However, designing good approximation algorithms for JSSPs often requires significant specialized knowledge and trial-and-error. In this paper, we present an end-to-end framework for solving JSSPs by using graph neural networks (GNNs) and deep Q-Learning. This single-policy model is suitable for solving instances that have similar sizes and is trained only by observing reward signals and following feasible rules. The trained model behaves like a constructive heuristic algorithm that incrementally constructs a solution, and each action is determined by the output of a GNN, which captures the current state of the partial solution. We test the proposed approach on JSSP instances with multiple sizes and demonstrate its competitive performance in all cases (after training). Our proposed framework also has the potential to be applied to other JSS subproblems.

Reinforcement Learning for Motion Policies in Mobile Relaying Networks

We consider joint beamforming and relay motion control in mobile relay beamforming networks, operating in a spatio-temporally varying channel environment. A time slotted approach is adopted, where in each slot, the relays implement optimal beamforming and estimate their optimal positions for the next slot. We place the problem of relay motion control in a sequential decision-making framework. We employ Reinforcement Learning (RL) to guide the relay motion, with the goal of maximizing the cumulative Signal-to-Interference+Noise Ratio (SINR) at the destination. First, we present a model based RL approach, which predictively estimates the SINR and accordingly determines the relay motion, based on partial knowledge of the channel model along with channel measurements at the current relay positions. Second, we propose a model-free deep Q-learning approach, which does not rely on channel models. For the deep Q-learning approach, we propose two modified Multilayer Perceptron Neural Networks (MLPs) for approximating the value function Q. The first modification applies a Fourier feature mapping of the state before passing it through the MLP. The second modification constitutes a different neural network architecture that uses sinusoids as activations between layers. Both modifications enable the MLP to better learn the high frequency value function and have a profound effect on convergence speed and SINR performance. Finally, we conduct a comparative analysis of all the presented approaches and provide insights on advantages and drawbacks.

A Data-Driven Waveform Adaptation Method for Mm-Wave Gait Classification at the Edge

The ever-increasing need for real-time 3D perception for autonomy has resulted in development of a new generation of high-resolution perception sensors paired with powerful edge processors and advanced perception algorithms. However, processing of the large data volumes generated by such high-resolution sensors still imposes a great challenge for resource-limited edge devices and systems. In this work, we visit this problem in the context of pedestrian gait analysis using high-resolution mm-Wave sensors with application in autonomous driving. We propose reducing the total computational load of a resource-limited perception system by dynamically controlling the mm-Wave sensor's resolution on the fly. We provide results on three dynamic sensor adaptation approaches including an end-to-end Deep Q-Learning approach that can overcome the resource and performance limitations inherent to conventional static methods.

Deep Q-learning Approach based on CNN and XGBoost for Traffic Signal Control

Traffic signal control is a way for reducing traffic jams in urban areas, and to optimize the flow of vehicles by minimizing the total waiting times. Several intelligent methods have been proposed to control the traffic signal. However, these methods use a less efficient road features vector, which can lead to suboptimal controls. The objective of this paper is to propose a deep reinforcement learning approach as the hybrid model that combines the convolutional neural network with eXtreme Gradient Boosting to traffic light optimization. We first introduce the deep convolutional neural network architecture for the best features extraction from all available traffic data and then integrated the extracted features into the eXtreme Gradient Boosting model to improve the prediction accuracy. In our approach; cross-validation grid search was used for the hyper-parameters tuning process during the training of the eXtreme Gradient Boosting model, which will attempt to optimize the traffic signal control. Our system is coupled to a microscopic agent-based simulator (Simulation of Urban MObility). Simulation results show that the proposed approach improves significantly the average waiting time when compared to other well-known traffic signal control algorithms.

Self-supervised reinforcement learning-based energy management for a hybrid electric vehicle

Reinforcement learning is a new research hotspot in the energy management strategy. At present, some problems in the application of reinforcement learning to energy management control still exist, including sparse reward, convergence speed, generalization ability, etc. This paper proposes a self-supervised reinforcement learning method based on a Deep Q-learning approach for fuel-saving optimization of a plug-in hybrid electric vehicle (PHEV). First, a detailed vehicle powertrain model of the Prius is built. Second, we use the self-supervised learning model to enrich the reward function. The reward function consists of two parts: internal and external rewards. Finally, to prevent the self-supervised model from falling into the “self-good” situation, a reinforcement learning calibration method is proposed. The vehicle exploration method is more effective because of the enrichment of the reward function. Furthermore, following the characteristics of self-supervised learning, we have also constructed a new driving cycle to verify the generalization ability. Results show that our proposed deep reinforcement learning method based on self-supervised and learning calibration realizes faster training convergence and lower fuel consumption than the traditional policy, and its fuel economy can reach approximately the global optimum under our new driving cycle. • The method used in this can enhance the vehicle's ability to explore the environment. • Internal rewards were constructed using self-supervised learning methods. • The algorithm has generalization ability in energy management control strategy. • The reward function is corrected with the reinforcement learning calibration.

Deep Reinforcement Learning-Based Resource Allocation in Cooperative UAV-Assisted Wireless Networks

We consider the downlink of an unmanned aerial vehicle (UAV) assisted cellular network consisting of multiple cooperative UAVs, whose operations are coordinated by a central ground controller using wireless fronthaul links, to serve multiple ground user equipments (UEs). A problem of jointly designing UAVs’ positions, transmit beamforming, as well as UAV-UE association is formulated in the form of mixed integer nonlinear programming (MINLP) to maximize the sum UEs’ achievable rate subject to limited fronthaul capacity constraints. Solving the considered problem is hard owing to its non-convexity and the unavailability of channel state information (CSI) due to the movement of UAVs. To tackle these effects, we propose a novel algorithm comprising of two distinguishing features: (i) exploiting a deep Q-learning approach to tackle the issue of CSI unavailability for determining UAVs’ positions, (ii) developing a difference of convex algorithm (DCA) to efficiently solve for the UAV’s transmit beamforming and UAV-UE association. The proposed algorithm recursively solves the problem of interest until convergence, where each recursion executes two steps. In the first step, the deep Q-learning (DQL) algorithm allows UAVs to learn the overall network state and account for the joint movement of all UAVs to adapt their locations. In the second step, given the determined UAVs’ positions from the DQL algorithm, the DCA iteratively solves a convex approximate subproblem of the original non-convex MINLP problem with the updated parameters, where the problem’s variables are transmit beamforming and UAV-UE association. Numerical results show that our design outperforms the existing algorithms in terms of algorithmic convergence and network performance with a gain of up to 70%.

AIOC[formula omitted]: A deep Q-learning approach to autonomic I/O congestion control in Lustre

In high performance computing systems, I/O congestion is a common problem in large-scale distributed file systems. However, the current implementation mainly requires administrator to manually design low-level implementation and optimization, we proposes an adaptive I/O congestion control framework, named AIOC2, which can not only adaptively tune the I/O congestion control parameters, but also exploit the deep Q-learning method to start the training parameters and optimize the tuning for different types of workloads from the server and the client at the same time. AIOC2 combines the feedback-based dynamic I/O congestion control and deep Q-learning parameter tuning technology to achieve autonomic I/O congestion control, improve system I/O throughput, and thus reduce I/O latency without human interference. Experimental results show that AIOC2 can greatly reduce the impact of I/O congestion on I/O throughput and I/O latency performance in Lustre clusters. Compared to existing Lustre cluster systems, AIOC2 can increase write I/O throughput by 34.82% and decrease I/O latency by 26.17% on average.

Dynamic selective auditory attention detection using RNN and reinforcement learning

The cocktail party phenomenon describes the ability of the human brain to focus auditory attention on a particular stimulus while ignoring other acoustic events. Selective auditory attention detection (SAAD) is an important issue in the development of brain-computer interface systems and cocktail party processors. This paper proposes a new dynamic attention detection system to process the temporal evolution of the input signal. The proposed dynamic SAAD is modeled as a sequential decision-making problem, which is solved by recurrent neural network (RNN) and reinforcement learning methods of Q-learning and deep Q-learning. Among different dynamic learning approaches, the evaluation results show that the deep Q-learning approach with RNN as agent provides the highest classification accuracy (94.2%) with the least detection delay. The proposed SAAD system is advantageous, in the sense that the detection of attention is performed dynamically for the sequential inputs. Also, the system has the potential to be used in scenarios, where the attention of the listener might be switched in time in the presence of various acoustic events.