Q-learning Approach Research Articles

Output feedback fault-tolerant Q-learning for discrete-time linear systems with actuator faults

This paper presents an innovative output feedback fault-tolerant Q-learning algorithm that can be implemented online without relying on explicit system models or fault details. In the face of actuator faults, finding optimal Fault-Tolerant Control (FTC) solutions that can stabilize the faulty system poses significant challenges. The proposed approach is implemented online without the need for system dynamics and actuator fault information. Furthermore, it operates without relying on full state measurements, utilizing only the input-output data of the faulty system. An innovative representation of the output feedback Fault-Tolerant Q-function (FTQF) is established using input-output data. Subsequently, a model-free optimal output feedback FTC policy is obtained from the developed FTQF. Then, a fault-tolerant Q-learning algorithm is formulated to iteratively acquire the optimal FTC policy in real-time, eliminating the necessity for system dynamics and actuator fault information. The proposed algorithm exhibits immunity to excitation noise bias, even without considering a discounting factor. Furthermore, the proposed Q-learning approach is proven to be effective in stabilizing faulty closed-loop system. Finally, the proposed algorithm's efficiency is validated through numerical simulations of F-16 autopilot aircraft dynamics.

A Vision-based Robotic Navigation Method Using an Evolutionary and Fuzzy Q-Learning Approach

The paper presents a Fuzzy Q-Learning (FQL) and optical flow based autonomous navigation approach. The FQL method takes decisions in an unknown environment and without mapping, using motion information and through a reinforcement signal into an evolutionary algorithm. The reinforcement signal is calculated by estimating the optical flow densities in areas of the camera to determine whether they are “dense” or “thin” which has a relationship with the proximity of objects. The results obtained show that the present approach improves the rate of learning compared with a method with a simple reward system and without the evolutionary component. The proposed system was implemented in a virtual robotics system using the CoppeliaSim software and in communication with Python.

A scalable Deep Q-Learning approach for hot stamping process under dynamic control environment

Despite the significant advancements in Artificial Intelligence and its applications, traditional control techniques still dominate industrial and manufacturing processes. These techniques often rely on static setups and default values, which make them resistant to environmental changes. However, in dynamic and uncertain contexts, real-time process adaptation offers significant advantages in decision-making and control. In this research, we present a novel approach utilising Deep Reinforcement Learning for real-time dynamic parameter adaptation and autonomous control of processes in unpredictable environments. The effectiveness of our proposal is illustrated through the development and training of a Deep Q-Learning control system tailored for optimising a hot stamping process. The control model aims to minimise defect rates and enhance efficiency by reducing forming times during batch productions. Comparative analysis against conventional industry practices, characterised by static and non-adaptive time settings, validates the efficacy of our solution, with improvements of over 20% in the times, depending on the scenario considered. Experimental evaluation in a semi-industrial pilot setting not only demonstrates the adaptability of our approach but also highlights its potential scalability for diverse industrial applications.

A reinforcement learning approach to solving very-short term train rescheduling problem for a single-track rail corridor

Railway operations are regularly affected by incidents such as disturbances and disruptions, which cause temporary operational restrictions to the trains in the network. Compared to real-time disturbances and disruptions, sometimes these incidents can be known at a short notice, e.g., 24–48 h beforehand, which is known as the Very-Short-Term-Planning in British rail operations. This paper presents a novel reinforcement learning based approach for rescheduling train services for in a single-track corridor with bi-directional traffic. As an important subfield of machine learning, reinforcement learning offers an alternate strategy for tackling the NP-hard train (re)scheduling problems and shows its advantages in balancing computational efficiency and solution quality. We propose a Q-learning approach with a tiered rewarding strategy and lightweight train representation in state vectors, which enables more efficient learning and knowledge sharing among homogeneous trains. Compared with an existing reinforcement learning approach, our proposed method can find better quality solutions due to its unique representation of state vectors and a novel tiered rewarding/punishing mechanism, overcoming certain disadvantages in existing approaches. Knowledge reusability is another advantage of the proposed approach, as prior knowledge obtained from training one instance can significantly enhance the performance of another, potentially more challenging, instance on the same corridor with substantially reduced computational time and effort on algorithm development. We also discuss the potential applications of the knowledge reusability feature inherent in reinforcement learning algorithms, which we believe will benefit the entire industry in addressing NP-hard problems through data-driven technologies.

Calibration Method for Relativistic Navigation System Using Parallel Q-Learning Extended Kalman Filter.

For the relativistic navigation system where the position and velocity of the spacecraft are determined through the observation of the relativistic perturbations including stellar aberration and starlight gravitational deflection, a novel parallel Q-learning extended Kalman filter (PQEKF) is presented to implement the measurement bias calibration. The relativistic perturbations are extracted from the inter-star angle measurement achieved with a group of high-accuracy star sensors on the spacecraft. Inter-star angle measurement bias caused by the misalignment of the star sensors is one of the main error sources in the relativistic navigation system. In order to suppress the unfavorable effect of measurement bias on navigation performance, the PQEKF is developed to estimate the position and velocity, together with the calibration parameters, where the Q-learning approach is adopted to fine tune the process noise covariance matrix of the filter automatically. The high performance of the presented method is illustrated via numerical simulations in the scenario of medium Earth orbit (MEO) satellite navigation. The simulation results show that, for the considered MEO satellite and the presented PQEKF algorithm, in the case that the inter-star angle measurement accuracy is about 1 mas, after calibration, the positioning accuracy of the relativistic navigation system is less than 300 m.

Heat Conduction Control Using Deep Q-Learning Approach with Physics-Informed Neural Networks

As modern systems become more complex, their control strategy no longer relies only on measurement data from probes; it also requires information from mathematical models for non-measurable places. On the other hand, those mathematical models can lead to unbearable computation times due to their own complexity, making the control process non-viable. To overcome this problem, it is possible to implement any kind of surrogate model that enables the computation of such estimates within an acceptable time frame, which allows for making decisions. Using a Physics-Informed Neural Network as a surrogate model, it is possible to compute the temperature distribution at each time step, replacing the need for running direct numerical simulations. This approach enables the use of a Deep Reinforcement Learning algorithm to train a control strategy. On this work, we considered a one-dimensional heat conduction problem, in which temperature distribution feeds a control system. Such control system has the objective of reacing and maintaining constant temperature value at a specific location of the 1D problem by activating a heat source; the desired location somehow cannot be directly measured so, the PINN approach allows to estimate its temperature with a minimum computational workload. With this approach, the control training becomes much faster without the need of performing numerical simulations or laboratory measurements.

Optimal Electric Vehicle Battery Management Using Q-learning for Sustainability

This paper presents a comprehensive study on the optimization of electric vehicle (EV) battery management using Q-learning, a powerful reinforcement learning technique. As the demand for electric vehicles continues to grow, there is an increasing need for efficient battery-management strategies to extend battery life, enhance performance, and minimize operating costs. The primary objective of this research is to develop and assess a Q-learning-based approach to address the intricate challenges associated with EV battery management. This paper starts by elucidating the key challenges inherent in EV battery management and discusses the potential advantages of incorporating Q-learning into the optimization process. Leveraging Q-learning’s capacity to make dynamic decisions based on past experiences, we introduce a framework that considers state-of-charge, state-of-health, charging infrastructure, and driving patterns as critical state variables. The methodology is detailed, encompassing the selection of state, action, reward, and policy, with the training process informed by real-world data. Our experimental results underscore the efficacy of the Q-learning approach in optimizing battery management. Through the utilization of Q-learning, we achieve substantial enhancements in battery performance, energy efficiency, and overall EV sustainability. A comparative analysis with traditional battery-management strategies is presented to highlight the superior performance of our approach. A comparative analysis with traditional battery-management strategies is presented to highlight the superior performance of our approach, demonstrating compelling results. Our Q-learning-based method achieves a significant 15% improvement in energy efficiency compared to conventional methods, translating into substantial savings in operational costs and reduced environmental impact. Moreover, we observe a remarkable 20% increase in battery lifespan, showcasing the effectiveness of our approach in enhancing long-term sustainability and user satisfaction. This paper significantly enriches the body of knowledge on EV battery management by introducing an innovative, data-driven approach. It provides a comprehensive comparative analysis and applies novel methodologies for practical implementation. The implications of this research extend beyond the academic sphere to practical applications, fostering the broader adoption of electric vehicles and contributing to a reduction in environmental impact while enhancing user satisfaction.

A Q-Learning Approach for Optimizing the Impact of Musical Education Using Virtual Reality and Social Robots

A Q-Learning Approach for Optimizing the Impact of Musical Education Using Virtual Reality and Social Robots

MAM-STM: A software for autonomous control of single moieties towards specific surface positions

In this publication we introduce MAM-STM, a software to autonomously manipulate arbitrary moieties towards specific positions on a metal surface utilizing the tip of a scanning tunneling microscope (STM). Finding the optimal manipulation parameters for a specific moiety is challenging and time consuming, even for human experts. MAM-STM combines autonomous data acquisition with a sophisticated Q-learning implementation to determine the optimal bias voltage, the z-approach distance, and the tip position relative to the moiety. This then allows to arrange single molecules and atoms at will. In this work, we provide a tutorial based on a simulated response to offer a comprehensive explanation on how to use and customize MAM-STM. Additionally, we assess the performance of the machine learning algorithm by benchmarking it within a simulated stochastic environment. PROGRAM SUMMARYProgram title: MAM-STMCPC Library link to program files: https://doi.org/10.17632/gtf3bt4v47.1Developer's repository link: https://gitlab.tugraz.at/software_public/mam_stm.gitLicensing provisions: GNU General Public License 3 (GPL)Programming language: Python 3Nature of problem: Achieving precise control over the arrangement of individual molecules on surfaces is essential for advancing nanofabrication and understanding molecular interaction processes. While self-assembly offers a method for forming nanostructures, achieving arbitrary arrangements of moieties remains difficult. Current approaches, such as scanning probe microscopy (SPM), require extensive manual intervention and precise control is difficult to achieve consistently due to the stochastic nature of quantum mechanical systems at the nanoscale. Thus, learning to manipulate several moieties in order to build even relatively small structures is challenging and time consuming and the automation through conventional expert systems is hindered by the lack of prior knowledge about the surface-moiety interaction processes.Solution method: This scenario is ideal for machine learning algorithms, like reinforcement learning (RL), which do not require an underlying model but are able to autonomously learn the optimal manipulation parameters by performing manipulations directly at the machine. Introducing MAM-STM, which stands for Molecular and Atomic Manipulation via Scanning Tunneling Microscopy. MAM-STM allows to control molecules and atoms by learning the manipulation parameters for either vertical or lateral manipulations. However, the vast number of manipulation parameter combinations and the inefficient learning procedure of RL agents exhibit several challenges. MAM-STM overcomes these challenges with an autonomous masking routine that eliminates manipulation parameters that induce structural changes to the moiety or lift it off the surface. Additionally, a sophisticated Q-learning approach is developed that speeds up the learning procedure, enabling molecular manipulations within one day of training.

Minimizing Negative Side Effects in Cooperative Multi-Agent Systems using Distributed Coordination

Autonomous agents operating in real-world environments frequently encounter undesirable outcomes or negative side effects (NSEs) when working collaboratively alongside other agents. Even when agents can execute their primary task optimally when operating in isolation, their training may not account for potential negative interactions that arise in the presence of other agents. We frame the challenge of minimizing NSEs as a lexicographic decentralized Markov decision process in which we assume independence of rewards and transitions with respect to the primary assigned tasks, but recognize that addressing negative side effects creates a form of dependence among the agents. We present a lexicographic Q-learning approach to mitigate the NSEs using human feedback models while maintaining near-optimality with respect to the assigned tasks---up to some given slack. Our empirical evaluation across two domains demonstrates that our collaborative approach effectively mitigates NSEs, outperforming non-collaborative methods.

Social-Aware Peer Selection for Energy Efficient D2D Communications in UAV-Assisted Networks: A Q-Learning Approach

Social-Aware Peer Selection for Energy Efficient D2D Communications in UAV-Assisted Networks: A Q-Learning Approach

A Subgraph-Based Hierarchical Q-Learning Approach to Optimal Resource Scheduling for Complex Industrial Networks

A Subgraph-Based Hierarchical Q-Learning Approach to Optimal Resource Scheduling for Complex Industrial Networks

Multiagent Q-Learning Approach for the Recharging Scheduling of Electric Automated Guided Vehicles in Container Terminals

In recent years, advancements in battery technology have led to increased adoption of electric automated guided vehicles in container terminals. Given how critical these vehicles are to terminal operations, this trend requires efficient recharging scheduling for automated guided vehicles, and the main challenges arise from limited charging station capacity and tight vehicle schedules. Motivated by the dynamic nature of the problem, the recharging scheduling problem for an entire vehicle fleet given capacitated stations is formulated as a Markov decision process model. Then, it is solved using a multiagent Q-learning (MAQL) approach to produce a recharging schedule that minimizes the delay of jobs. Numerical experiments show that under a stochastic environment in terms of vehicle travel time, MAQL enables the exploration of better scheduling by coordinating across the entire vehicle fleet and charging facilities and outperforms various benchmark approaches, with an additional improvement of 18.8% on average over the best rule-based heuristic and 5.4% over the predetermined approach. Funding: This work was supported by the National Natural Science Foundation of China [Grant 72101203], the Shaanxi Provincial Key R&D Program, China [Grant 2022KW-02], and the Singapore Maritime Institute [Grant SMI-2017-SP-002]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.0113 .

Model-Free Q-Learning for the Tracking Problem of Linear Discrete-Time Systems.

In this article, a model-free Q-learning algorithm is proposed to solve the tracking problem of linear discrete-time systems with completely unknown system dynamics. To eliminate tracking errors, a performance index of the Q-learning approach is formulated, which can transform the tracking problem into a regulation one. Compared with the existing adaptive dynamic programming (ADP) methods and Q-learning approaches, the proposed performance index adds a product term composed of a gain matrix and the reference tracking trajectory to the control input quadratic form. In addition, without requiring any prior knowledge of the dynamics of the original controlled system and command generator, the control policy obtained by the proposed approach can be deduced by an iterative technique relying on the online information of the system state, the control input, and the reference tracking trajectory. In each iteration of the proposed method, the desired control input can be updated by the iterative criteria derived from a precondition of the controlled system and the reference tracking trajectory, which ensures that the obtained control policy can eliminate tracking errors in theory. Moreover, to effectively use less data to obtain the optimal control policy, the off-policy approach is introduced into the proposed algorithm. Finally, the effectiveness of the proposed algorithm is verified by a numerical simulation.

Predictive mobility and cost-aware flow placement in SDN-based IoT networks: a Q-learning approach

Software-Defined Networking (SDN) has emerged as an innovative networking method that offers effective management and remarkable flexibility. However, current SDN-based solutions primarily focus on static networks or concentrate on backbone networks, where network dynamics have minimal impact. The existing methods for placing flow entries in Software-Defined Networking (SDN)-based Internet of Things (IoT) systems exhibit shortcomings in accurately predicting outcomes and efficiently reducing table misses and associated performance metrics. This research introduces a new approach, specifically the mobility-aware adaptive flow entry placement scheme for SDN-based Internet of Things (IoT) environments, to address the mobility aspect of networks. The proposed scheme utilizes the Q-learning algorithm to predict the next possible location of end devices, while the cost-sensitive AdaBoost algorithm is employed to select heavy and active flows. As a result, efficient flow rules for incoming flows can be dynamically implemented without controller intervention. Extensive computer simulations demonstrate that this approach significantly enhances match probability and prediction accuracy while concurrently reducing the number of table misses and resource expenditure compared to existing schemes.

A dynamic reward-enhanced Q-learning approach for efficient path planning and obstacle avoidance in mobile robotics

PurposeThe purpose of the paper is to propose and demonstrate a novel approach for addressing the challenges of path planning and obstacle avoidance in the context of mobile robots (MR). The specific objectives and purposes outlined in the paper include: introducing a new methodology that combines Q-learning with dynamic reward to improve the efficiency of path planning and obstacle avoidance. Enhancing the navigation of MR through unfamiliar environments by reducing blind exploration and accelerating the convergence to optimal solutions and demonstrating through simulation results that the proposed method, dynamic reward-enhanced Q-learning (DRQL), outperforms existing approaches in terms of achieving convergence to an optimal action strategy more efficiently, requiring less time and improving path exploration with fewer steps and higher average rewards.Design/methodology/approachThe design adopted in this paper to achieve its purposes involves the following key components: (1) Combination of Q-learning and dynamic reward: the paper’s design integrates Q-learning, a popular reinforcement learning technique, with dynamic reward mechanisms. This combination forms the foundation of the approach. Q-learning is used to learn and update the robot’s action-value function, while dynamic rewards are introduced to guide the robot’s actions effectively. (2) Data accumulation during navigation: when a MR navigates through an unfamiliar environment, it accumulates experience data. This data collection is a crucial part of the design, as it enables the robot to learn from its interactions with the environment. (3) Dynamic reward integration: dynamic reward mechanisms are integrated into the Q-learning process. These mechanisms provide feedback to the robot based on its actions, guiding it to make decisions that lead to better outcomes. Dynamic rewards help reduce blind exploration, which can be time-consuming and inefficient and promote faster convergence to optimal solutions. (4) Simulation-based evaluation: to assess the effectiveness of the proposed approach, the design includes a simulation-based evaluation. This evaluation uses simulated environments and scenarios to test the performance of the DRQL method. (5) Performance metrics: the design incorporates performance metrics to measure the success of the approach. These metrics likely include measures of convergence speed, exploration efficiency, the number of steps taken and the average rewards obtained during the robot’s navigation.FindingsThe findings of the paper can be summarized as follows: (1) Efficient path planning and obstacle avoidance: the paper’s proposed approach, DRQL, leads to more efficient path planning and obstacle avoidance for MR. This is achieved through the combination of Q-learning and dynamic reward mechanisms, which guide the robot’s actions effectively. (2) Faster convergence to optimal solutions: DRQL accelerates the convergence of the MR to optimal action strategies. Dynamic rewards help reduce the need for blind exploration, which typically consumes time and this results in a quicker attainment of optimal solutions. (3) Reduced exploration time: the integration of dynamic reward mechanisms significantly reduces the time required for exploration during navigation. This reduction in exploration time contributes to more efficient and quicker path planning. (4) Improved path exploration: the results from the simulations indicate that the DRQL method leads to improved path exploration in unknown environments. The robot takes fewer steps to reach its destination, which is a crucial indicator of efficiency. (5) Higher average rewards: the paper’s findings reveal that MR using DRQL receive higher average rewards during their navigation. This suggests that the proposed approach results in better decision-making and more successful navigation.Originality/valueThe paper’s originality stems from its unique combination of Q-learning and dynamic rewards, its focus on efficiency and speed in MR navigation and its ability to enhance path exploration and average rewards. These original contributions have the potential to advance the field of mobile robotics by addressing critical challenges in path planning and obstacle avoidance.

A deep Q-learning approach to optimize ordering and dynamic pricing decisions in the presence of strategic customers

In this paper, we present an optimization method to analyze the simultaneous decisions on dynamic pricing and ordering quantities for seasonal products, by a retailer in monopolistic condition. Customers are assumed to be strategic and may postpone their purchase to get a lower price in future. The problem has been investigated in the context of multiple substitute products. We have developed a model based on deep neural networks to estimate customers’ demand. The problem is complex and cannot be solved using classical optimization methods. Therefore, we have developed a reinforcement learning algorithm called deep Q-learning algorithm (DQL) to solve the problem. The proposed algorithm is a combination of a Q-learning algorithm and two deep neural networks for the primary and discount sales periods, which uses the neural network to estimate the Q-values in a large space of states and actions. The performances of the demand model and the proposed optimization algorithm have been tested using a real-world dataset taken from the clothing industry. The results of our experiments demonstrate that the proposed demand model performs better than a fully connected neural network-based model and a latent class model tested in this paper. Furthermore, the performance of the DQL algorithm is significantly superior to those of two simulated annealing and genetic algorithms. In addition, the results of a comparison between the DQL algorithm and another reinforcement learning algorithm called State-Action-Reward-State-Action (SARSA) indicate that the proposed algorithm results in higher revenues and takes less time to converge. Consequently, the proposed algorithm has a high potential for solving such a large scale integrated pricing and ordering optimization problem.

Influence Maximization in Dynamic Networks Using Reinforcement Learning

Influence maximization (IM) has been widely studied in recent decades, aiming to maximize the spread of influence over networks. Despite many works for static networks, fewer research studies have been dedicated to the IM problem for dynamic networks, which creates many challenges. An IM method for such an environment, should consider its dynamics and perform well under different network structures. To fulfill this objective, more computations are required. Hence, an IM approach should be efficient enough to be applicable for the ever-changing structure of a network. In this research, an IM method for dynamic networks has been proposed which uses a deep Q-learning (DQL) approach. To learn dynamic features from the network and retain previously learned information, incremental and transfer learning methods have been applied. Experiments substantiate the good performance of the DQL methods and their superiority over compared methods on larger sizes of tested synthetic and real-world networks. These experiments illustrate better performance for incremental and transfer learning methods on real-world networks.

Joint Content Caching, Recommendation, and Transmission for Layered Scalable Videos Over Dynamic Cellular Networks: A Dueling Deep Q-Learning Approach

Joint Content Caching, Recommendation, and Transmission for Layered Scalable Videos Over Dynamic Cellular Networks: A Dueling Deep Q-Learning Approach

Spectrum Efficient Mode Selection and Resource Allocation Optimization for D2D Communication in HetNet: A Multi-Agent Q-Learning Approach

Spectrum Efficient Mode Selection and Resource Allocation Optimization for D2D Communication in HetNet: A Multi-Agent Q-Learning Approach