Q-learning Approach Research Articles

An Enhanced Anti-Phishing Technique for Social Media Users: A Multilayer Q-Learning Approach

An Enhanced Anti-Phishing Technique for Social Media Users: A Multilayer Q-Learning Approach

Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care.

Medical treatments often involve a sequence of decisions, each informed by previous outcomes. This process closely aligns with reinforcement learning (RL), a framework for optimizing sequential decisions to maximize cumulative rewards under unknown dynamics. While RL shows promise for creating data-driven treatment plans, its application in medical contexts is challenging due to the frequent need to use sparse rewards, primarily defined based on mortality outcomes. This sparsity can reduce the stability of offline estimates, posing a significant hurdle in fully utilizing RL for medical decision-making. We introduce a deep Q-learning approach to obtain more reliable critical care policies by integrating relevant but noisy frequently measured biomarker signals into the reward specification without compromising the optimization of the main outcome. Our method prunes the action space based on all available rewards before training a final model on the sparse main reward. This approach minimizes potential distortions of the main objective while extracting valuable information from intermediate signals to guide learning. We evaluate our method in off-policy and offline settings using simulated environments and real health records from intensive care units. Our empirical results demonstrate that our method outperforms common offline RL methods such as conservative Q-learning and batch-constrained deep Q-learning. By disentangling sparse rewards and frequently measured reward proxies through action pruning, our work represents a step towards developing reliable policies that effectively harness the wealth of available information in data-intensive critical care environments.

Optimal Deception Asset Deployment in Cybersecurity: A Nash Q-Learning Approach in Multi-Agent Stochastic Games

In the face of an increasingly intricate network structure and a multitude of security threats, cyber deception defenders often employ deception assets to safeguard critical real assets. However, when it comes to the intranet lateral movement attackers in the cyber kill chain, the deployment of deception assets confronts the challenges of lack of dynamics, inability to make real-time decisions, and not considering the dynamic change of an attacker’s strategy. To address these issues, this study introduces a novel maze pathfinding model tailored to the lateral movement context, in which we try to find out the attacker’s location to deploy deception assets accurately for interception. The attack–defense process is modeled as a multi-agent stochastic game, by comparing it with random action policy and Minimax-Q algorithm, we choose Nash Q-learning to solve the deception asset’s deployment strategy to achieve the optimal solution effect. Extensive simulation tests reveal that our proposed model exhibits good convergence properties. Moreover, the average defense success rate surpasses 70%, attesting to the model’s efficacy.

A Q-learning approach to the continuous control problem of robot inverted pendulum balancing

This study evaluates the application of a discrete action space reinforcement learning method (Q-learning) to the continuous control problem of robot inverted pendulum balancing. To speed up the learning process and to overcome technical difficulties related to the direct learning on the real robotic system, the learning phase is performed in simulation environment. A mathematical model of the system dynamics is implemented, deduced by curve fitting on data acquired from the real system. The proposed approach demonstrated feasible, featuring its application on a real world robot that learned to balance an inverted pendulum. This study also reinforces and demonstrates the importance of an accurate representation of the physical world in simulation to achieve a more efficient implementation of reinforcement learning algorithms in real world, even when using a discrete action space algorithm to control a continuous action.

A Collaborative Computation and Offloading for Compute-Intensive and Latency-Sensitive Dependency-Aware Tasks in Dew-Enabled Vehicular Fog Computing: A Federated Deep Q-Learning Approach

A Collaborative Computation and Offloading for Compute-Intensive and Latency-Sensitive Dependency-Aware Tasks in Dew-Enabled Vehicular Fog Computing: A Federated Deep Q-Learning Approach

Energy Efficient CH Selection Scheme Based on ABC and Q-Learning Approaches for IoUT Applications

Nowadays, the Internet of Underwater Things (IoUT) provides many marine 5G applications. However, it has some issues with energy efficiency and network lifetime. The network clustering approach is efficient for optimizing energy consumption, especially for underwater acoustic communications. Recently, many algorithms have been developed related to clustering-based underwater communications for energy efficiency. However, these algorithms have drawbacks when considered for heterogeneous IoUT applications. Clustering efficiency in heterogeneous IoUT is influenced by the uniform distribution of cluster heads (CHs). As a result, conventional schemes are inefficient when CHs are arranged in large and dense nodes since they are unable to optimize the right number of CHs. Consequently, the clustering approach cannot improve the IoUT network, and many underwater nodes will rapidly consume their energies and be exhausted because of the large number of clusters. In this paper, we developed an efficient clustering scheme to effectively select the best CHs based on artificial bee colony (ABC) and Q-learning optimization approaches. The proposed scheme enables an effective selection of the CHs based on four factors, the residual energy level, the depth and the distance from the base station, and the signal quality. We first evaluate the most suitable swarm algorithms and their impact on improving the CH selection mechanism. The evaluated algorithms are generic algorithm (GA), particle swarm optimization (PSO), ant colony optimization (ACO), and ABC. Then, the ABC algorithm process is improved by using the Q-learning approach to improve the process of ABC and its fitness function to optimize the CH selection. We observed from the simulation performance result that an improved ABC-QL scheme enables efficient selection of the best CHs to increase the network lifetime and reduce average energy consumption by 40% compared to the conventional ABC.

AI-Based Q-Learning Approach for Performance Optimization in MIMO-NOMA Wireless Communication Systems

In this paper, we investigate the performance enhancement of Multiple Input, Multiple Output, and Non-Orthogonal Multiple Access (MIMO-NOMA) wireless communication systems using an Artificial Intelligence (AI) based Q-Learning reinforcement learning approach. The primary challenge addressed is the optimization of power allocation in a MIMO-NOMA system, a complex task given the non-convex nature of the problem. Our proposed Q-Learning approach adaptively adjusts power allocation strategy for proximal and distant users, optimizing the trade-off between various conflicting metrics and significantly improving the system’s performance. Compared to traditional power allocation strategies, our approach showed superior performance across three principal parameters: spectral efficiency, achievable sum rate, and energy efficiency. Specifically, our methodology achieved approximately a 140% increase in the achievable sum rate and about 93% improvement in energy efficiency at a transmitted power of 20 dB while also enhancing spectral efficiency by approximately 88.6% at 30 dB transmitted Power. These results underscore the potential of reinforcement learning techniques, particularly Q-Learning, as practical solutions for complex optimization problems in wireless communication systems. Future research may investigate the inclusion of enhanced channel simulations and network limitations into the machine learning framework to assess the feasibility and resilience of such intelligent approaches.

Robust and efficient task scheduling for robotics applications with reinforcement learning

Effective task scheduling can significantly impact on performance, productivity, and profitability in many real-world settings, such as production lines, logistics, and transportation systems. Traditional approaches to task scheduling rely on heuristics or simple rule-based methods. However, with the emergence of machine learning and artificial intelligence, there is growing interest in using these methods to optimize task scheduling. In particular, reinforcement learning is a promising task scheduling approach, because it can learn from experience and adapt to changing conditions. One step often missed or neglected is choosing optimal algorithm parameters and different ways the environment could be implemented. The study analyzes the performance possibilities of task scheduling using reinforcement learning. The deep analysis allows to select highly efficient environment models and Q-learning parameters. Moreover, automatic selection based on optimization algorithms has been proposed. Regardless of the selected optimal parameters, the resilience to environmental changes seems poor. The deducted analysis motivated the Authors to develop a novel Hybrid Q-learning approach. It allows to provide superior efficiency regardless of the environmental parameters.

Reinforcement Learning Approach for Adaptive C-V2X Resource Management

The modulation coding scheme (MCS) index is the essential configuration parameter in cellular vehicle-to-everything (C-V2X) communication. As referenced by the 3rd Generation Partnership Project (3GPP), the MCS index will dictate the transport block size (TBS) index, which will affect the size of transport blocks and the number of physical resource blocks. These numbers are crucial in the C-V2X resource management since it is also bound to the transmission power used in the system. To the authors’ knowledge, this particular area of research has not been previously investigated. Ultimately, this research establishes the fundamental principles for future studies seeking to use the MCS adaptability in many contexts. In this work, we proposed the application of the reinforcement learning (RL) algorithm, as we used the Q-learning approach to adaptively change the MCS index according to the current environmental states. The simulation results showed that our proposed RL approach outperformed the static MCS index and was able to attain stability in a short number of events.

Adaptive multi-layer reward based Q-learning for source tracing in diffusive molecular communications environment with obstacles

Benefiting by the fast development of nanotechnology, molecular communication (MC) has received great attention in recent years. In many potential applications of MC, such as drug delivery and pollution prevention, it is essential to locate or trace the target. In this paper, we consider a 3D diffusive MC environment consisting of several obstacles, a molecule-releasing source (RS) and a mobile molecule sensor (MS) which aims to find the RS within a time constraint. The problem is reformulated using Markov Decision Process (MDP) and an adaptive multi-layer reward based Q-Learning (AMR-Q Learning) approach is proposed. By exploiting information from the number of received molecules and adaptively setting multi-layer rewards, MS with AMR-Q Learning can find the RS efficiently, unlike the gradient based method which is usually trapped in locally optimal points. Numerical results demonstrate that the proposed AMR-Q Learning approach outperforms existing path-planning schemes with significantly reduced training overhead.

Joint impacts of ambient temperature, path flexibility, and hybrid fleet on energy-conserving supply chain network

The increasing urban logistics transportation activities hugely impact the economy and environment. This paper investigates the combined impact of ambient temperature, path flexibility, and hybrid fleet on the economy and environment in urban logistics. The proposed problem is a variant of the location-routing problem (LRP) named dual-mode energy-conserving LRP (DMECLRP) that handles the combination of three types of decisions: the location of depots, the design of the distribution routes, and the location of charging stations (CSs) and battery swapping stations (BSSs). The objective is to minimize the total cost, including the daily fixed costs of operating depots, the cost of renting vehicles and depreciation, driver compensation, and the routing costs, where the latter can be defined concerning the cost of the consumed energy and CO2 emissions. Due to the NP-hardness of the problem, this paper presents a Q-learning-based hyper-heuristic (QLHH) algorithm to address the DMECLRP. The QLHH employs a Q-learning approach to select appropriate heuristics through its search process and simulated annealing to determine the acceptance of solutions. Results show that the proposed algorithm is effective, providing competitive results for LRP benchmark instances within reasonable computing time, and the proposed model can effectively reduce logistics costs, energy consumption, and CO2 emissions. Extensive analyses are carried out to empirically assess the effect of ambient temperature, path flexibility, and hybrid fleet on key performance indicators, including energy consumption, carbon emissions, and operational costs. Several managerial insights are provided.

Model-based versus model-free feeding control and water-quality monitoring for fish-growth tracking in aquaculture systems

This paper proposes model-based and model-free control approaches to monitor the feeding rate and water quality for fish-growth tracking in aquaculture systems. The representative fish-growth model is revisited, which describes the total biomass change by incorporating the fish population density and mortality. Due to the challenging task of measuring the total fish biomass and population data, the new dynamic population model is validated with individual fish-growth data for tracking control. Ammonia exposure is a significant challenge in the fish-population growth tracking problem, affecting fish health and survival. To address this challenge, traditional and optimal controllers are first designed to track the weight reference within suboptimal temperature and dissolved oxygen (DO) profiles under various un-ionized ammonia (UIA) exposure levels by manipulating relative feeding. Then, a Q-learning approach is proposed to learn an optimal feeding-control policy from simulated data on fish-growth weight trajectories while managing ammonia effects. The proposed Q-learning feeding control prevents fish mortality and achieves good tracking errors for fish weight under UIA levels. However, it maintains a relative food consumption that potentially underfeeds fish. Finally, an optimal predictive algorithm that includes the temperature, DO, and UIA is proposed to optimize the feeding and water quality of the dynamic fish-population growth process, indicating that fish mortality is decreased and food consumption is reduced in all cases of UIA exposure.

Hierarchical Q-learning-enabled neutrosophic AHP scheme in candidate relay set size adaption in vehicular networks

Cooperative candidate relay sets (CRS) in opportunistic broadcast paradigms can significantly improve the performance efficiency of traditional broadcast schemes by simultaneously scheduling multiple vehicles in the data packet transmission process. Nonetheless, multiple relay synchronization may cause enormous data redundancy and high delays in data delivery, resulting in a high cooperative transmission cost. High vehicle velocities and vehicular topology restrictions, including intersections, exacerbate the cooperative transmission cost. Motivated by a trade-off between delay and delivery ratio via reducing duplicate data packets and redundant relays, we address this gap in this paper. To this end, we propose an adaptive candidate relay set (ACRS) size scheme based on the Neutrosophic Set Analytic Hierarchy Process (NS-AHP) technique assisted by a Q-learning algorithm. The NS-AHP pair-wise comparison matrix includes; redundancy, cooperative-based efficiency, and relay set stability as the decision-making attributes. Meanwhile, we use a hierarchical Q-learning approach based on the time-varying vehicular environment to adjust the relative preference of the attributes and alternatives (i.e., various CRS sizes) over each other. To optimally learn the relative preference values of the attributes and alternatives over each other, reward functions are set dynamically at both levels. Comprehensive simulations via NS-2 and SUMO are carried out to assess the efficacy of our ACRS scheme compared to three commonly used baseline techniques of fixed size, threshold-based, and probability-based techniques. Overall, the average yields obtained from the experimental outcomes in the urban and highway scenarios show that the ACRS scheme, on average, improves the packet delivery ratio by 3.77%, 5.36%, and 1.35%; reduces the single-hop delay by 34.78%, 42.085%, and 19.39%; reduces the end-to-end delay by 34.76%, 42.72%, and 21.35%; improves the transmission throughput by 34.24%, 41.54%, and 19.305%; and reduces the redundancy ratio by 43.69%, 50.66%, and 24.32% compared to the fixed size, threshold-based, and probability-based techniques, respectively.

Reinforcement Learning Based Efficient Power Control and Spectrum Utilization for D2D Communication in 5G Network

There are billions of inter-connected devices by the help of Internet-of-Things (IoT) that have been used in a number of applications such as for wearable devices, e-healthcare, agriculture, transportation, etc. Interconnection of devices establishes a direct link and easily shares the information by utilizing the spectrum of cellular users to enhance the spectral efficiency with low power consumption in an underlaid Device-to-Device (D2D) communication. Due to reuse of the spectrum of cellular devices by D2D users causes severe interference between them which may impact on the network performance. Therefore, we proposed a Q-Learning based low power selection scheme with the help of multi-agent reinforcement learning to detract the interference that helps to increase the capacity of the D2D network. For the maximization of capacity, the updated reward function has been reformulated with the help of a stochastic policy environment. With the help of a stochastic approach, we figure out the proposed optimal low power consumption techniques which ensures the quality of service (QoS) standards of the cellular devices and D2D users for D2D communication in 5G Networks and increase the utilization of resources. Numerical results confirm that the proposed scheme improves the spectral efficiency and sum rate as compared to Q-Learning approach by 14% and 12.65%.

Multi-UAV computing enabling efficient clustering-based IoT for energy reduction and data transmission

Internet of Things (IoT) technologies increasingly integrate unmanned aerial vehicles (UAVs). IoT devices that are becoming more networked produce massive data. The process and memory of this enormous volume of data at local nodes, particularly when utilizing artificial intelligence (AI) algorithms to collect and utilize useful information, have been declared vital issues. In this paper, we introduce UAV computing to solve greater energy consumption, delay difficulties using task offload and clustered approaches, and make cloud computing operations accessible to IoT devices. First, we present a clustering technique to group IoT devices for data transmission. After that, we apply the Q-learning approach to accomplish task offloading and allocate the difficult tasks to UAVs that are not yet fully loaded. The sensor readings from the CHs are then collected using UAV path planning. Furthermore, We use a convolutional neural network (CNN) to achieve UAV route planning. In terms of coverage ratio, clustering efficiency, UAV motion, energy consumption, and the number of collected packets, the effectiveness of the current study is finally compared with the existing techniques using UAVs. The results showed that the suggested strategy outperformed the current approaches in terms of coverage ratio, clustering efficiency, UAV motion, energy consumption, and the number of collected packets. Additionally, the proposed technique consumed less energy due to CNN-based route planning and dynamic positioning, which reduced UAV transmits power. Overall, the study concluded that the suggested approach is effective for improving energy-efficient and responsive data transmission in crises.

Nonparametric Bayesian Q-learning for optimization of dynamic treatment regimes in the presence of partial compliance.

Existing methods for estimation of dynamic treatment regimes are mostly limited to intention-to-treat analyses-which estimate the effect of randomization to a particular treatment regime without considering the compliance behavior of patients. In this article, we propose a novel nonparametric Bayesian Q-learning approach to construct optimal sequential treatment regimes that adjust for partial compliance. We consider the popular potential compliance framework, where some potential compliances are latent and need to be imputed. The key challenge is learning the joint distribution of the potential compliances, which we accomplish using a Dirichlet process mixture model. Our approach provides two kinds of treatment regimes: (1) conditional regimes that depend on the potential compliance values; and (2) marginal regimes where the potential compliances are marginalized. Extensive simulation studies highlight the usefulness of our method compared to intention-to-treat analyses. We apply our method to the Adaptive Treatment for Alcohol and Cocaine Dependence (ENGAGE) study , where the goal is to construct optimal treatment regimes to engage patients in therapy.

Unconstrained feedback controller design using Q-learning from noisy process data

This paper develops a novel model-free Q-learning based approach to estimate linear, unconstrained feedback controllers from noisy process data. The proposed method is based on an extension of an available approach developed to estimate the linear quadratic regulator (LQR) for linear systems with full state measurements driven by Gaussian process noise of known covariance. First, we modify the approach to treat the case of an unknown noise covariance. Then, we use the modified approach to estimate a feedback controller for linear systems with both process and measurement noise and only output measurements. We also present a model-based maximum likelihood estimation (MLE) approach to determine a linear dynamic model and noise covariances from data, which is used to construct a regulator and state estimator for comparisons in simulation studies. The performances of the model-free and model-based controller estimation approaches are compared with an example heating, ventilation, and air-conditioning (HVAC) system. We show that the proposed Q-learning approach estimates a reasonably accurate feedback controller from 24 h of noisy data. The controllers estimated using both the model-free and model-based approaches provide similar closed-loop performances with 3.5 and 2.7% losses respectively, compared to a perfect controller that uses the true dynamic model and noise covariances of the HVAC system. Finally, we give future work directions for the model-free controller design approaches by discussing some remaining advantages of the model-based approaches.

Robust flipping stabilization of Boolean networks: A Q-learning approach

Robust flipping stabilization of Boolean networks: A Q-learning approach

Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents

Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.

A Q-learning approach to generating behavior of emotional persuasion with adaptive time belief in decision-making of agent-based negotiation

There is an increasing research interest in agent-based automated negotiation. However, how to design an agent that can adaptively compete with its human counterpart is still in its early stage. This research makes efforts toward this end by modeling two essential elements of human-led negotiation: emotion and timing, to adapt to negotiation dynamics. In multi-issue-multi-round bilateral negotiation, the Web-Fechner law is applied to model an agent’s emotion generation and the emotion decay law is used to realize a decaying property of emotion. Meanwhile, rather than a fixed time belief, agents are endowed with time beliefs that can vary with selected negotiation tactics and be adjusted by agents’ emotions. A proposal updating algorithm integrating those two elements is consequently proposed based on Q-learning. Then, 135 numerical experiments were conducted with comparative analyses and statistical tests. The results show that the proposed model can significantly increase an individual agent’s utility, the joint utility, and the probability of successful negotiations by an average of 76.14%, 172.22%, and 66.10%, respectively, whereas reducing the utility difference and negotiation rounds by an average of 97.39% and 29.57%, respectively. These results imply that the proposed negotiation model has advantages in improving negotiation utility, efficiency, and outcome fairness.