Partially Observable Markov Decision Process Model Research Articles

Strong Simple Policies for POMDPs

The synthesis problem for partially observable Markov decision processes (POMDPs) is to compute a policy that provably adheres to one or more specifications. Yet, the general problem is undecidable, and policies require full (and thus potentially unbounded) traces of execution history. To provide good approximations of such policies, POMDP agents often employ randomization over action choices. We consider the problem of computing simpler policies for POMDPs, and provide several approaches to still ensure their expressiveness. Key aspects are (1) the combination of an arbitrary number of specifications the policies need to adhere to, (2) a restricted form of randomization, and (3) a light-weight preprocessing of the POMDP model to encode memory. We provide a novel encoding as a mixed-integer linear program as baseline to solve the underlying problems. Our experiments demonstrate that the policies we obtain are more robust, smaller, and easier to implement for an engineer than those obtained from state-of-the-art POMDP solvers.

POMDP-based probabilistic decision making for path planning in wheeled mobile robot

Path Planning in a collaborative mobile robot system has been a research topic for many years. Uncertainty in robot states, actions, and environmental conditions makes finding the optimum path for navigation highly challenging for the robot. To achieve robust behavior for mobile robots in the presence of static and dynamic obstacles, it is pertinent that the robot employs a path-finding mechanism that is based on the probabilistic perception of the uncertainty in various parameters governing its movement. Partially Observable Markov Decision Process (POMDP) is being used by many researchers as a proven methodology for handling uncertainty. The POMDP framework requires manually setting up the state transition matrix, the observation matrix, and the reward values. This paper describes an approach for creating the POMDP model and demonstrates its working by simulating it on two mobile robots destined on a collision course. Selective test cases are run on the two robots with three categories – MDP (POMDP with belief state spread of 1), POMDP with distribution spread of belief state over ten observations, and distribution spread across two observations. Uncertainty in the sensor data is simulated with varying levels of up to 10 %. The results are compared and analyzed. It is demonstrated that when the observation probability spread is increased from 2 to 10, collision reduces from 34 % to 22 %, indicating that the system's robustness increases by 12 % with only a marginal increase of 3.4 % in the computational complexity.

POMDP inference and robust solution via deep reinforcement learning: an application to railway optimal maintenance

Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the unavailability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), typically benefit from the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of Transformers and long short-term memory networks, which constitute model-free RL solutions and work directly on the observation space, with an approach termed the belief-input method, which works on the belief space by exploiting the learned POMDP model for belief inference. We apply these methods to the real-world problem of optimal maintenance planning for railway assets and compare the results with the current real-life policy. We show that the RL policy learned by the belief-input method is able to outperform the real-life policy by yielding significantly reduced life-cycle costs.

Optimal Joint Defense and Monitoring for Networks Security under Uncertainty: A POMDP-Based Approach

The increasing interconnectivity in our infrastructure poses a significant security challenge, with external threats having the potential to penetrate and propagate throughout the network. Bayesian attack graphs have proven to be effective in capturing the propagation of attacks in complex interconnected networks. However, most existing security approaches fail to systematically account for the limitation of resources and uncertainty arising from the complexity of attacks and possible undetected compromises. To address these challenges, this paper proposes a partially observable Markov decision process (POMDP) model for network security under uncertainty. The POMDP model accounts for uncertainty in monitoring and defense processes, as well as the probabilistic attack propagation. This paper develops two security policies based on the optimal stationary defense policy for the underlying POMDP state process (i.e., a network with known compromises): the estimation-based policy that performs the defense actions corresponding to the optimal minimum mean square error state estimation and the distribution-based policy that utilizes the posterior distribution of network compromises to make defense decisions. Optimal monitoring policies are designed to specifically support each of the defense policies, allowing dynamic allocation of monitoring resources to capture network vulnerabilities/compromises. The performance of the proposed policies is examined in terms of robustness, accuracy, and uncertainty using various numerical experiments.

Cancer Screening Benefits Maximization Using Markov Decision Process Models: A Systematic Review

Context: Due to the chronic nature of cancer, screening programs were a set of sequential decisions taken over time. Markov decision process (MDP) and partially observable Markov decision process (POMDP) models were the mathematical tools applied in studies, including sequential decision-making such as screening protocols for medical decision-making. Objectives: The main goal of this study was to investigate optimal policy for cancer screening using MDP and POMDP models. Methods: We performed a review of articles published within July 2000 to November 2022 in PubMed, Web of Science, and Scopus databases. The stopping age, the type of optimal strategy, the benefits of the optimal policy, and the relationship between age and risk threshold were extracted. Studies that did not use MDPs and POMDPs as the mathematical maximization models in cancer screening, review articles, editorials or commentaries, non-English articles, and those that did not focus on optimization were excluded. Results: From 532 articles, 6 studies met the study criteria. All studies suggested that the optimal policy was control-limit, and the cancer risk threshold was a non-decreasing function of age. Three studies specified a stopping age for cancer screening. In five studies, the optimal policies outperformed the guidelines or no screening strategy. Conclusions: Two essential factors in screening decisions were cancer risk and age, which were individual variables. The control-limit policy included these factors in decision-making for cancer screening. These policies highlighted personalized screening and showed that this type of screening could outperform cancer screening guidelines regarding economic and clinical benefits.

An algorithm to create model file for Partially Observable Markov Decision Process for mobile robot path planning

The Partially Observable Markov Decision Process (POMDP), a mathematical framework for decision-making in uncertain environments suffers from the curse of dimensionality. There are various methods that can handle huge sizes of POMDP matrices to create approximate solutions, but no serious effort has been reported to effectively control the size of the POMDP matrices. Manually creating the high-dimension matrices of a POMDP model is a cumbersome and sometimes even impossible task. The PCMRPP (POMDP file Creator for Mobile Robot Path Planning) software package implements a novel algorithm to programmatically generate these matrices such that:•The sizes of the matrices can be controlled by configuring the granularity of discretization of the components of the state and•The sparseness of the matrices can be controlled by configuring the spread of the observation probability distribution.This kind of flexibility allows one to achieve a trade-off between time complexity and the level of robustness of the POMDP solution.

EPPTA: Efficient partially observable reinforcement learning agent for penetration testing applications

AbstractIn recent years, penetration testing (pen‐testing) has emerged as a crucial process for evaluating the security level of network infrastructures by simulating real‐world cyber‐attacks. Automating pen‐testing through reinforcement learning (RL) facilitates more frequent assessments, minimizes human effort, and enhances scalability. However, real‐world pen‐testing tasks often involve incomplete knowledge of the target network system. Effectively managing the intrinsic uncertainties via partially observable Markov decision processes (POMDPs) constitutes a persistent challenge within the realm of pen‐testing. Furthermore, RL agents are compelled to formulate intricate strategies to contend with the challenges posed by partially observable environments, thereby engendering augmented computational and temporal expenditures. To address these issues, this study introduces EPPTA (efficient POMDP‐driven penetration testing agent), an agent built on an asynchronous RL framework, designed for conducting pen‐testing tasks within partially observable environments. We incorporate an implicit belief module in EPPTA, grounded on the belief update formula of the traditional POMDP model, which represents the agent's probabilistic estimation of the current environment state. Furthermore, by integrating the algorithm with the high‐performance RL framework, sample factory, EPPTA significantly reduces convergence time compared to existing pen‐testing methods, resulting in an approximately 20‐fold acceleration. Empirical results across various pen‐testing scenarios validate EPPTA's superior task reward performance and enhanced scalability, providing substantial support for efficient and advanced evaluation of network infrastructure security.

Improving Knowledge Learning Through Modelling Students’ Practice-Based Cognitive Processes

Practice is an essential means by which humans and animals engage in cognitive activities. Intelligent tutoring systems, with a crucial component of modelling learners’ cognitive processes during learning and optimizing their learning strategies, offer an excellent platform to investigate students’ practice-based cognitive processes. In related studies, modelling methods for cognitive processes have demonstrated commendable performance. Furthermore, researchers have extended their investigations using decision-theoretic approaches, such as a partially observable Markov decision process (POMDP), to induce learning strategies by modelling the students’ cognitive processes. However, the existing research has primarily centered around the modelling of macro-level instructional behaviors rather than the specific practice selection made by the students within the intricate realms of cognitive domains. In this paper, we adapt the POMDP model to represent relations between the student’s performance on cognitive tasks and his/her cognitive states. By doing so, we can predict his/her performance while inducing learning strategies. More specifically, we focus on question selection during the student’s real-time learning activities in an intelligent tutoring system. To address the challenges on modelling complex cognitive domains, we exploit the question types to automate parameter learning and subsequently employ information entropy techniques to refine learning strategies in the POMDP. We conduct the experiments in two real-world knowledge concept learning domains. The experimental results show that the performance of the learning strategies induced by our new model is superior to that of other learning strategies. Moreover, the new model has good reliability in predicting the student’s performance. Utilizing an intelligent tutoring system as the research platform, this article addresses the modelling and strategy induction challenges of practice-based cognitive processes with intricate structures, aiming to tutor students effectively. Our work provides a new approach of predicting the students’ performance as well as personalizing their learning strategies.

Simulation Evidence of Trust Calibration: Using POMDP with Signal Detection Theory to Adapt Agent Features for Optimised Task Outcome During Human-Agent Collaboration

Appropriately calibrated human trust is essential for successful Human-Agent collaboration. Probabilistic frameworks using a partially observable Markov decision process (POMDP) have been previously employed to model the trust dynamics of human behavior, optimising the outcomes of a task completed with a collaborative recommender system. A POMDP model utilising signal detection theory to account for latent user trust is presented, with the model working to calibrate user trust via the implementation of three distinct agent features: disclaimer message, request for additional information, and no additional feature. A simulation experiment is run to investigate the efficacy of the proposed POMDP model compared against a random feature model and a control model. Evidence demonstrates that the proposed POMDP model can appropriately adapt agent features in-task based on human trust belief estimates in order to achieve trust calibration. Specifically, task accuracy is highest with the POMDP model, followed by the control and then the random model. This emphasises the importance of trust calibration, as agents that lack considered design to implement features in an appropriate way can be more detrimental to task outcome compared to an agent with no additional features.

A Corrosion Maintenance Model Using Continuous State Partially Observable Markov Decision Process for Oil and Gas Pipelines

This paper proposes a continuous state partially observable Markov decision process (POMDP) model for the corrosion maintenance of oil and gas pipelines. The maintenance operations include complex and extensive activities to detect the corrosion type, determine its severity, predict the deterioration rate, and plan future inspection (monitoring) schemes and maintenance policy. A POMDP model is formulated as a decision-making support tool to effectively handle partially observed corrosion defect levels. It formulates states as the pipeline’s degradation level using a probability distribution. Inline inspection (ILI) methods estimate the latest state of the pipeline, which also defines the initial state of the optimization process. The set of actions comprises corrosion mitigation operations. The errors associated with the ILI method are used to construct the observation function for the model. The sum of inspection, maintenance operations, and failure costs for a given state and action are formulated as rewards. Numerical experiments are made based on data collected from the literature. The results showed that different policies, whether derived from solvers (theoretical) or determined from practical experience, can be compared to identify the best maintenance alternative using the model. It was also observed that the choice of the solvers is important since they affect the discounted rewards and the run time to obtain them. The model approximates the parameters and uncertainty associated with the propagation of corrosion, proficiency of inspection methods, and implementation of maintenance policies. Overall, it can be applied to improve the maintenance decision-making process for the oil and gas pipeline as it incorporates the stochastic features of the operation.

Predictive trajectory planning for autonomous vehicles at intersections using reinforcement learning

In this work we put forward a predictive trajectory planning framework to help autonomous vehicles plan future trajectories. We develop a partially observable Markov decision process (POMDP) to model this sequential decision making problem, and a deep reinforcement learning solution methodology to learn high-quality policies. The POMDP model utilizes driving scenarios, condensed into graphs, as inputs. More specifically, an input graph contains information on the history trajectory of the subject vehicle, predicted trajectories of other agents in the scene (e.g., other vehicles, pedestrians, and cyclists), as well as predicted risk levels posed by surrounding vehicles to devise safe, comfortable, and energy-efficient trajectories for the subject vehicle to follow. In order to obtain sufficient driving scenarios to use as training data, we propose a simulation framework to generate socially acceptable driving scenarios using a real world autonomous vehicle dataset. The simulation framework utilizes Bayesian Gaussian mixture models to learn trajectory patterns of different agent types, and Gibbs sampling to ensure that the distribution of simulated scenarios matches that of the real-world dataset collected by an autonomous fleet. We evaluate the proposed work in two complex urban driving environments: a non-signalized T-junction and a non-signalized lane merge intersection. Both environments provide vastly more complex driving scenarios compared to a highway driving environment, which has been mostly the focus of previous studies. The framework demonstrates promising performance for planning horizons as long as five seconds. We compare safety, comfort, and energy efficiency of the planned trajectories against human-driven trajectories in both experimental driving environments, and demonstrate that it outperforms human-driven trajectories in a statistically significant fashion in all aspects.

Investigations on Speed Planning Algorithm and Trajectory Tracking Control of Intersection Scenarios Without Traffic Signs

Efficient intersection planning is one of the most challenging tasks for an autonomous vehicle at present. The degree of response to other traffic participants and whether the path tracking effect is good are the key aspects that determine the performance of planning algorithm for an autonomous vehicle. In order to deal with these problems, in this paper, the frame structure of POMDP (Partially Observable Markov Decision Processes) and the time distance coordinate system are combined to carry out the speed planning for the self-propelled vehicle. The speed planning designs a novel POMDP model to ensure effective prediction of behavioral intentions and decision-making at intersections with no traffic signs. Specifically, the method is considered both the driving position and the time range of arrival of the vehicle at the intersection in real-world situations. At the same time, the algorithm is projected the state information into the time distance coordinate system in the form of probability, and used the value iteration method to solve the optimal solution, so as to improve the adaptability of the algorithm to real-world scenarios. Furthermore, MPC control is added on this basis. Rational use of dynamic MPC and kinematic MPC can effectively reduce the tracking error, improve the stability of tracking, and carry out finer control when the speed frequently fluctuates.

An energy cooperation method of wireless sensor networks based on partially observable Markov decision processes

The performance of routing protocols in wireless sensor networks, such as energy conservation, energy and load balancing. These indicators have a very important impact on the network lifetime. Energy cooperation technology refers to the purposeful transfer of energy from nodes with sufficient energy to nodes with insufficient energy. Firstly, according to POMDP (Partially Observable Markov Decision Process) model, related theories and solving algorithms, this paper proposes an energy cooperation method based on POMDP, establishes an interference decision model based on POMDP, and solves the model. This paper proposes an energy balance routing algorithm based on multi-attribute decision making (EBR_MDM). This algorithm is used to solve the application problem of wireless sensor networks with stable data flow. Including the uneven distribution of network energy, hot spots and energy holes, packet loss caused by node overload and retransmission energy consumption. According to the relative entropy, the attribute weight is calculated, and the optimal forward data is selected to the neighboring nodes. The simulation results show that the network running time of EBR_MDM is increased by 22.1 %, 48.7 % and 101.3 % respectively compared with other comparison protocols. Finally, because the sensor nodes are randomly arranged in the predetermined monitoring area, and the distribution of each node is chaotic, this paper establishes TMA (Trilateral Measurement Algorithm) algorithm to locate the target. The simulation results show that the positioning error of TMA may be about 1.2 m due to trilateration, but it is still within an acceptable range.

A reactive power optimization partially observable Markov decision process with data uncertainty using multi-agent actor-attention-critic algorithm

We present an innovative partially observable Markov decision process (POMDP) modelling method for the reactive power optimization process of the active distribution network (ADN) under the high permeability of the distributed generation. This model is tolerant of the uncertainty resulting from data uncertainty. We believe that the belief state space in the POMDP model corresponds to the state space in the Markov decision process (MDP) model, and we apply the multi-agent actor-attention-critic (MAAC) reinforcement learning (RL) algorithm to the proposed model. This technique extracts the most effective information with the highest quality from the huge historical measurement database, hence enhancing the learning effectiveness of agents and the stability of the optimization strategy. We simulate reactive power optimization in a modified IEEE-33 nodes ADN and a modified IEEE-123nodes ADN. The simulation demonstrates the stability and economic superiority of the proposed approach under varying degrees of data uncertainty relative to previous RL algorithms based on the MDP model. The simulation demonstrates that the proposed POMDP model is more appropriate for the real operation of the partially observable distribution network than the MDP models. And the optimal strategy obtained by the proposed MAAC algorithm is reliable with deteriorating data quality.

A Dynamic Deep Reinforcement Learning-Bayesian Framework for Anomaly Detection

To assure the successful operation of connected and automated vehicles, it is critical to detect and isolate anomalous and/or faulty information in a timely manner. To do so, anomaly detection techniques should be implemented in real-time where if the probability of anomalous information exceeds a certain threshold, the information is dealt with accordingly. Traditionally, the threshold for judging whether the data is anomalous is fixed and determined a priori. However, not only does this approach fail to account for the feedback obtained during a trip on the performance of the algorithms, but it also fails to respond to potential changes in rates of anomalies. Hence, it is important to develop an approach that can dynamically alter this threshold in response to exogenous factors to assure reliable and robust system operation. We develop a mathematical framework which utilizes a dynamic threshold for an anomaly classification algorithm in order to maximize the safety of a trip. Specifically, we develop and pair an anomaly classification algorithm based on convolutional neural networks (CNN), with a partially observable Markov decision process (POMDP) model. We solve the resulting POMDP model using the asynchronous advantage actor critic (A3C) deep reinforcement learning algorithm. The prescribed policy determines the anomaly classification threshold in real-time that maximizes the performance. Our numerical experiments show that the POMDP model outperforms state-of-the-art benchmarks, especially under more difficult to detect anomaly profiles.

Fight inventory shrinkage: Simultaneous learning of inventory level and shrinkage rate

In 2020, inventory shrinkage eroded $61.7 billion profit in the U.S. retail industry. Unfortunately, fighting inventory shrinkage to protect retailers' already slim profits is challenging due to unknown shrinkage rates and invisible inventory levels. While the latter has been studied in the literature, the former has not. To deal with this challenge, we introduce two new features to the Bayesian inventory models: (1) interleaving customer and theft arrival processes that contribute to actual sales and shrinkages, respectively, and (2) learning of both inventory level and shrinkage rate. We first derive the learning formulae using the triple‐censored sales data (invisible lost sales, shrinkages, and “lost shrinkages”) and then use them to construct a POMDP (partially observable Markov decision process) model for making inventory and loss prevention decisions. For a different level of information deficiency, we analyze the model property and design heuristic order policies to capture the benefit of learning. Through a numerical study, we show that our estimated shrinkage rate converges quickly and monotonically to the actual value. For products with high shrinkage rates (5–12%), our heuristic policy can help seize 82–94% of the ideal profit retailers could earn under full information. We note that feature (1) of our model is crucial. It not only reflects the actual arrival order but also allows us to learn the unknown shrinkage rate, which, in turn, can prevent serious underordering and vicious inventory cycles and can increase the profit by 108% in some cases. Our approach thus enables both effective inventory management and early identification of ineffective loss prevention strategies, reducing shrinkage, and increasing sales and profit.

Value of information analysis in non-stationary stochastic decision environments: A reliability-assisted POMDP approach

Optimal management of systems over their service life as they face a multitude of uncertainties remains a significant challenge. While additional information can reduce uncertainties, collecting new information incurs cost and may include observation error. Value of Information (VoI) analysis facilitates quantitative assessment of the expected net benefits of collecting new information. Moreover, partially observable Markov decision processes (POMDPs) can be integrated within VoI analysis to efficiently capture the sequential decision-making environments for systems. The assumption of stationary environment in existing POMDP frameworks for VoI analysis may not be valid, however, in many applications such as deterioration processes which are often non-stationary. To address this gap, this paper presents a new approach called VoI-R-POMDP. A new POMDP framework is proposed to accurately describe non-stationary processes using multiple integrated transition models. New strategies based on reliability concepts are developed to accurately and efficiently determine the parameters of the proposed POMDP model based on prior information. A new formulation of the observation function based on Bayes’ theorem is also derived. The proposed framework is applied to a corroding beam example. Results indicate that VoI-R-POMDP can accurately and efficiently describe the deterioration process and thus provide accurate VoI estimates for non-stationary systems.

An Analytics‐Driven Approach for Optimal Individualized Diabetes Screening

Type 2 diabetes is a chronic disease that affects millions of Americans and puts a significant burden on the healthcare system. The medical community sees screening patients to identify and treat prediabetes and diabetes early as an important goal; however, universal population screening is operationally not feasible, and screening policies need to take characteristics of the patient population into account. For instance, the screening policy for a population in an affluent neighborhood may differ from that of a safety‐net hospital. The problem of optimal diabetes screening—whom to screen and when to screen—is clearly important, and small improvements could have an enormous impact. However, the problem is typically only discussed from a practical viewpoint in the medical literature; a thorough theoretical framework from an operational viewpoint is largely missing. In this study, we propose an approach that builds on multiple methods—partially observable Markov decision process (POMDP), hidden Markov model (HMM), and predictive risk modeling (PRM). It uses available clinical information, in the form of electronic health records (EHRs), on specific patient populations to derive an optimal policy, which is used to generate screening decisions, individualized for each patient. The POMDP model, used for determining optimal decisions, lies at the core of our approach. We use HMM to estimate the cohort‐specific progression of diabetes (i.e., transition probability matrix) and the emission matrix. We use PRM to generate observations—in the form of individualized risk scores—for the POMDP. Both HMM and PRM are learned from EHR data. Our approach is unique because (i) it introduces a novel way of incorporating predictive modeling into a formal decision framework to derive an optimal screening policy; and (ii) it is based on real clinical data. We fit our model using data on a cohort of more than 60,000 patients over 5 years from a large safety‐net health system and then demonstrate the model's utility by conducting a simulation study. The results indicate that our proposed screening policy outperforms existing guidelines widely used in clinical practice. Our estimates suggest that implementing our policy for the studied cohort would add one quality‐adjusted life year for every patient, and at a cost that is 35% lower, compared with existing guidelines. Our proposed framework is generalizable to other chronic diseases, such as cancer and HIV.

Effectual learning toward causal competence –a POMDP model of decision-making sequences

Effectuation is often viewed as more preferable to causation for managing uncertainty. Introducing process theory’s perspective on sequencing, we challenge this view by arguing that entrepreneurs navigate better under uncertainty by interpolating between decision logics. We show that the rate of dynamic switching between decision logics (causal versus effectual) increases as entrepreneurs approach the plunge decision. To unpack the evolution of entrepreneurs’ decision-making sequences in the lead-up to new venture creation, we conducted 3 studies. From a sample of 389 decision events identified from the narratives of 30 successful entrepreneurs, in Study 1, we derive model parameters for a partially observable Markov decision process (POMDP) solution. In Study 2, we analyse the decision-making pattern where effectual logics predominate up until the ‘plunge’ decision and proceeds recursively until that plunge decision results in a ‘go’ or ‘abandon’ decision. Finally, in Study 3, we examine the conditions under which the rate of switching between decision logics becomes more dynamic. We conclude by exploring possible explanations for these findings in the qualitative data.

Joint beamforming and user association with reduced CSI signaling in mobile environments: A Deep Q-learning approach

Heterogeneous Cloud Radio Access Network (H-CRAN) is considered as a cost-efficient network solution to meet 5G data traffic requirements. In this paper, we consider the problem of beamforming and user clustering (user-to-Remote Radio Head (RRH) association) in the downlink of a H-CRAN where users have heterogeneous mobility profiles. Given the rapidly time-varying nature of the mobile wireless environment, it is challenging to offer an optimal beamforming and user association performance during a long-term allocation process without incurring large Channel State Information (CSI) and signaling overheads. For that purpose, we proposed in Ha et al. (2019) an Adaptive Beamforming and User Clustering (ABUC) algorithm which resolves the joint beamforming and user clustering problem when considering CSI cost and imperfectness under user mobility assumptions. In this paper, we design a deep reinforcement-learning framework which enables the proposed ABUC algorithm to select on-the-fly its best feedback scheduling parameters, namely the period and type of CSI feedback, given each user mobility profile. The proposed ABUC-DQL approach can overcome the scalability limitation of the Q-learning approach (Ha et al., 2019) and better handle the problem when formulated using a POMDP (Partially Observable Markov Decision Process) model. The simulation results show that the convergence time is mainly impacted by the number of users in the network, and the online-learning ability of the framework can quickly adapt to the changes of users mobility.