Markov Decision Process-Based Research Articles

Online Learning Strategy Induction through Partially Observable Markov Decision Process-Based Cognitive Experience Model

Inducing learning strategies is a crucial component of intelligent tutoring systems. Previous research has predominantly focused on the induction of offline learning strategies. Although the existing offline learning strategy induction methods can also be used for real-time updates of learning strategies, their update efficiency is not high, making it difficult to capture the characteristics exhibited by learners during the learning process in a timely manner. With the superior performance of the Partially Observable Markov Decision Process (POMDP), this paper proposes a POMDP-based cognitive experience model, which can be quickly updated during interactions and enables the real-time induction of learning strategies by weighting the learning experiences of different learners. Experimental results demonstrate that the learning strategies induced by PCEM are more personalized and exhibit superior performance.

Distributed Handoff Problem in Heterogeneous Networks With End-to-End Network Slicing: Decentralized Markov Decision Process-Based Modeling and Solution

Heterogeneous networks (HetNets) with end-to-end (E2E) network slicing are regarded as effective approaches to meet diverse service requirements from vertical industries. Due to the dense deployment of base stations (BSs) and the complicated associations between BSs and E2E network slices (NSs) in the scenario, the handoff problem faces challenges of the huge system state space and handoff action space and the considerable communication overhead. In this paper, we take these issues into account and consider a distributed E2E NS handoff decision framework in the HetNet. A decentralized Markov decision process (DEC-MDP)-based model is formulated for the distributed E2E NS handoff problem, and the jointly observable and random characteristics of the DEC-MDP are analyzed. To obtain a theoretical performance reference, the original distributed E2E NS handoff problem is simplified, and a Nash equilibrium-based performance bound is given. More practically, the multi-agent double deep Q-network-based distributed handoff (MA-DDQN-DH) algorithm with the centralized training and decentralized executing framework is proposed. Simulation results show that the Nash equilibrium-based performance bound is reasonable, and the proposed MA-DDQN-DH algorithm performs well in the comparison.

A Fast Markov Decision Process-Based Algorithm for Collision Avoidance in Urban Air Mobility

Multiple aircraft collision avoidance is a challenging problem due to a stochastic environment and uncertainty in the intent of other aircraft. Traditionally a layered approach to collision avoidance has been employed using a centralized air traffic control system, established rules of the road, separation assurance, and last minute pairwise collision avoidance. With the advent of Urban Air Mobility (air taxis), the expected increase in traffic density in urban environments, short time scales, and small distances between aircraft favor decentralized decision making on-board the aircraft. In this paper, we present a Markov Decision Process (MDP) based method, named FastMDP, which can solve a certain subclass of MDPs quickly, and demonstrate using the algorithm online to safely maintain separation and avoid collisions with multiple aircraft (1-on-n) while remaining computationally efficient. We compare the FastMDP algorithm’s performance against two online collision avoidance algorithms that have been shown to be both efficient and scale to large numbers of aircraft: Optimal Reciprocal Collision Avoidance (ORCA) and Monte Carlo Tree Search (MCTS). Our simulation results show that under the assumption that aircraft do not have perfect knowledge of other aircraft intent FastMDP outperforms ORCA and MCTS in collision avoidance behavior in terms of loss of separation and near mid-air collisions while being more computationally efficient. We further show that in our simulation FastMDP behaves nearly as well as MCTS with perfect knowledge of other aircraft intent. Our results show that FastMDP is a promising algorithm for collision avoidance that is also computationally efficient.

A Novel Markov Decision Process-Based Solution for Improved Quality Prioritized Video Delivery

The recent growth in both number of high specification mobile devices and network multimedia data demand, has made difficult supporting access to the multimedia content at high user Quality of Experience (QoE) levels. It is even more challenging to support offering such services adjusted to user specific requirements in current heterogeneous wireless network environments (HWNE). This paper proposes a novel Prioritized Adaptive Real-time Multi-user Access Network Selection framework (P-ARMANS) which employs a Markov Decision Process (MDP) solution to perform improved bandwidth resource allocation. P-ARMANS enables load balancing during multimedia delivery when users with diverse priority and different device screen resolutions access various services. Modeling and simulations show how P-ARMANS distributes content with different types of traffic and load among typical and business users at higher QoE levels compared to a classic no-priority approach.

A Bayesian regression approach to handicapping tennis players based on a rating system

Abstract This paper builds on a recently developed Markov Decision Process-based (MDP) handicap system for tennis, which aims to make amateur matches more competitive. The system gives points to the weaker player based on skill difference, which is measured by the point-win probability. However, estimating point-win probabilities at the amateur level is challenging since point-level data is generally only available at the professional level. On the other hand, tennis rating systems are widely used and provide an estimate of the difference in ability between players, but a rigorous determination of handicap using rating systems is lacking. Therefore, our goal is to develop a mapping between the Universal Tennis Rating (UTR) system and the MDP-based handicaps, so that two amateur players can determine an appropriate handicap for their match based only on their UTRs. We first develop and validate an approach to extract server-independent point-win probabilities from match scores. Then, we show how to map server-independent point-win probabilities to server-specific point-win probabilities. Finally, we use the estimated probabilities to produce handicaps via the MDP model, which are regressed against UTR differences between pairs of players. We conclude with thoughts on how a handicap system could be implemented in practice.

Partially Observable Markov Decision Process-Based Transmission Policy over Ka-Band Channels for Space Information Networks

The Ka-band and higher Q/V band channels can provide an appealing capacity for the future deep-space communications and Space Information Networks (SIN), which are viewed as a primary solution to satisfy the increasing demands for high data rate services. However, Ka-band channel is much more sensitive to the weather conditions than the conventional communication channels. Moreover, due to the huge distance and long propagation delay in SINs, the transmitter can only obtain delayed Channel State Information (CSI) from feedback. In this paper, the noise temperature of time-varying rain attenuation at Ka-band channels is modeled to a two-state Gilbert–Elliot channel, to capture the channel capacity that randomly ranging from good to bad state. An optimal transmission scheme based on Partially Observable Markov Decision Processes (POMDP) is proposed, and the key thresholds for selecting the optimal transmission method in the SIN communications are derived. Simulation results show that our proposed scheme can effectively improve the throughput.

Markov Decision Process-Based Distributed Conflict Resolution for Drone Air Traffic Management

Ensuring safety and providing timely conflict alerts to small unmanned aircraft, commonly known as drones, is important to their integration into civil airspace. This paper proposes a short-term conflict avoidance algorithm for an automated low-altitude, small unmanned aircraft traffic management system. The goal is to balance between aircraft safety and efficiency subject to uncertainty in the environment, aircraft, and pilot response. The proposed algorithm generates advisories for each aircraft to follow, and is based on decomposing a large multiagent Markov decision process and fusing their solutions. As a result, the method scales well and resolves conflicts efficiently. Further, this paper presents a massively distributed architecture used to implement the conflict avoidance algorithm on a practical scale in which simultaneous conflicts can be solved in parallel in tens of milliseconds. Our controller significantly outperforms two baseline algorithms based on uncoordinated and closest-threat heuristics. In terms of conflict probability, our simulations demonstrate that our method is an order of magnitude better than the latter and about 10% better than the former.

A Markov Decision Process-Based Opportunistic Spectral Access

Recent interest in Internet of Things has resulted in increased activities in the area of dynamic spectrum access, which fosters cognitive radio networks that provide opportunistic access. This letter proposes optimal opportunistic access through a Markov decision process for a user of interest in a single channel environment competing with other users. Consequently, we analyze the model in terms of interference and wastage and compare it with other Markov models from the existing literature. One should note that the models are compared to give an overall evaluation of performance, as a detailed comparison cannot be made, due to the use of different system models, transitional probabilities, etc.

Markov Decision Process-Based Switching Algorithm for Sustainable Rechargeable Wireless Sensor Networks

In a tree-based wireless sensor network (WSN), a tree structure rooted at sink node is usually created for efficient data collection. Recently, the use of solar harvesting technologies for rechargeable sensor nodes is evolving. Moreover, in a tree-based rechargeable WSN, the nodes that belong to different routes will have different energy dissipation due to unequal harvested-energy and utilized-energy. Network sustainability and energy efficiency are important issues in a tree-based rechargeable sensor network. In this paper, a Markov decision process-based switching algorithm has been designed for a sustainable data collection tree while reducing energy consumption in the network. Furthermore, an analysis of energy consumption has been performed using a real-time sensor traffic pattern. A prediction model has been adopted to estimate the harvesting energy (based on solar power) for the rechargeable sensor nodes. In this paper, the state of each node is defined based on different independent energy levels. The state of each node may change with time depending on harvested-energy and utilized-energy. The proposed Markov decision process approach finds the optimal switching policy for sensor nodes, which switch from one parent to another based on energy levels to preserve sustainability. A detailed theoretical analysis has been performed along with simulation results to show the efficacy of the proposed approach.

Dynamic-cost-reward connection admission control for maximizing system reward in 4G wireless multihop relaying networks

The 3GPP Long Term Evolution (LTE) Advanced and IEEE 802.16j specifications adopt the mobile multi-hop relaying (MMR) mechanism for enlarging service area and improving wireless transmission quality simultaneously. By deploying different types of Relay Stations (RSs), MMR can bring some advantages: (1) the signal fading and wireless interference of a single long wireless link is improved obviously; (2) the ranges of wireless access and relay area are extended, etc. MMR can offer a high data rate transmission for packet services and can increase system capacity. Note that MMR can be applied to the public transportation system, e.g., equipped a mobile RS on a high-speed train. A mobile RS handoff initializes a multiple handoff requests of different types of traffics. It becomes as a critical handoff issue in 4G MMR. Thus, the MMR handoff needs a new efficient Connection Admission Control (CAC) to guarantee qualities for various types of traffics and to increase system revenue. However, traditional CACs are difficult to fulfill the objectives. This paper thus proposes the Dynamic Cost-Reward-based (DCR) CAC that consists of two key mechanisms: (1) adopting a Markov decision process-based (MDP) cost function and (2) providing different reward functions for different types of nodes and various types of connection. Additionally, a mathematical analytical Markov chain is modeled for DCR. The simulation results are very close to the analysis results, which justifies the correctness of the analytical model. Numerical results demonstrate that DCA outperforms the compared CACs in the probabilities of new blocking, MS-handoff, and RS-handoff dropping, FRL, GoS, and system reward.

Modeling appraisal in theory of mind reasoning

Cognitive appraisal theories, which link human emotional experience to their interpretations of events happening in the environment, are leading approaches to model emotions. Cognitive appraisal theories have often been used both for simulating “real emotions” in virtual characters and for predicting the human user’s emotional experience to facilitate human–computer interaction. In this work, we investigate the computational modeling of appraisal in a multi-agent decision-theoretic framework using Partially Observable Markov Decision Process-based (POMDP) agents. Domain-independent approaches are developed for five key appraisal dimensions (motivational relevance, motivation congruence, accountability, control and novelty). We also discuss how the modeling of theory of mind (recursive beliefs about self and others) is realized in the agents and is critical for simulating social emotions. Our model of appraisal is applied to three different scenarios to illustrate its usages. This work not only provides a solution for computationally modeling emotion in POMDP-based agents, but also illustrates the tight relationship between emotion and cognition—the appraisal dimensions are derived from the processes and information required for the agent’s decision-making and belief maintenance processes, which suggests a uniform cognitive structure for emotion and cognition.

Reliability-based bridge design and life cycle management with Markov decision processes

A Markov Decision Process-based (MDP) modelling approach is used to identify optimal structural designs and their associated maintenance policies. This approach has the desirable attributes of capturing the dynamics of the coupled structural design/maintenance problem while yieldding a mathematical model that is computationally tractable. Implicit in the MDP is the computationally and theoretically usful choice of modelling the dynamics of structural performance and maintenance actions as two related but separate processes. Structural resistance deterioration due to corrosion and stochastic structural loading are modelled in the MDP as a “self-transition” process, that is, a process that models structural performance without maintenance intervention. A suite of maintenance actions and their associated costs constitute the core of the “decision effect” process. These two processes, made computationally operational through the development of self-transition and decision effect matrices, are then combined into a joint transition matrix, with which we can model the effects of maintenance actions on bridge performance and costs. Initial designs and their associated minimum expected, discounted lifetime costs are identified with the MDP for a two girder bridge.