Markov Decision Chains Research Articles

Reinforcement learning strategies using Monte-Carlo to solve the blackjack problem

Blackjack is a classic casino game in which the player attempts to outsmart the dealer by drawing a combination of cards with face values that add up to just under or equal to 21 but are more incredible than the hand of the dealer he manages to come up with. This study considers a simplified variation of blackjack, which has a dealer and plays no active role after the first two draws. A different game regime will be modeled for everyone to ten multiples of the conventional 52-card deck. Irrespective of the number of standard decks utilized, the game is played as a randomized discrete-time process. For determining the optimum course of action in terms of policy, we teach an agent-a decision maker-to optimize across the decision space of the game, considering the procedure as a finite Markov decision chain. To choose the most effective course of action, we mainly research Monte Carlo-based reinforcement learning approaches and compare them with q-learning, dynamic programming, and temporal difference. The performance of the distinct model-free policy iteration techniques is presented in this study, framing the game as a reinforcement learning problem.

A decision-making design approach to optimize the hole arrangement of effusion cooling plates under non-uniform heat load conditions

The harsh operating conditions of aero-engines contain non-uniform temperatures and variable hotspots, which require adaptive cooling techniques to accommodate the heat loads on the external surfaces of metal. Among the existing technologies, effusion cooling is an important way to fulfill such quests due to its flexibility in controlling coolant locally. As an effusively cooled component generally contains hundreds of holes, hole arrangement optimization is a high-dimensional problem that is difficult to solve using existing evolutionary algorithms. Recently matured deep reinforcement learning algorithms showed success in many research fields by incorporating data modeling with sequential optimization, which demonstrated the ability to convert high dimensional optimization problems into very low dimensional iterative optimization problems. The present study proposed a decision-making design approach to optimize the hole arrangement of a series of effusion cooling plates. Hole arrangements were transformed into Markov decision chains of opening or closing each hole. The decision chains were solved by a reinforcement learning framework integrated with a deep learning-based environment simulator. Results indicated that the proposed approach could yield an elevation of cooling performance by 20% as compared with the results from genetic algorithms. Moreover, an analytical correlation was extracted from the clear decision boundaries of reinforcement learning. Upon further validation, the computational cost to apply the correlation to other non-uniform heat loads was within seconds, while the cooling effect could reach the same optimization level as reinforcement learning. The idea proposed in this study could be applied to the arrangement optimization of multiple cooling technologies, such as film/effusion cooling, impingement cooling, and pin–fin array cooling.

Characterization of the optimal average cost in Markov decision chains driven by a risk-seeking controller

AbstractThis work concerns Markov decision chains on a denumerable state space endowed with a bounded cost function. The performance of a control policy is assessed by a long-run average criterion as measured by a risk-seeking decision maker with constant risk-sensitivity. Besides standard continuity–compactness conditions, the framework of the paper is determined by the following conditions: (i) the state process is communicating under each stationary policy, and (ii) the simultaneous Doeblin condition holds. Within this framework it is shown that (i) the optimal superior and inferior limit average value functions coincide and are constant, and (ii) the optimal average cost is characterized via an extended version of the Collatz–Wielandt formula in the theory of positive matrices.

Arthroscopic Bankart repair versus nonoperative management for first-time anterior shoulder instability: A cost-effectiveness analysis.

Arthroscopic Bankart repair (ABR) may be more effective than nonoperative management for patients with anterior shoulder instability following first-time dislocation. The purpose of the study was to determine the most cost-effective treatment strategy by evaluating the incremental cost-effectiveness ratio (ICER) for ABR versus nonoperative treatment. This cost-effectiveness study utilized a Markov decision chain and Monte Carlo simulation. Probabilities, health utility values, and outcome data regarding ABR and nonoperative management of first-time shoulder instability derived from level I/II evidence. Costs were tabulated from Centers for Medicaid & Medicare Services. Probabilistic sensitivity analysis was performed using >100,000 repetitions of the Monte Carlo simulation. A willingness-to-pay (WTP) threshold was set at $50,000. The expected cost for operative management higher than nonoperative management ($32,765 vs $29,343). However, ABR (5.48 quality-adjusted life years (QALYs)) was the more effective treatment strategy compared to nonoperative management (4.61 QALYs). The ICER for ABR was $3943. Probabilistic sensitivity analysis showed that ABR was the most cost-effective strategy in 100% of simulations. ABR is more cost-effective than nonoperative management for first-time anterior shoulder dislocation. The threshold analysis demonstrated that when accounting for WTP, ABR was found to be the more cost-effective strategy.

Contractive approximations in average Markov decision chains driven by a risk-seeking controller

Contractive approximations in average Markov decision chains driven by a risk-seeking controller

Cracking propagation in expansive soils under desiccation and stabilization planning using Bayesian inference and Markov decision chains.

Desiccation cracking endangers the stability of expansive soils subjected to cyclic moisture variations. In the current research, prominent cracking prediction models including linear, linear elastic, linear elastoplastic, and linear elastic fracture were studied. Then, Monte Carlo limit state functions were generated based on predictions. Results indicate that there is less than 5% chance of cracking for depths beyond 0.5, 6, 8, and 9 m as predicted by the linear elastoplastic, linear elastic, linear, and linear elastic fracture models, respectively. Moreover, a series of sensitivity analysis was performed to evaluate model and parameter uncertainties. Comparatively, it was found that the linear model exhibits the highest uncertainty while linear elastoplastic model possesses the least uncertainty thus yielding a reasonable prediction. Additionally, soil parameters including matric suction followed by dry density were identified to govern the overall cracking. Using Bayesian inference, numerous conditional probabilities of variation of soil properties were investigated. Then, several cracking probabilities under history of low to high matric suction and dry density were obtained. Accordingly, Monte Carlo Markov decision chains were established based on several ecofriendly and feasible stabilization policies and their performance was also evaluated. The obtained safety factors (SF) suggest that stabilization plans resulting in high moisture and dry density have the least likelihood of cracking with a SF equal to 5.1. However, stabilization policies having low dry density and moisture yield have the least SF of 0.39. Findings of this study can improve the decision-making processes for expansive soil stabilization by considering a variety of environmental conditional probabilities.

An efficient spectrum scheduling mechanism using Markov decision chain for 5G mobile network

Recently, the 5G as the next-generation network is a popular research and discussed widely. The architecture of 5G is a heterogeneous network, and it can support more networked types, like the ultra-dense network, traditional cellular network, and Machine to Machine communication. Although the high frequency and larger bandwidth have been using in 5G, resource allocation is still a critical issue that needs to be discussed and solved. Consider the spectrum resource is limited, but almost all users hope that equipment can get a better quality of services. Therefore, how to manage the spectrum resource and allocation is a big problem. Consider the fast-growing devices and traffic in the future; hence, task scheduling for UEs to reduce energy consumption will be focused on. To solve resource allocation and minimise energy consumption, the Markov decision chain is proposed to be used to predict the channel state. The modified particle swarm optimization (MPSO) is also used in this paper to find the best task scheduling. The simulation will be used to verify the performance of the mechanism that is used and compare it with PSO and first-in-first-service (FIFS). The result shows the method used can be scheduled for the task efficiently.

A Discounted Approach in Communicating Average Markov Decision Chains Under Risk-Aversion

This work concerns with discrete-time Markov decision processes on a denumerable state space. Assuming that the decision maker is risk-averse with constant risk-sensitivity coefficient, the performance of a control policy is measured by an average criterion associated with a non-negative and bounded cost function. Under conditions ensuring that the optimal average cost is constant, but not necessarily determined via the average cost optimality equation, it is shown that a discounted criterion can be used to approximate the optimal average index.

Risk-sensitive average optimality in Markov decision processes

In this note attention is focused on finding policies optimizing risk-sensitive optimality criteria in Markov decision chains. To this end we assume that the total reward generated by the Markov process is evaluated by an exponential utility function with a given risk-sensitive coefficient. The ratio of the first two moments depends on the value of the risk-sensitive coefficient; if the risk-sensitive coefficient is equal to zero we speak on risk-neutral models. Observe that the first moment of the generated reward corresponds to the expectation of the total reward and the second central moment of the reward variance. For communicating Markov processes and for some specific classes of unichain processes long run risk-sensitive average reward is independent of the starting state. In this note we present necessary and sufficient condition for existence of optimal policies independent of the starting state in unichain models and characterize the class of average risk-sensitive optimal policies.

Characterization of the Optimal Risk-Sensitive Average Cost in Denumerable Markov Decision Chains

This work is concerned with Markov decision chains on a denumerable state space. The controller has a positive risk-sensitivity coefficient, and the performance of a control policy is measured by a risk-sensitive average cost criterion. Besides standard continuity-compactness conditions, it is assumed that the state process is communicating under any stationary policy, and that the simultaneous Doeblin condition holds. In this context, it is shown that if the cost function is bounded from below, and the superior limit average index is finite at some point, then (i) the optimal superior and inferior limit average value functions coincide and are constant, (ii) the optimal average cost is characterized via an extended version of the Collatz-Wielandt formula in the theory of positive matrices, and (iii) an optimality inequality is established, from which a stationary optimal policy is obtained. Moreover, an explicit example is given to show that, even if the cost function is bounded, the strict inequality may occur in the optimality relation.

Vanishing discount approximations in controlled Markov chains with risk-sensitive average criterion

Abstract This work concerns Markov decision chains on a finite state space. The decision-maker has a constant and nonnull risk sensitivity coefficient, and the performance of a control policy is measured by two different indices, namely, the discounted and average criteria. Motivated by well-known results for the risk-neutral case, the problem of approximating the optimal risk-sensitive average cost in terms of the optimal risk-sensitive discounted value functions is addressed. Under suitable communication assumptions, it is shown that, as the discount factor increases to 1, appropriate normalizations of the optimal discounted value functions converge to the optimal average cost, and to the functional part of the solution of the risk-sensitive average cost optimality equation.

Second order optimality in Markov decision chains

The article is devoted to Markov reward chains in discrete-time setting with finite state spaces. Unfortunately, the usual optimization criteria examined in the literature on Markov decision chains, such as a total discounted, total reward up to reaching some specific state (called the first passage models) or mean (average) reward optimality, may be quite insufficient to characterize the problem from the point of a decision maker. To this end it seems that it may be preferable if not necessary to select more sophisticated criteria that also reflect variability-risk features of the problem. Perhaps the best known approaches stem from the classical work of Markowitz on mean variance selection rules, i. e. we optimize the weighted sum of average or total reward and its variance. The article presents explicit formulae for calculating the variances for transient and discounted models (where the value of the discount factor depends on the current state and action taken) for finite and infinite time horizon. The same result is presented for the long run average nondiscounted models where finding stationary policies minimizing the average variance in the class of policies with a given long run average reward is discussed.

The cost-effectiveness of the arthroscopic Bankart versus open Latarjet in the treatment of primary shoulder instability

The purpose of this study was to conduct a cost-effectiveness analysis of the arthroscopic Bankart and the open Latarjet in the treatment of primary shoulder instability. This cost-effectiveness study used a Markov decision chain and Monte-Carlo simulation. Existing literature was reviewed to determine the survivorship and complication rates of these procedures. Health utility states (EQ-5D and quality-adjusted life-years) of the Bankart and Latarjet were prospectively collected. Using these variables, the Monte-Carlo simulation was modeled 100,000 times. In reviewing the literature, the overall recurrence rate is 14% after the arthroscopic Bankart and 8% after the open Latarjet. Postoperative health utility states were equal between the 2 procedures (mean EQ-5D, 0.930; P = .775). The Monte-Carlo simulation showed that the Bankart had an incremental cost-effectiveness ratio of $4214 and the Latarjet had an incremental cost-effectiveness ratio of $4681 (P < .001). Both the arthroscopic Bankart and open Latarjet are highly cost-effective; however, the Bankart is more cost-effective than the Latarjet, primarily because of a lower health utility state after a failed Latarjet. Ultimately, the clinical scenario may favor Latarjet (ie, critical glenoid bone loss) in certain circumstances, and decisions should be made on a case by case basis.

Polynomial-Time Computation of Strong and n-Present-Value Optimal Policies in Markov Decision Chains

This paper studies the problem of finding a stationary strong present-value optimal and, more generally, an n-present-value optimal, policy for an infinite-horizon stationary finite-state-action substochastic Markov decision chain. A policy is strong present-value optimal if it has maximum present value for all small positive interest rates ρ. A policy is n-present-value optimal if its present value falls below the maximum present value for all small positive ρ by O(ρn+1). The importance of stationary n-present-value optimal policies arises from the following known facts. The set of such policies diminishes with n and, for n ≥ S, is precisely the set of stationary strong present-value optimal policies. For 0 ≤ n &lt; S, stationary n-present-value optimal policies are nearly strong present-value optimal and are of independent interest. The best algorithms for finding stationary strong present-value optimal policies find stationary n-present-value policies for n = −1,…,S in that order. This paper decomposes the problem of finding a stationary n-present-value optimal policy given a stationary (n − 1)-present-value optimal policy into a sequence of three subproblems, each entailing either maximizing transient value or reward rate. It is known that both subproblem types can be represented as linear programs and solved in polynomial time, e.g., by interior-point and ellipsoid methods. This paper also establishes the following results. The size of the input to each subproblem is polynomial in the size of the original problem data. A stationary strong present-value (respectively, n-present-value) optimal policy can be found in polynomial time. For the case of unique-transition systems, i.e., each action in a state sends the system to at most one state, a stationary strong present-value (respectively, n-present-value) optimal policy can be found in strongly polynomial time using combinatorial methods on the subproblems. The last case includes standard deterministic dynamic programs.

A Characterization of the Optimal Certainty Equivalent of the Average Cost via the Arrow-Pratt Sensitivity Function

This work is concerned with finite-state irreducible Markov decision chains satisfying continuity-compactness requirements. It is supposed that the system is driven by a decision maker with utility function U, which, aside mild conditions, is arbitrary, and the performance of a control policy is measured by the long-run average cost criterion induced by U. The main conclusions about this performance index are as follows: (i) the optimal U-average value function coincides with the optimal V-average index for a certain exponential utility V, and (ii) the average criteria associated with U and V have the same class of optimal stationary policies.

Sample-Path Optimal Stationary Policies in Stable Markov Decision Chains with the Average Reward Criterion

This paper concerns discrete-time Markov decision chains with denumerable state and compact action sets. Besides standard continuity requirements, the main assumption on the model is that it admits a Lyapunov function ℓ. In this context the average reward criterion is analyzed from the sample-path point of view. The main conclusion is that if the expected average reward associated to ℓ2 is finite under any policy then a stationary policy obtained from the optimality equation in the standard way is sample-path average optimal in a strong sense.

Sample-Path Optimal Stationary Policies in Stable Markov Decision Chains with the Average Reward Criterion

This paper concerns discrete-time Markov decision chains with denumerable state and compact action sets. Besides standard continuity requirements, the main assumption on the model is that it admits a Lyapunov function ℓ. In this context the average reward criterion is analyzed from the sample-path point of view. The main conclusion is that if the expected average reward associated to ℓ2is finite under any policy then a stationary policy obtained from the optimality equation in the standard way is sample-path average optimal in a strong sense.

OPTIMIZATION OF OVERFLOW POLICIES IN CALL CENTERS

We examine how overflow policies in a multi-skill call center should be designed to accommodate performance measures that depend on waiting time percentiles such as service level. This is done using a discrete Markovian approximation of the waiting time of the first customers waiting in line. A Markov decision chain is used to determine the optimal policy. This policy outperforms considerably the ones used most often in practice, which use a fixed threshold. The present method can be used also for other call-center models and other situations where performance is based on actual waiting times and customers are treated in a FCFS order.

Continuity of the optimal average cost in Markov decision chains with small risk-sensitivity

This note concerns discrete-time controlled Markov chains driven by a decision maker with constant risk-sensitivity \(\lambda \). Assuming that the system evolves on a denumerable state space and is endowed with a bounded cost function, the paper analyzes the continuity of the optimal average cost with respect to the risk-sensitivity parameter, a property that is promptly seen to be valid at each no-null value of \(\lambda \). Under standard continuity-compactness conditions, it is shown that a general form of the simultaneous Doeblin condition allows to establish the continuity of the optimal average cost at \(\lambda = 0\), and explicit examples are given to show that, even if every state is positive recurrent under the action of any stationary policy, the above continuity conclusion can not be ensured under weaker recurrence requirements, as the Lyapunov function condition.

Research on Prediction of Traffic Congestion State

This method of prediction using the data mining to analyze huge amounts of data as a preferred tool has been widely used in various fields. In the midst of it, the routine traffic data exists in a large number of isolated data in real time without establishing relationships with other data, and detects the amount of data which is greater than that at present. The usage of these data which is relatively shallow requires an in-depth analysis of its data model. Therefore, this paper uses a fuzzy clustering analysis method of feature points to study the traffic flow, uses a Markov decision chain model to study traffic jams, uses quantitative sample points based on the information entropy to calculate traffic flow trends and uses a heuristic prediction model to predict the road con-gestion. Through the simulation experiment which verifies the correctness of the model, this research is to advance the development of the road and to provide a basis for a dredging plan.