Markov Reward Processes Research Articles

Dynamic optimisation of unbalanced distribution network management by model predictive control with Markov reward processes

In this work, a two-level control system is used to minimize the total active power losses of an active distribution system connected to the external grid and composed of a wind turbine, two photovoltaic power sources, and two batteries. At the first control level, a model-based predictive control (MPC) is run, using non-homogeneous Markov reward models for wind power prediction and homogeneous Markov reward models for photovoltaic power. At the second level, an algorithm is run for optimal management of voltage control assets, such as voltage regulating transformers, to minimize losses. Different scenarios have been considered, highlighting the advantages of using an MPC framework. This results in an optimization process that can be influenced by different time horizons depending on whether or not the MPC is applied. The predictions allow considering a long-horizon stepwise optimization process that leads to an increasing number of variables along with the decrease of total active power losses. When the MPC is not applied, a short-horizon analysis is performed with a decrease in both the number of variables and the quality of the results. Different cases are considered in which the nominal power of a photovoltaic unit and the battery capacity are modified.

A tutorial introduction to reinforcement learning

In this paper, we present a brief survey of reinforcement learning, with particular emphasis on stochastic approximation (SA) as a unifying theme. The scope of the paper includes Markov reward processes, Markov decision processes, SA algorithms, and widely used algorithms such as temporal difference learning and Q-learning.

VPint: value propagation-based spatial interpolation

Given the common problem of missing data in real-world applications from various fields, such as remote sensing, ecology and meteorology, the interpolation of missing spatial and spatio-temporal data can be of tremendous value. Existing methods for spatial interpolation, most notably Gaussian processes and spatial autoregressive models, tend to suffer from (a) a trade-off between modelling local or global spatial interaction, (b) the assumption there is only one possible path between two points, and (c) the assumption of homogeneity of intermediate locations between points. Addressing these issues, we propose a value propagation-based spatial interpolation method called VPint, inspired by Markov reward processes (MRPs), and introduce two variants thereof: (i) a static discount (SD-MRP) and (ii) a data-driven weight prediction (WP-MRP) variant. Both these interpolation variants operate locally, while implicitly accounting for global spatial relationships in the entire system through recursion. We evaluated our proposed methods by comparing the mean absolute error, root mean squared error, peak signal-to-noise ratio and structural similarity of interpolated grid cells to those of 8 common baselines. Our analysis involved detailed experiments on a synthetic and two real-world datasets, as well as experiments on convergence and scalability. Empirical results demonstrate the competitive advantage of VPint on randomly missing data, where it performed better than baselines in terms of mean absolute error and structural similarity, as well as spatially clustered missing data, where it performed best on 2 out of 3 datasets.

A Markov process approach to untangling intention versus execution in tennis

Abstract Value functions are used in sports to determine the optimal action players should employ. However, most literature implicitly assumes that players can perform the prescribed action with known and fixed probability of success. The effect of varying this probability or, equivalently, “execution error” in implementing an action (e.g., hitting a tennis ball to a specific location on the court) on the design of optimal strategies, has received limited attention. In this paper, we develop a novel modeling framework based on Markov reward processes and Markov decision processes to investigate how execution error impacts a player’s value function and strategy in tennis. We power our models with hundreds of millions of simulated tennis shots with 3D ball and 2D player tracking data. We find that optimal shot selection strategies in tennis become more conservative as execution error grows, and that having perfect execution with the empirical shot selection strategy is roughly equivalent to choosing one or two optimal shots with average execution error. We find that execution error on backhand shots is more costly than on forehand shots, and that optimal shot selection on a serve return is more valuable than on any other shot, over all values of execution error.

A New Theoretical Framework of Pyramid Markov Processes for Blockchain Selfish Mining

In this paper, we provide a new theoretical framework of pyramid Markov processes to solve some open and fundamental problems of blockchain selfish mining under a rigorous mathematical setting. We first describe a more general model of blockchain selfish mining with both a two-block leading competitive criterion and a new economic incentive mechanism. Then we establish a pyramid Markov process and show that it is irreducible and positive recurrent, and its stationary probability vector is matrix-geometric with an explicitly representable rate matrix. Also, we use the stationary probability vector to study the influence of orphan blocks on the waste of computing resource. Next, we set up a pyramid Markov reward process to investigate the long-run average mining profits of the honest and dishonest mining pools, respectively. As a by-product, we build one-dimensional Markov reward processes and provide some new interesting interpretation on the Markov chain and the revenue analysis reported in the seminal work by Eyal and Sirer (2014). Note that the pyramid Markov (reward) processes can open up a new avenue in the study of blockchain selfish mining. Thus we hope that the methodology and results developed in this paper shed light on the blockchain selfish mining such that a series of promising research can be developed potentially.

Loop Estimator for Discounted Values in Markov Reward Processes

At the working heart of policy iteration algorithms commonly used and studied in the discounted setting of reinforcement learning, the policy evaluation step estimates the value of states with samples from a Markov reward process induced by following a Markov policy in a Markov decision process. We propose a simple and efficient estimator called loop estimator that exploits the regenerative structure of Markov reward processes without explicitly estimating a full model. Our method enjoys a space complexity of O(1) when estimating the value of a single positive recurrent state s unlike TD with O(S) or model-based methods with O(S^2). Moreover, the regenerative structure enables us to show, without relying on the generative model approach, that the estimator has an instance-dependent convergence rate of O~(\sqrt{\tau_s/T}) over steps T on a single sample path, where \tau_s is the maximal expected hitting time to state s. In preliminary numerical experiments, the loop estimator outperforms model-free methods, such as TD(k), and is competitive with the model-based estimator.

A Finite Time Analysis of Temporal Difference Learning with Linear Function Approximation

Temporal difference learning (TD) is a simple iterative algorithm widely used for policy evaluation in Markov reward processes. Bhandari et al. prove finite time convergence rates for TD learning with linear function approximation. The analysis follows using a key insight that establishes rigorous connections between TD updates and those of online gradient descent. In a model where observations are corrupted by i.i.d. noise, convergence results for TD follow by essentially mirroring the analysis for online gradient descent. Using an information-theoretic technique, the authors also provide results for the case when TD is applied to a single Markovian data stream where the algorithm’s updates can be severely biased. Their analysis seamlessly extends to the study of TD learning with eligibility traces and Q-learning for high-dimensional optimal stopping problems.

Instance-Dependent ℓ∞-Bounds for Policy Evaluation in Tabular Reinforcement Learning

Markov reward processes (MRPs) are used to model stochastic phenomena arising in operations research, control engineering, robotics, and artificial intelligence, as well as communication and transportation networks. In many of these cases, such as in the policy evaluation problem encountered in reinforcement learning, the goal is to estimate the long-term value function of such a process without access to the underlying population transition and reward functions. Working with samples generated under the synchronous model, we study the problem of estimating the value function of an infinite-horizon discounted MRP with finite state space in the ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -norm. We analyze both the standard plug-in approach to this problem and a more robust variant, and establish non-asymptotic bounds that depend on the (unknown) problem instance, as well as data-dependent bounds that can be evaluated based on the observations of state-transitions and rewards. We show that these approaches are minimax-optimal up to constant factors over natural sub-classes of MRPs. Our analysis makes use of a leave-one-out decoupling argument tailored to the policy evaluation problem, one which may be of independent interest.

Incremental Improvements of Heuristic Policies for Average-Reward Markov Decision Processes

Abstract Within the realm of Discrete Event Systems (DES) theory, the problem of performance optimization for many applications can be modeled as an infinite-horizon, average-reward Markov Decision Process (MDP) with a finite state space. In principle, these MDPs can be solved by various well-developed methods like value iteration, policy iteration and linear programming. But in reality, the tractability of these methods in the context of the aforementioned applications is compromised by the explosive size of the underlying state spaces, a problem that is known as “the curse of dimensionality”. Hence, the corresponding performance optimization problems are frequently addressed by heuristic control policies. The considered work uses results from (i) the sensitivity analysis of Markov reward processes and (ii) the ranking & selection theory in statistics in order to develop a methodology for assessing the optimality of isolated decisions in the context of any well-defined heuristic control policy for the aforementioned MDPs. It also determines an improved decision when the current one is found to be suboptimal. Hence, when embedded in an iterative scheme, this methodology can support the incremental enhancement of the original heuristic policy in a way that controls, both, the computational and also the representational complexity of the new policy. Finally, an additional important feature of the presented methodology is that it can be executed either in an “off-line” mode, using a simulation of the dynamics of the underlying DES, or in an “on-line” mode, based on the sample path that is defined by the real-time dynamics of the controlled system.

State-Augmentation Transformations for Risk-Sensitive Reinforcement Learning

In the framework of MDP, although the general reward function takes three arguments—current state, action, and successor state; it is often simplified to a function of two arguments—current state and action. The former is called a transition-based reward function, whereas the latter is called a state-based reward function. When the objective involves the expected total reward only, this simplification works perfectly. However, when the objective is risk-sensitive, this simplification leads to an incorrect value. We propose three successively more general state-augmentation transformations (SATs), which preserve the reward sequences as well as the reward distributions and the optimal policy in risk-sensitive reinforcement learning. In risk-sensitive scenarios, firstly we prove that, for every MDP with a stochastic transition-based reward function, there exists an MDP with a deterministic state-based reward function, such that for any given (randomized) policy for the first MDP, there exists a corresponding policy for the second MDP, such that both Markov reward processes share the same reward sequence. Secondly we illustrate that two situations require the proposed SATs in an inventory control problem. One could be using Q-learning (or other learning methods) on MDPs with transition-based reward functions, and the other could be using methods, which are for the Markov processes with a deterministic state-based reward functions, on the Markov processes with general reward functions. We show the advantage of the SATs by considering Value-at-Risk as an example, which is a risk measure on the reward distribution instead of the measures (such as mean and variance) of the distribution. We illustrate the error in the reward distribution estimation from the reward simplification, and show how the SATs enable a variance formula to work on Markov processes with general reward functions.

Real-time management of complex resource allocation systems: Necessity, achievements and further challenges

Many contemporary applications, ranging from flexibly automated production systems, to automated material handling and intelligent transportation systems, to internet-based workflow management systems, and more recently, to the massively parallelized software systems that emerge in the context of the novel multi-core computing architectures, can be perceived as a set of finite resources that support a number of concurrently running processes. These processes execute in a staged manner and, at each stage, they vie for the allocation of various subsets of the system resources. To effectively support and manage the extensive levels of concurrency and operational flexibility that are contemplated for these environments, and the ensuing complexity, there is a substantial need for formal models and tools that will enable the modeling, analysis and eventually the control of the aforementioned resource allocation function so that the resulting dynamics are, both, behaviorally correct and operationally efficient. This article overviews a research program that seeks to address the aforementioned need by using the unifying abstraction of the resource allocation system (RAS) and supporting modeling frameworks, like automata, Petri nets, and Markov reward and decision processes, borrowed from the area of Discrete Event Systems (DES) theory. The presented results take advantage of the special structure that exists in the considered RAS classes, and they are characterized by their analytical rigor and computational tractability. The article also highlights the further challenges that must be addressed for the successful completion and promotion of the pursued framework.

THE BIAS OPTIMAL K IN THE M/M/1/K QUEUE: AN APPLICATION OF THE DEVIATION MATRIX

We study the optimal buffer capacity K for the M/M/1/K queue under some standard cost and reward structures by comparing various Markov reward processes. Using explicit expressions for the deviation matrix of the underlying Markov chains, we find the bias optimal value for K in the case of a tie between two consecutive optimal gain policies. We show that the bias optimal value depends both on whether the reward is granted upon arrival or departure of the customers, and on the initial queue size. Moreover, we demonstrate that in some specific cases the optimal policy is threshold-based with respect to the initial queue size.

Markov counting and reward processes for analysing the performance of a complex system subject to random inspections

In this paper, a discrete complex reliability system subject to internal failures and external shocks, is modelled algorithmically. Two types of internal failure are considered: repairable and non-repairable. When a repairable failure occurs, the unit goes to corrective repair. In addition, the unit is subject to external shocks that may produce an aggravation of the internal degradation level, cumulative damage or extreme failure. When a damage threshold is reached, the unit must be removed. When a non-repairable failure occurs, the device is replaced by a new, identical one. The internal performance and the external damage are partitioned in performance levels. Random inspections are carried out. When an inspection takes place, the internal performance of the system and the damage caused by external shocks are observed and if necessary the unit is sent to preventive maintenance. If the inspection observes minor state for the internal performance and/or external damage, then these states remain in memory when the unit goes to corrective or preventive maintenance. Transient and stationary analyses are performed. Markov counting and reward processes are developed in computational form to analyse the performance and profitability of the system with and without preventive maintenance. These aspects are implemented computationally with Matlab.

Real-Time Management of Complex Resource Allocation Systems: Necessity, Achievements and Further Challenges

Abstract: Many contemporary applications, ranging from flexibly automated production systems, to automated material handling and intelligent transportation systems, to internetbased workflow management systems, and more recently, to the massively parallelized software systems that emerge in the context of the novel multi-core computing architectures, can be perceived as a set of finite resources that support a number of concurrently running processes; these processes execute in a staged manner and vie for the allocation of various subsets of the system resources. To effectively support and manage the extensive levels of concurrency and operational flexibility that are contemplated for these environments, and the ensuing complexity, there is a substantial need for formal models and tools that will enable the modeling, analysis and eventually the control of the aforementioned resource allocation function so that the resulting dynamics are, both, behaviorally correct and operationally efficient. This write-up overviews a research program that seeks to address the aforementioned need by using the unifying abstraction of the resource allocation system (RAS) and supporting modeling frameworks, like automata, Petri nets, and Markov reward and decision processes, borrowed from the area of Discrete Event Systems (DES) theory. The presented results take advantage of the special structure that exists in the considered RAS classes, and are further characterized by, both, their analytical rigor and computational tractability. The write-up also highlights the further challenges that must be addressed for the successful completion and promotion of the pursued framework.

Discrete Time Homogeneous Markov Processes for the Study of the Basic Risk Processes

In this paper Markov models useful for following the time evolution of the aggregate claim amount and the claim number in the homogeneous time environment are presented. More precisely the homogeneous Markov reward processes in both discounted and not discounted cases are applied to solve the aggregate claim amount and the claim number processes respectively. In the last section the application of the proposed models is presented. Two different real-world databases are mixed for the construction of input data.

Performance optimization for a class of generalized stochastic Petri nets

This paper considers the problem of optimizing the (long-term) performance of operations that are modeled by Generalized Stochastic Petri nets. The proposed methodology employs the representational power of the GSPN framework in order to articulate an explicit trade-off between the computational tractability of the formulated problem and the operational efficiency of the derived solutions. On the other hand, the solution of the considered formulations is based on recent results regarding the sensitivity analysis of Markov reward processes. A more expansive treatment of the presented results, together with a case study that highlights the relevance of the considered problem and the efficacy of the proposed methodology, can be found in a companion document that is accessible from the website of the second author.

Analytical Models and Performance Evaluation of Drive-thru Internet Systems

Drive-thru Internet systems are multiple-access wireless networks in which users in moving vehicles can connect to a roadside access point (AP) to obtain Internet connectivity for some period of time as the vehicles pass through the AP's coverage range. In order to evaluate the type of communication services and the quality-of-service that these systems can provide, in this paper, we investigate the data communication performance of a vehicle in Drive-thru Internet systems. In particular, we derive analytical models with tractable solutions to characterize the average and the distribution of the number of bytes downloaded by a vehicle by the end of its sojourn through an AP's coverage range, in the presence of other vehicles contending for the same AP's resources. Our models are able to quantify the impact of road traffic density, vehicle speed, service penetration rate, AP's transmission range and the corresponding bit rate, on the amount of data downloaded by an individual vehicle. In terms of analysis technique, we map the study of our vehicular data downloading process into the transient analysis of a series of Markov reward processes. Our use of Markov reward model is novel in the sense that we only select from the corresponding Markov chain, a subset of relevant sample paths that matches the required behavior of our vehicular flow model. We also validate our proposed analytical models through extensive simulations, driven by empirical vehicular traffic traces. We believe our work offers a unique analytical framework based on which the interplay between vehicular traffic parameters and a vehicle's data communication performance in a Drive-thru Internet system can be studied and optimized in a systematic, quantitative manner.

Discrete Time Markov Reward Processes a Motor Car Insurance Example

In this paper, a full treatment of homogeneous discrete time Markov reward processes is presented. The higher order moments of the homogeneous reward process are determined. In the last part of the paper, an application to the bonus-malus car insurance is presented. The application was constructed using real data.

Measuring the quality of life through Markov reward processes: Analysis and inference

AbstractIn this paper, the time evolution of the quality of life of a person is studied using Markov chains. A new index measuring the quality of life is proposed. It is based on Markov reward processes which turn out to be a useful tool in the modelling of the accumulated quality of life. Later on, a nonparametric estimator of the index is presented. Its asymptotic properties are established and as a consequence of the asymptotic normality the confidence sets are obtained. A numerical example illustrates the results. Copyright © 2009 John Wiley &amp; Sons, Ltd.

A preorder relation for Markov reward processes

A new preorder relation is introduced that orders states of a Markov process with an additional reward structure according to the reward gained over any interval of finite or infinite length. The relation allows the comparison of different Markov processes and includes as special cases monotone and lumpable Markov processes.