Parametric Markov Decision Processes Research Articles

Convex Optimization for Parameter Synthesis in MDPs

Probabilistic model-checking aims to prove whether a Markov decision process (MDP) satisfies a temporal logic specification. The underlying methods rely on an often unrealistic assumption that the MDP is precisely known. Consequently, parametric MDPs (pMDPs) extend MDPs with transition probabilities that are functions over unspecified parameters. The parameter synthesis problem is to compute an instantiation of these unspecified parameters such that the resulting MDP satisfies the temporal logic specification. We formulate the parameter synthesis problem as a quadratically constrained quadratic program, which is nonconvex and is NP-hard to solve in general. We develop two approaches that iteratively obtain locally optimal solutions. The first approach exploits the so-called convex–concave procedure (CCP), and the second approach utilizes a sequential convex programming (SCP) method. The techniques improve the runtime and scalability by multiple orders of magnitude compared to black-box CCP and SCP by merging ideas from convex optimization and probabilistic model-checking. We demonstrate the approaches on a satellite collision avoidance problem with hundreds of thousands of states and tens of thousands of parameters and their scalability on a wide range of commonly used benchmarks.

Scenario-based verification of uncertain parametric MDPs

We consider parametric Markov decision processes (pMDPs) that are augmented with unknown probability distributions over parameter values. The problem is to compute the probability to satisfy a temporal logic specification with any concrete MDP that corresponds to a sample from these distributions. As solving this problem precisely is infeasible, we resort to sampling techniques that exploit the so-called scenario approach. Based on a finite number of samples of the parameters, the proposed method yields high-confidence bounds on the probability of satisfying the specification. The number of samples required to obtain a high confidence on these bounds is independent of the number of states and the number of random parameters. Experiments on a large set of benchmarks show that several thousand samples suffice to obtain tight and high-confidence lower and upper bounds on the satisfaction probability.

Verifiable and Compositional Reinforcement Learning Systems

We propose a framework for verifiable and compositional reinforcement learning (RL) in which a collection of RL subsystems, each of which learns to accomplish a separate subtask, are composed to achieve an overall task. The framework consists of a high-level model, represented as a parametric Markov decision process (pMDP) which is used to plan and to analyze compositions of subsystems, and of the collection of low-level subsystems themselves. By defining interfaces between the subsystems, the framework enables automatic decompositions of task specifications, e.g., reach a target set of states with a probability of at least 0.95, into individual subtask specifications, i.e. achieve the subsystem's exit conditions with at least some minimum probability, given that its entry conditions are met. This in turn allows for the independent training and testing of the subsystems; if they each learn a policy satisfying the appropriate subtask specification, then their composition is guaranteed to satisfy the overall task specification. Conversely, if the subtask specifications cannot all be satisfied by the learned policies, we present a method, formulated as the problem of finding an optimal set of parameters in the pMDP, to automatically update the subtask specifications to account for the observed shortcomings. The result is an iterative procedure for defining subtask specifications, and for training the subsystems to meet them. As an additional benefit, this procedure allows for particularly challenging or important components of an overall task to be identified automatically, and focused on, during training. Experimental results demonstrate the presented framework's novel capabilities in both discrete and continuous RL settings. A collection of RL subsystems are trained, using proximal policy optimization algorithms, to navigate different portions of a labyrinth environment. A cross-labyrinth task specification is then decomposed into subtask specifications. Challenging portions of the labyrinth are automatically avoided if their corresponding subsystems cannot learn satisfactory policies within allowed training budgets. Unnecessary subsystems are not trained at all. The result is a compositional RL system that efficiently learns to satisfy task specifications.

Resilient monitoring in self-adaptive systems through behavioral parameter estimation

Cyber-physical systems need self-adaptation as a mean to autonomously deal with changes. For runtime adaptation, a cyber-physical system repeatedly monitors the environment for detecting possible changes. Faults in the monitoring devices due to the dynamic and uncertain environment is very likely, necessitating resilient monitoring. In this paper, we discuss imperfect monitoring in self-adaptive systems, and propose a model-driven methodology to represent the self-adaptive system using a parametric Markov decision process, where the changes are reflected by a set of model parameters. Fault in the monitoring device may result in some parameter valuation miss. We propose a comprehensive framework for parameter estimation using behavioral patterns of the system implemented by a pattern-matching component. The proposed method simulates the current behavior of the system using random walk patterns, and matches it to a history of patterns for estimating the valuation of the omitted data. The results show that the accuracy of the framework is evaluated about 96% under imperfect monitoring. In addition, we elaborate a set of theoretical proofs to support rigorous error analysis, and determine a certain upper-bound of errors to guarantee an accurate decision-making outcome. We establish a logical connection between the error and the accuracy of decisions, and introduce tolerable error metric that is used to guarantee the accuracy of decisions under estimation.

The complexity of reachability in parametric Markov decision processes

This article presents the complexity of reachability decision problems for parametric Markov decision processes (pMDPs), an extension to Markov decision processes (MDPs) where transitions probabilities are described by polynomials over a finite set of parameters. In particular, we study the complexity of finding values for these parameters such that the induced MDP satisfies some maximal or minimal reachability probability constraints. We discuss different variants depending on the comparison operator in the constraints and the domain of the parameter values. We improve all known lower bounds for this problem, and notably provide ETR-completeness results for distinct variants of this problem.

Probabilistic approximation of runtime quantitative verification in self-adaptive systems

Cyber-physical systems (CPS) are expected to continuously monitor the physical components to autonomously calculate appropriate runtime reactions to deal with the uncertain environmental conditions. Self-adaptation, as a promising concept to fulfill a set of provable rules, majorly needs runtime quantitative verification (RQV). Taking a few probabilistic variables into account to represent the uncertainties, the system configuration will be extremely large. Thus, efficient approaches are needed to reduce the model state-space, preferably with certain bounds on the approximation error. In this paper, we propose an approximation framework to efficiently approximate the entire model of a self-adaptive system. We split up the large model into strongly-connected components (SCCs), apply the approximation algorithm separately on each SCC, and integrate the result of each part using a centralized algorithm. Due to a number of changes in probabilistic variables, it is not possible to use static models. Addressing this issue, we have deployed parametric Markov decision process. In order to apply approximation on the model, the notion of ε-approximate probabilistic bisimulation is utilized that introduces the approximation level ε. We show that our approximation framework offers a certain error bound on each level of approximation. Then, we denote that the approximation framework is appropriate to be applied in decision-making process of self-adaptive systems where the models are relatively large. The results reveal that we can achieve up to 50% size reduction in the approximate model while maintaining the accuracy about 95%. In addition, we discuss about the trade-off between efficiency and accuracy of our approximation framework.

SEA-PARAM: Exploring Schedulers in Parametric MDPs

We study parametric Markov decision processes (PMDPs) and their reachability probabilities "independent" of the parameters. Different to existing work on parameter synthesis (implemented in the tools PARAM and PRISM), our main focus is on describing different types of optimal deterministic memoryless schedulers for the whole parameter range. We implement a simple prototype tool SEA-PARAM that computes these optimal schedulers and show experimental results.

Operational versus weakest pre-expectation semantics for the probabilistic guarded command language

This paper proposes a simple operational semantics of pGCL, Dijkstra’s guarded command language extended with probabilistic choice, and relates this to pGCL’s wp-semantics by McIver and Morgan. Parametric Markov decision processes whose state rewards depend on the post-expectation at hand are used as the operational model. We show that the weakest pre-expectation of a pGCL-program w.r.t. a post-expectation corresponds to the expected cumulative reward to reach a terminal state in the parametric MDP associated to the program. In a similar way, we show a correspondence between weakest liberal pre-expectations and liberal expected cumulative rewards. The verification of probabilistic programs using wp-semantics and operational semantics is illustrated using a simple running example.

A variance minimization problem for a Markov decision process

In the steady state of a discrete time Markov decision process, we consider the problem to find an optimal randomized policy that minimizes the variance of the reward in a transition among the policies which give the mean not less than a specified value. The problem is solved by introducing a parametric Markov decision process with average cost criterion. It is shown that there exists an optimal policy which is a mixture of at most two pure policies. As an application, the toymaker's problem is discussed.

分数制約をもつマルコフ決定過程

The problem/'considered for a Markov decision process is to find an optimal randomized policy that maximizes the expected reward in a transition in the steady state among the policies which secure the variance not larger than a specified value. A solution method by a parametric Markov decision process is developed. An optimal policy is shown to be a mixture of at most two pure policies. As an application, a sequential replacement problem of a Markovian deterioration system is discussed.