Abstract
Accurate Modelling of a real-world system with probabilistic behaviour is a difficult task. Sensor noise and statistical estimations, among other imprecisions, make the exact probability values impossible to obtain. In this article, we consider Interval Markov decision processes ( IMDP s), which generalise classical MDP s by having interval-valued transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that prevents the knowledge of the exact transition probabilities. We investigate the problem of robust multi-objective synthesis for IMDP s and Pareto curve analysis of multi-objective queries on IMDP s. We study how to find a robust (randomised) strategy that satisfies multiple objectives involving rewards, reachability, and more general ω-regular properties against all possible resolutions of the transition probability uncertainties, as well as to generate an approximate Pareto curve providing an explicit view of the trade-offs between multiple objectives. We show that the multi-objective synthesis problem is PSPACE -hard and provide a value iteration-based decision algorithm to approximate the Pareto set of achievable points. We finally demonstrate the practical effectiveness of our proposed approaches by applying them on several case studies using a prototype tool.
Highlights
Interval Markov Decision Processes (IMDPs) [Givan et al 2000] extend classical Markov Decision Processes (MDPs) [Bellman 1957] by including uncertainty over the transition probabilities
27:25 control of IMDPs where our goal is to generate an approximation of the Pareto curve for synthesis, quantitative, and Pareto queries
The core part of our approach to approximate Pareto curves of the multi-objective queries was to optimise the weighted sum of objectives, which was in turn achieved through a value iteration algorithm
Summary
Interval Markov Decision Processes (IMDPs) [Givan et al 2000] extend classical Markov Decision Processes (MDPs) [Bellman 1957] by including uncertainty over the transition probabilities. Concerning strategy synthesis algorithms, the works of Hahn et al [2011] and Nilim and El Ghaoui [2005] considered synthesis for parametric MDPs and MDPs with ellipsoidal uncertainty in the verification community In control community, such synthesis problems were mostly studied for uncertain Markov models in Givan et al [2000]; Nilim and El Ghaoui [2005]; Wu and Koutsoukos [2008] with the aim to maximise expected finite-horizon (un)discounted rewards. All these works, consider solely single objective properties, and their extension to multi-objective synthesis is not trivial. To keep the presentation clear, non-trivial proofs have been moved to the Appendix A
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: ACM Transactions on Modeling and Computer Simulation
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
