Discrete Time Stochastic Hybrid Systems Research Articles

Energy Management of a Building Cooling System With Thermal Storage: An Approximate Dynamic Programming Solution

This paper concerns the design of an energy management system for a building cooling system that includes a chiller plant (with two or more chiller units), a thermal storage unit, and a cooling load. The latter is modeled in a probabilistic framework to account for the uncertainty in the building occupancy. The energy management task essentially consists in the minimization of the energy consumption of the cooling system, while preserving comfort in the building. This is achieved by a twofold strategy. The cooling power request is optimally distributed among the chillers and the thermal storage unit. At the same time, a slight modulation of the temperature set-point of the zone is allowed, trading energy saving for comfort. The problem can be decoupled into a static optimization problem (mainly addressing the chiller plant optimization) and a dynamic programming (DP) problem for a discrete time stochastic hybrid system (SHS) that takes care of the overall energy minimization. The DP problem is solved by abstracting the SHS to a (finite) controlled Markov chain, where costs associated with state transitions are computed by simulating the original model and determining the corresponding energy consumption. A numerical example shows the efficacy of the approach.

Reachability for partially observable discrete time stochastic hybrid systems

When designing optimal controllers for any system, it is often the case that the true state of the system is unknown to the controller. Imperfect state information must be taken into account in the controller’s design in order to preserve its optimality. The same is true when performing reachability calculations. To estimate the probability that the state of a stochastic system reaches, or stays within, some set of interest in a given time horizon, it is necessary to find a controller that drives the system to that set with maximum probability, given the controller’s knowledge of the true state of the system. To date, little work has been done on stochastic reachability calculations with partially observable states. The work that has been done relies on converting the reachability optimization problem to one with an additive cost function, for which theoretical results are well known. Our approach is to preserve the multiplicative cost structure when deriving a sufficient statistic that reduces the problem to one of perfect state information. Our transformation includes a change of measure that simplifies the distribution of the sufficient statistic conditioned on its previous value. We develop a dynamic programming recursion for the solution of the equivalent perfect information problem, proving that the recursion is valid, an optimal solution exists, and results in the same solution as to the original problem. We also show that our results are equivalent to those for the reformulated additive cost problem, and so such a reformulation is not required.

A stochastic games framework for verification and control of discrete time stochastic hybrid systems

We describe a framework for analyzing probabilistic reachability and safety problems for discrete time stochastic hybrid systems within a dynamic games setting. In particular, we consider finite horizon zero-sum stochastic games in which a control has the objective of reaching a target set while avoiding an unsafe set in the hybrid state space, and a rational adversary has the opposing objective. We derive an algorithm for computing the maximal probability of achieving the control objective, subject to the worst-case adversary behavior. From this algorithm, sufficient conditions of optimality are also derived for the synthesis of optimal control policies and worst-case disturbance strategies. These results are then specialized to the safety problem, in which the control objective is to remain within a safe set. We illustrate our modeling framework and computational approach using both a tutorial example with jump Markov dynamics and a practical application in the domain of air traffic management.

Probabilistic-constrained optimal control of a class of stochastic hybrid systems

Stochastic hybrid systems have several applications such as biological systems and communication networks, but it is difficult to consider control of general stochastic hybrid systems. In this paper, a class of discrete-time stochastic hybrid systems, in which only discrete dynamics are stochastic, is considered. For this system, a solution method for the optimal control problem with probabilistic constraints is proposed. Probabilistic constraints guarantee that the probability that the continuous state reaches a given unsafe region is less than a given constant. In the propose method, first, continuous state regions, from which the state reaches a given unsafe region, are computed by a backward-reachability graph. Next, mixed integer quadratic programming problems with constraints derived from the backward-reachability graph are solved. The proposed method can be applied to model predictive control.

Hybrid power control on a wireless networked control system

We use a discrete-time stochastic hybrid system (dtSHS) to model a networked control system where measures and control signals are exchanged via a single-hop wireless communication channel affected by slow fading. We apply an abstraction algorithm to approximate the infinite dimensional dtSHS as a finite dimensional Markov Decision Process (MDP), and combine standard optimization methods over MDPs and stochastic model checking algorithms to perform power control in order to minimize the energy consumption while satisfying probabilistic safety constraints. We use error bounds on the approximation error induced by the abstraction to validate the effectiveness of the control policy designed over the MDP, when applied to the original dtSHS model.

Approximate Abstractions of Stochastic Hybrid Systems

We present a constructive procedure for obtaining a finite approximate abstraction of a discrete-time stochastic hybrid system. The procedure consists of a partition of the state space of the system and depends on a controllable parameter. Given proper continuity assumptions on the model, the approximation errors introduced by the abstraction procedure are explicitly computed and it is shown that they can be tuned through the parameter of the partition. The abstraction is interpreted as a Markov set-Chain. We show that the enforcement of certain ergodic properties on the stochastic hybrid model implies the existence of a finite abstraction with finite error in time over the concrete model, and allows introducing a finite-time algorithm that computes the abstraction.

A two-step scheme for approximate model checking of stochastic hybrid systems

AbstractIn this paper we describe a two-step scheme for approximate model checking of discrete time stochastic hybrid systems. In the first step, the stochastic hybrid system is approximated by a finite state Markov chain. In the second step, the Markov chain is model checked for the desired property. In particular, we consider the probabilistic invariance property and show that, under certain regularity conditions, the invariance probability computed using the approximating Markov chain converges to the invariance probability of the original stochastic hybrid system, as the grid used in the approximation gets finer. A bound on the convergence rate is also provided.

An Excursion-Theoretic Approach to Stability of Discrete-Time Stochastic Hybrid Systems

We address stability of a class of Markovian discrete-time stochastic hybrid systems. This class of systems is characterized by the state-space of the system being partitioned into a safe or target set and its exterior, and the dynamics of the system being different in each domain. We give conditions for L 1-boundedness of Lyapunov functions based on certain negative drift conditions outside the target set, together with some more minor assumptions. We then apply our results to a wide class of randomly switched systems (or iterated function systems), for which we give conditions for global asymptotic stability almost surely and in L 1. The systems need not be time-homogeneous, and our results apply to certain systems for which functional-analytic or martingale-based estimates are difficult or impossible to get.

Verification of discrete time stochastic hybrid systems: A stochastic reach-avoid decision problem

We present a dynamic programming based solution to a probabilistic reach-avoid problem for a controlled discrete time stochastic hybrid system. We address two distinct interpretations of the reach-avoid problem via stochastic optimal control. In the first case, a sum-multiplicative cost function is introduced along with a corresponding dynamic recursion which quantifies the probability of hitting a target set at some point during a finite time horizon, while avoiding an unsafe set during each time step preceding the target hitting time. In the second case, we introduce a multiplicative cost function and a dynamic recursion which quantifies the probability of hitting a target set at the terminal time, while avoiding an unsafe set during the preceding time steps. In each case, optimal reach while avoid control policies are derived as the solution to an optimal control problem via dynamic programming. Computational examples motivated by two practical problems in the management of fisheries and finance are provided.

Approximate Model Checking of Stochastic Hybrid Systems

A method for approximate model checking of stochastic hybrid systems with provable approximation guarantees is proposed. We focus on the probabilistic invariance problem for discrete time stochastic hybrid systems and propose a two-step scheme. The stochastic hybrid system is first approximated by a finite state Markov chain. The approximating chain is then model checked for probabilistic invariance. Under certain regularity conditions on the transition and reset kernels governing the dynamics of the stochastic hybrid system, the invariance probability computed using the approximating Markov chain is shown to converge to the invariance probability of the original stochastic hybrid system, as the grid used in the approximation gets finer. A bound on the convergence rate is also provided. The performance of the two-step approximate model checking procedure is assessed on a case study of a multi-room heating system.

A Probabilistic Reach-Avoid Problem for Controlled Discrete Time Stochastic Hybrid Systems

Abstract In this paper we present a dynamic programming based solution to a probabilistic reach-avoid problem for a controlled discrete time stochastic hybrid system. Similar to recent results related to the problem of probabilistic safety and stochastic target hitting, we introduce a sum-multiplicative cost function and a dynamic recursion that quantifies the probability of hitting a target set while avoiding an unsafe set. Optimal and worst-case reach while avoid control policies are introduced as the solution to an optimal control problem via dynamic programming. A computational example is provided.

Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems

In this work, probabilistic reachability over a finite horizon is investigated for a class of discrete time stochastic hybrid systems with control inputs. A suitable embedding of the reachability problem in a stochastic control framework reveals that it is amenable to two complementary interpretations, leading to dual algorithms for reachability computations. In particular, the set of initial conditions providing a certain probabilistic guarantee that the system will keep evolving within a desired ‘safe’ region of the state space is characterized in terms of a value function, and ‘maximally safe’ Markov policies are determined via dynamic programming. These results are of interest not only for safety analysis and design, but also for solving those regulation and stabilization problems that can be reinterpreted as safety problems. The temperature regulation problem presented in the paper as a case study is one such case.

Exact Bayesian and particle filtering of stochastic hybrid systems

The standard way of applying particle filtering to stochastic hybrid systems is to make use of hybrid particles, where each particle consists of two components, one assuming Euclidean values, and the other assuming discrete mode values. This paper develops a novel particle filter (PF) for a discrete-time stochastic hybrid system. The novelty lies in the use of the exact Bayesian equations for the conditional mode probabilities given the observations. Therefore particles are needed for the Euclidean valued state component only. The novel particle filter is referred to as the interacting multiple model (IMM) particle filter (IMMPF) because it incorporates a filter step which is of the same form as the interaction step of the IMM algorithm. Through Monte Carlo simulations, it is shown that the IMMPF has significant advantage over the standard PF, in particular for situations where conditional switching rate or conditional mode probabilities have small values