semi-Markov Games Research Articles

A Note on Zero-Sum Two-Person Undiscounted One Player Control Semi-Markov Games

Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games are considered where both the transition probabilities and the transition times are controlled by a fixed player in all states. We prove that, if such a game is unichain, the value and optimal stationary strategies of the players can be obtained from an optimal solution of the linear programming algorithm for the associated one player control stochastic game using a data transformation method. An example is provided for such class of games.

Multi-Agent Attention Double Actor-Critic Framework for Intelligent Traffic Light Control in Urban Scenarios With Hybrid Traffic

In real-world urban environments, hybrid and disorder traffic brings new challenges for the intelligent traffic light control system (ITLCS). Apart from coordinating traffic flows around intersections, the ITLCS is responsive to ensuring high priority vehicles pass through intersections quickly. To this end, we formulate the multiple intersections' decision-making problem as a Semi-Markov game and propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-agent attention double actor-critic (MAADAC)</i> framework to solve this game, integrating the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">options framework</i> with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">graph attention networks (GATs)</i> . Specifically, the options framework empowers agents to learn to make a long sequence of satisfactory decisions, such as keeping a reasonable phase for a short period to ensure high priority vehicles pass through intersections quickly. Besides, we adopt GATs to capture graph-structure mutual influences among agents. We set up a simulator based on real-world city road networks and conduct extensive experiments to evaluate the performance of MAADAC. The experimental results show that MAADAC can reduce high priority vehicles' waiting time in the interval of 18.16%-38.14% versus the density of vehicles in real-world urban scenarios over several state-of-the-art approaches. Also, our framework can guarantee the passing efficiency of high priority vehicles under various traffic conditions with the change in the proportion of high priority vehicles.

Double Graph Attention Actor-Critic Framework for Urban Bus-Pooling System

To unleash the power of buses, we propose a bus-pooling system that keeps the notion of bus stops and terminals but discards the concept of fixed bus lines by enabling buses to choose the next stop or terminal based on orders submitted by passengers. Each bus, unlike a taxi, must consider the additional delays experienced by the passengers already on board when deciding how to adapt its route to serve new orders. This paper treats each bus as an agent and formulates the buses’ re-routing decision-making process as a Semi-Markov game. Then, we propose a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">double graph attention actor-critic (DGAAC)</i> framework by integrating high-level and low-level actor-critics separately with graph attention networks (GATs) to solve the game. Specifically, GATs embedded in high-level and low-level critics take a large-scale graph covering a city-scale area as input and capture graph-structured mutual influences among buses. In contrast, the high-level and low-level actors equipped with GATs only take the n-hop sub-graph with local information as the input and are employed as the distributed decision module of each bus. We conduct extensive experiments on one of the largest real-world datasets in Shenzhen, China, and validate that the proposed DGAAC framework greatly outperforms all baselines.

Zero-sum semi-Markov games with state-action-dependent discount factors

Semi-Markov model is one of the most general models for stochastic dynamic systems. This paper deals with a two-person zero-sum game for semi-Markov processes. We focus on the expected discounted criterion with state-action-dependent discount factors. The state and action spaces are both Polish spaces, and the reward rate function is ω-bounded. We first construct a fairly general model of semi-Markov games under a given semi-Markov kernel and a pair of strategies. Next, based on the standard regularity condition and the continuity-compactness condition for semi-Markov games, we derive a “drift condition” on the semi-Markov kernel and suppose that the discount factors have a positive lower bound, under which the existence of the value function and an optimal pair of stationary strategies of our semi-Markov game are proved by using a general Shapley equation. Moreover, in the scenario of finite state and action spaces, a value iteration-type algorithm for approximating the value function and an optimal pair of stationary strategies is developed. The convergence and the error bound of the algorithm are also proved. Finally, we conduct numerical examples to demonstrate the main results.

Zero and non-zero sum risk-sensitive Semi-Markov games

In this article we consider zero and non-zero sum risk-sensitive average criterion games for semi-Markov processes with a finite state space. For the zero-sum case, under suitable assumptions we show that the game has a value. We also establish the existence of a stationary saddle point equilibrium. For the non-zero sum case, under suitable assumptions we establish the existence of a stationary Nash equilibrium.

Zero-sum semi-Markov games with a probability criterion

In this paper, we consider a semi-Markov zero-sum stochastic game with a general state and finite action spaces. The performance is analysed via a probability criterion. Under suitable assumptions, we establish the existence of the value of the game and also characterize it through an optimality equation. In the process, we also prescribe a saddle point equilibrium.

Discounted Semi-Markov Games and Algorithms for Solving Two Structured Classes

Two structured classes of zero-sum two-person finite (state and action spaces) semi-Markov games with discounted payoffs, namely, Additive Reward-Additive Transition and Action Independent Transition Time (AR-AT-AITT) and Additive Reward-Action Independent Transition and Additive Transition Time (AR-AIT-ATT) have been studied. We propose two practical situations of economic competition viz. petroleum game and groundwater game that suitably fit into such classes of games, respectively. Solution (value and pure stationary optimals) to such classes of games can be derived from optimal solution to appropriate bilinear programs with linear constraints. We present a stepwise generalized principal pivoting algorithm for solving the vertical linear complementarity problem (VLCP) obtained from such game problems. Moreover, a neural network dynamics is proposed for solving such structured classes. Examples are worked out to compare the usefulness of the above three algorithms.

A Policy Improvement Algorithm for Solving a Mixture Class of Perfect Information and AR-AT Semi-Markov Games

Zero-sum two-person discounted semi-Markov games with finite state and action spaces are studied where a collection of states having Perfect Information (PI) property is mixed with another collection of states having Additive Reward–Additive Transition and Action Independent Transition Time (AR-AT-AITT) property. For such a PI/AR-AT-AITT mixture class of games, we prove the existence of an optimal pure stationary strategy for each player. We develop a policy improvement algorithm for solving discounted semi-Markov decision processes (one player version of semi-Markov games) and using it we obtain a policy-improvement type algorithm for computing an optimal strategy pair of a PI/AR-AT-AITT mixture semi-Markov game. Finally, we extend our results when the states having PI property are replaced by a subclass of Switching Control (SC) states.

Finite-horizon semi-Markov game for time-sensitive attack response and probabilistic risk assessment in nuclear power plants

Cybersecurity has drawn increasing attention in the nuclear industry. To improve the cyber-security posture, it is important to develop effective methods for cyber-attack response and cyber-security risk assessment. In this research, we develop a finite-horizon semi-Markov general-sum game between the defender (i.e., plant operator) and the attacker to obtain the time-sensitive attack response strategy and the real-time risk assessment in nuclear power plants. We propose methods for identifying system states of concern to reduce the state space and for determining state transition probabilities by integrating probabilistic risk assessment techniques. After a proper discretization of the developed continuous-time model, we use dynamic programming to derive the time-varying and state-dependent strategy of the defender based on the solution concept of the mixed-strategy Nash equilibrium. For risk assessment, three risk metrics are considered, and an exact analytical algorithm and a Monte Carlo simulation-based algorithm for obtaining the metrics are developed. Both players’ strategies and the risk metrics are illustrated using a digital feedwater control system used in pressurized water reactors. The results show that the proposed method can support plant operators in timely cyber-attack response and effective risk assessment, reduce the risk, and improve the resilience of nuclear power plants to malicious cyber-attacks.

On zero-sum two-person undiscounted semi-Markov games with a multichain structure

Abstract Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games (SMGs) are considered with a general multichain structure. We derive the strategy evaluation equations for stationary strategies of the players. A relation between the payoff in the multichain SMG and that in the associated stochastic game (SG) obtained by a data-transformation is established. We prove that the multichain optimality equations (OEs) for an SMG have a solution if and only if the associated SG has optimal stationary strategies. Though the solution of the OEs may not be optimal for an SMG, we establish the significance of studying the OEs for a multichain SMG. We provide a nice example of SMGs in which one player has no optimal strategy in the stationary class but has an optimal semistationary strategy (that depends only on the initial and current state of the game). For an SMG with absorbing states, we prove that solutions in the game where all players are restricted to semistationary strategies are solutions for the unrestricted game. Finally, we prove the existence of stationary optimal strategies for unichain SMGs and conclude that the unichain condition is equivalent to require that the game satisfies some recurrence/ergodicity/weakly communicating conditions.

Completely mixed strategies for single controller unichain semi-Markov games with undiscounted payoffs

Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games are considered where the transition probabilities and the transition times are controlled by a fixed player in all states. We prove that if such a game is unichain, the value and optimal stationary strategies can be obtained from an optimal solution of a linear programming algorithm for the associated undiscounted unichain single controller stochastic game (obtained by a data transformation method). The single controller undiscounted unichain semi-Markov games have been formulated as a linear complementarity problem and solved using a stepwise principal pivoting algorithm. We provide necessary and sufficient conditions for such games to be completely mixed (i.e., every optimal stationary strategy for each player assigns a positive probability to every action in every state). Some properties analogous to completely mixed matrix games are also established in this paper.

Completely Mixed Strategies for Two Structured Classes of Semi-Markov Games, Principal Pivot Transform and Its Generalizations

In this article, we revisit the applications of principal pivot transform and its generalization for a rectangular matrix (in the context of vertical linear complementarity problem) to solve some structured classes of zero-sum two-person discounted semi-Markov games with finitely many states and actions. The single controller semi-Markov games have been formulated as a linear complementarity problem and solved using a stepwise principal pivoting algorithm. We provide a sufficient condition for such games to be completely mixed. The concept of switching controller semi-Markov games is introduced and we prove the ordered field property and the existence of stationary optimal strategies for such games. Moreover, such games are formulated as a vertical linear complementarity problem and have been solved using a stepwise generalized principal pivoting algorithm. Sufficient conditions are also given for such games to be completely mixed. For both these classes of games, some properties analogous to completely mixed matrix games, are established.

Linear Programming and Zero-Sum Two-Person Undiscounted Semi-Markov Games

In this paper, zero-sum two-person finite undiscounted (limiting average) semi-Markov games (SMGs) are considered. We prove that the solutions of the game when both players are restricted to semi-Markov strategies are solutions for the original game. In addition, we show that if one player fixes a stationary strategy, then the other player can restrict himself in solving an undiscounted semi-Markov decision process associated with that stationary strategy. The undiscounted SMGs are also studied when the transition probabilities and the transition times are controlled by a fixed player in all states. If such games are unichain, we prove that the value and optimal stationary strategies of the players can be obtained from an optimal solution of a linear programming algorithm. We propose a realistic and generalized traveling inspection model that suitably fits into the class of one player control undiscounted unichain semi-Markov games.

Ordered Field Property for Semi-Markov Games when One Player Controls Transition Probabilities and Transition Times

Two-person finite semi-Markov games (SMGs) are studied when the transition probabilities and the transition times are controlled by one player at all states. For the discounted games in this class, we prove that the ordered field property holds and there exist optimal/Nash equilibrium stationary strategies for the players. We illustrate that the zero-sum SMGs where only transition probabilities are controlled by one player, do not necessarily satisfy the ordered field property. An algorithm along with a numerical example for the discounted one player control zero-sum SMGs is given via linear programming. For the undiscounted version of such games, we exhibit with an example that if the game ceases to be unichain, an optimal stationary or Markov strategy need not exist, (though in this example of a one-player game we exhibit a semi-stationary optimal strategy/policy). Lastly, we prove that if such games are unichain, then they possess the ordered field property for the undiscounted case as well.

On discounted AR–AT semi-Markov games and its complementarity formulations

In this paper, we introduce a class of two-person finite discounted AR–AT (Additive Reward–Additive Transition) semi-Markov games (SMGs). We provide counterexamples to show that AR–AT and AR–AT–PT (Additive Reward–Additive Transition Probability and Time) SMGs do not satisfy the ordered field property. Some results on AR–AT–AITT (Additive Reward–Additive Transition and Action Independent Transition Time) and AR–AIT–ATT (Additive Reward–Action Independent Transition and Additive Transition Time) games are obtained in this paper. For the zero-sum games, we prove the ordered field property and the existence of pure stationary optimals for the players. Moreover, such games are formulated as a vertical linear complementarity problem (VLCP) and have been solved by Cottle-Dantzig’s algorithm under a mild assumption. We illustrate that the nonzero-sum case of such games do not necessarily have pure stationary equilibria. However, there exists a stationary equilibria which has at most two pure actions in each state for each player.

ORDERED FIELD PROPERTY IN A SUBCLASS OF FINITE SER-SIT SEMI-MARKOV GAMES

In this paper, we deal with a subclass of two-person finite SeR-SIT (Separable Reward-State Independent Transition) semi-Markov games which can be solved by solving a single matrix/bimatrix game under discounted as well as limiting average (undiscounted) payoff criteria. A SeR-SIT semi-Markov game does not satisfy the so-called (Archimedean) ordered field property in general. Besides, the ordered field property does not hold even for a SeR-SIT-PT (Separable Reward-State-Independent Transition Probability and Time) semi-Markov game, which is a natural version of a SeR-SIT stochastic (Markov) game. However by using an additional condition, we have shown that a subclass of finite SeR-SIT-PT semi-Markov games have the ordered field property for both discounted and undiscounted semi-Markov games with both players having state-independent stationary optimals. The ordered field property also holds for the nonzero-sum case under the same assumptions. We find a relation between the values of the discounted and the undiscounted zero-sum semi-Markov games for this modified subclass. We propose a more realistic pollution tax model for this subclass of SeR-SIT semi-Markov games than pollution tax model for SeR-SIT stochastic game. Finite step algorithms are given for the discounted and for the zero-sum undiscounted cases.

Average optimal strategies for zero-sum Markov games with poorly known payoff function on one side

We are concerned with two-person zero-sum Markov games with Borel spaces under a long-run average criterion. The payoff function is possibly unbounded and depends on a parameter which is unknown to one of the players. The parameter and the payoff function can be estimated by implementing statistical methods. Thus, our main objective is to combine such estimation procedure with a variant of the so-called vanishing discount approach to construct an average optimal pair of strategies for the game. Our results are applied to a class of zero-sum semi-Markov games.

On a Continuous Solution to the Bellman-Poisson Equation in Stochastic Games

In this note, we study the Bellman-Poisson equation associated with a class of ergodic zero-sum semi-Markov games. The main objective is to prove that this equation has a continuous solution and both players have optimal stationary strategies.

Zero-Sum Ergodic Semi-Markov Games with Weakly Continuous Transition Probabilities

Zero-sum ergodic semi-Markov games with weakly continuous transition probabilities and lower semicontinuous, possibly unbounded, payoff functions are studied. Two payoff criteria are considered: the ratio average and the time average. The main result concerns the existence of a lower semicontinuous solution to the optimality equation and its proof is based on a fixed-point argument. Moreover, it is shown that the ratio average as well as the time average payoff stochastic games have the same value. In addition, one player possesses an e-optimal stationary strategy (e>0), whereas the other has an optimal stationary strategy.

Partially observed semi-Markov zero-sum games with average payoff

We study a zero-sum partially observed semi-Markov game with average payoff on a countable state space. Under certain ergodicity conditions we show that a saddle point equilibrium exists. We achieve this by solving the corresponding average cost optimality equation using a span contraction method. The average value is shown to be the unique zero of a Lipschitz continuous function. A value iteration scheme is developed to compute the value.