Game Value Function Research Articles

Improved model-free H∞ control for batch processes via off-policy 2D game Q-learning

To eliminate the requirement of the precise model for model-based control methods, an improved modelfree H∞ control method is designed for batch processes with unknown dynamics and disturbances. Firstly, both the zero-sum game value function and the Q-function are presented as two-dimensional (2D) forms, and their relation is analyzed to obtain the model-free Bellman equation. Secondly, an on policy 2D game Q-learning method is proposed for learning the optimal gains of the designed H∞ controller. On this basis, the behavior control policy and the behavior disturbance policy are individually applied by developing an off-policy 2D game Q-learning method. Subsequently, the strict proof about the convergence and the unbiasedness of the off-policy approach are given. Finally, the simulation results of injection velocity manifest the validity and effectivity of the proposed algorithm.

N-Player impartial combinatorial games with random players

In this paper we develop a method of analyzing n-player impartial combinatorial games where n−1 players behave optimally while one of the players plays randomly. More specifically, we develop a function gr that maps impartial combinatorial games to n-tuples called game values. A game value of some impartial combinatorial game G describes the probability that any player can win assuming player r plays randomly (i.e. player r chooses games at random without strategy).After devising our game value function gr, we apply it to the analysis of three-player Chomp whereby the second player plays randomly. Our analysis gives game values for single rowed Chomp positions and two rowed Chomp positions where the bottom row has 1 or 2 elements.We next discuss three alternative methods of analysis. First we extend our game value function to support more than just a single random player. Our second extension allows the random player to choose from an arbitrary distribution over games (as opposed to uniform which is assumed in our analysis of Chomp). This second extension also allows us to model a player who makes correct decisions for “simple” games (those that are, for instance, close to an end game) while playing randomly for more complex games. Finally, the third method considers the case where the random player only makes a mistake with probability ϵ and otherwise plays strategically.We conclude with many open problems and some interesting observations on the behavior of games containing a random player.

Integral Reinforcement Learning for Linear Continuous-Time Zero-Sum Games With Completely Unknown Dynamics

In this paper, we develop an integral reinforcement learning algorithm based on policy iteration to learn online the Nash equilibrium solution for a two-player zero-sum differential game with completely unknown linear continuous-time dynamics. This algorithm is a fully model-free method solving the game algebraic Riccati equation forward in time. The developed algorithm updates value function, control and disturbance policies simultaneously. The convergence of the algorithm is demonstrated to be equivalent to Newton's method. To implement this algorithm, one critic network and two action networks are used to approximate the game value function, control and disturbance policies, respectively, and the least squares method is used to estimate the unknown parameters. The effectiveness of the developed scheme is demonstrated in the simulation by designing an H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> state feedback controller for a power system.

Internal structure of coalitions in competitive and altruistic graphical coalitional games

This paper introduces a certain graphical coalitional game where the internal topology of the coalition depends on a prescribed communication graph structure among the agents. The game Value Function is required to satisfy four Axioms of Value. These axioms make it possible to provide a refined study of coalition structures on graphs by defining a formal graphical game and by assigning a Positional Advantage, based on the Shapley value, to each agent in a coalition based on its connectivity properties within the graph. Using the Axioms of Value the graphical coalitional game can be shown to satisfy properties such as convexity, fairness, cohesiveness, and full cooperativeness. Three measures of the contributions of agents to a coalition are introduced: marginal contribution, competitive contribution, and altruistic contribution. The mathematical framework given here is used to establish results regarding the dependence of these three types of contributions on the graph topology, and changes in these contributions due to changes in graph topology. Based on these different contributions, three online sequential decision games are defined on top of the graphical coalitional game, and the stable graphs under each of these sequential decision games are studied. It is shown that the stable graphs under the objective of maximizing the marginal contribution are any connected graph. The stable graphs under the objective of maximizing the competitive contribution are the complete graph. The stable graphs under the objective of maximizing the altruistic contribution are any tree.

Neural network solution for finite-horizon H ∞ state feedback control of nonlinear systems

In this article, neural networks are used to approximately solve the finite-horizon optimal H ∞ state feedback control problem. The method is based on solving a related Hamilton–Jacobi–Isaacs equation of the corresponding finite-horizon zero-sum game. The neural network approximates the corresponding game value function on a certain domain of the state-space and results in a control computed as the output of a neural network. It is shown that the neural network approximation converges uniformly to the game-value function and the resulting controller provides closed-loop stability and bounded L 2 gain. The result is a nearly exact H ∞ feedback controller with time-varying coefficients that is solved a priori offline. The results of this article are applied to the rotational/translational actuator benchmark nonlinear control problem.

Neural network solution for finite-horizon H-infinity constrained optimal control of nonlinear systems

In this paper, neural networks are used to approximately solve the finite-horizon constrained input H-infinity state feedback control problem. The method is based on solving a related Hamilton-Jacobi-Isaacs equation of the corresponding finite-horizon zero-sum game. The game value function is approximated by a neural network with time-varying weights. It is shown that the neural network approximation converges uniformly to the game-value function and the resulting almost optimal constrained feedback controller provides closed-loop stability and bounded L 2 gain. The result is an almost optimal H-infinity feedback controller with time-varying coefficients that is solved a priori off-line. The effectiveness of the method is shown on the Rotational/Translational Actuator benchmark nonlinear control problem.

Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control

In this paper, the optimal strategies for discrete-time linear system quadratic zero-sum games related to the H-infinity optimal control problem are solved in forward time without knowing the system dynamical matrices. The idea is to solve for an action dependent value function Q ( x , u , w ) of the zero-sum game instead of solving for the state dependent value function V ( x ) which satisfies a corresponding game algebraic Riccati equation (GARE). Since the state and actions spaces are continuous, two action networks and one critic network are used that are adaptively tuned in forward time using adaptive critic methods. The result is a Q -learning approximate dynamic programming (ADP) model-free approach that solves the zero-sum game forward in time. It is shown that the critic converges to the game value function and the action networks converge to the Nash equilibrium of the game. Proofs of convergence of the algorithm are shown. It is proven that the algorithm ends up to be a model-free iterative algorithm to solve the GARE of the linear quadratic discrete-time zero-sum game. The effectiveness of this method is shown by performing an H-infinity control autopilot design for an F-16 aircraft.

Construction of the value function in a pursuit-evasion game with three pursuers and one evader

A differential pursuit-evasion game is considered with three pursuers and one evader. It is assumed that all objects (players) have simple motions and that the game takes place in a plane. The control vectors satisfy geometrical constraints and the evader has a superiority in control resources. The game time is fixed. The value functional is the distance between the evader and the nearest pursuer at the end of the game. The problem of determining the value function of the game for any possible position is solved. Three possible cases for the relative arrangement of the players at an arbitrary time are studied: “one-after-one”, “two-after-one”, “three-after-one-in-the-middle” and “three-after-one”. For each of the relative arrangements of the players a guaranteed result function is constructed. In the first three cases the function is expressed analytically. In the fourth case a piecewise-programmed construction is presented with one switchover, on the basis of which the value of the function is determined numerically. The guaranteed result function is shown to be identical with the game value function. When the initial pursuer positions are fixed in an arbitrary manner there are four game domains depending on their relative positions. The boundary between the “three-after-one-in-the-middle” domain and the “three-after-one” domain is found numerically, and the remaining boundaries are interior Nicomedean conchoids, lines and circles. Programs are written that construct singular manifolds and the value function level lines.

On functional equations in games of encounter at a prescribed instant

Functional operators, simpler than in /1–3/, are determined in connection with games of encounter at a prescribed instant. These operators or their analogs can be used to obtain many of the fundamental results in /1–3/, in particular, iteration methods can be constructed, converging to the game's value function. As one more illustration of the capabilities of the method developed in /1–3/, the operators determined below are used to give a new proof of the well-known result /4/ on the identification of the game's value function by means of the so-called main equation.

Properties of a differential game's potential

Abstract A position differential game with fixed termination instant is examined. Stability properties are investigated, consisting mainly in the formulation of necessary and sufficient conditions satisfied by the differential game's value function (potential). The infinitesimal form of the stability properties leads to differential inequalities generalizing the fundamental equation for the potential to the case when the value function is nondifferentiable. As a corollary to the necessary and sufficient conditions obtained for the differential game's potential, corresponding results are presented for the optimal control problem. The paper relies on the results from /1–7/, borders on the studies in /8–12/ and continues the work in /13–15/.

On fixed-duration dynamic games

Four methods of successive approximations are examined for a dynamic game's value function, which are used to construct the players' ε-optimal strategies. General piecewise-programmed strategies are used to prove the convergence of these methods. The paper's contents about the investigations in /1–9/ and generalize the results in /8,9/ to the case of a general dynamic system.

Applications of Mills' Differential

AbstractUsing Mills Differential a game value function ϕ, used in the Kemeny‐Morgenstern‐Thompson growth model to show the existence of solutions, is discussed. In doing so, several proofs become shorter, in addition known results are seen in a new context. Finally connections between the existence of a so‐called economic solution and the differentiability or non‐differentiability of function ϕ are analyzed.

Axiomatic Definition of the Value of a Matrix Game

Let a real function f, whose argument is a matrix A, satisfy the following axioms: 1. $f(\bar A) > f(A)$ if $\bar A > A$ ; 2. $f(\tilde A) = f(A)$ if $\tilde A$ differs from A only by a row, which is dominated by others; 3. $f( - A^T ) = - f(A)$, the index T stands for transposition; 4. $f(x) > x$ for a real number x. Then $f(A)$ is the game value function. Axioms $1 - 4$ are independent. Another similar set of axioms is given.