Interception Domain Approach to Orbital Multi-Player “Encirclement-Capture” Games: Theoretical Foundations and Solutions

We prove the existence of a value for pursuit games with state constraints. We also prove that this value is lower semicontinuous.

In this thesis, we investigate how reinforcement learning algorithms can be applied to two different types of games. The first type of games are matrix and stochastic games, where the states and actions are represented in discrete domains. In this type of games, we propose two multi-agent reinforcement learning algorithms to solve the problem of learning when each learning agent has only minimum knowledge about the underlying game and the other learning agents. We mathematically show that the proposed CLR-EMAQL algorithm converges to Nash equilibrium in games with pure Nash equilibrium. We introduce the concept of Win-or-Learn-Slow (WoLS) mechanism for the proposed EMAQL algorithm so that the proposed algorithm learns slowly when it is losing. We also provide a theoretical proof of convergence to Nash equilibrium for the proposed EMAQL algorithm in games with pure Nash equilibrium. In games with mixed Nash equilibrium, our mathematical analysis shows that the proposed EMAQL algorithm converges to an equilibrium. Although our mathematical analysis does not explicitly show that the proposed EMAQL algorithm converges to Nash equilibrium, our simulation results indicate that the proposed EMAQL algorithm does converge to Nash equilibrium. The second type of games are differential games, where the states and actions are represented in continuous domains. We provide four main contributions. First, we propose a new fuzzy reinforcement learning algorithm for differential games that have continuous state and action spaces. Second, we propose a new fuzzy reinforcement learning algorithm for pursuit-evasion games so that the pursuer trained by the proposed algorithm can capture the evader even when the environment of the game is different from the training environment. Third, we propose a new decentralized fuzzy reinforcement learning algorithm for multi-pursuer pursuit-evasion differential games with a single-superior evader that has a speed similar to the speed of the pursuers. Fourth, we propose a new decentralized fuzzy reinforcement learning algorithm for multi-pursuer pursuit-evasion differential games with a single-superior evader that has a speed that is similar to or higher than the speed of each pursuer. Simulation results show the effectiveness of the proposed algorithms.

A linear two player zero-sum pursuit-evasion differential game is considered. The control functions of players are subject to integral constraints. In the game, the first player, the Pursuer, tries to force the state of the system towards the origin, while the aim of the second player, the Evader, is the opposite. We construct the optimal strategies of the players when the control resource of the Pursuer is greater than that of the Evader. The case where the control resources of the Pursuer are less than or equal to that of the Evader is studied to prove the main theorem. For this case a new method for solving of the evasion problem is proposed. We assume that the instantaneous control employed by the Evader is known to the Pursuer. For construction, the strategy of the Evader information about the state of the system and the control resources of the players is used. References R. Isaacs. Differential games. John Wiley and Sons, New York, 1965. L. S. Pontryagin. Collected works. Nauka, Moscow, 1988. (Russian) L. D. Berkovitz. Necessary conditions for optimal strategies in a class of differential games and control problems. SIAM Journal on Control , 5 , 1--24, 1967. L. D. Berkovitz. A survey of differential games. Mathematical Theory of Control , Edited by A. V. Balakrishnan and L. W. Neustadt, Academic Press, New York, 373--385, 1967. N. N. Krasovskii and A. I. Subbotin. Game-theoretical control problems. New York, Springer, 1988. W. H. Fleming. The convergence problem for differential games. Journal of Mathematical Analysis and Applications . 3 , 102--116, 1961. W. H. Fleming. The convergence problem for differential games, Part 2. Advances in Game Theory, Annals of Mathematics Studies , (52), Princeton University Press, Princeton, New Jersey,195--210, 1964. A. Friedman. Differential games. Wiley-Interscience, New York, 1971. R. J. Elliott and N. J. Kalton. The existence of value in differential games. Memoirs of the American Mathematical Society , 126 , 1--67, 1972. L. A. Petrosyan. Differential games of pursuit. World Scientific, Singapore, London, 1993. O. Hajek. Pursuit games. Academic Press, New York, San Francisco, 1975. A. Ya. Azimov. Linear differential pursuit game with integral constraints on the control. Differentsial'nye Uravneniya, 11 (10), 1975, 1723--1731; English transl. in Differential Equations 11 , 1283--1289, 1975. A. Ya. Azimov. A linear differential evasion game with integral constraints on the controls. USSR Computational Mathematics and Mathematical Physics, 14 (6), 56--65, 1974. M. S. Nikolskii. The direct method in linear differential games with integral constraints. Controlled systems, IM, IK, SO AN SSSR, (2), 49--59, 1969. A. I. Subbotin and V. N. Ushakov. Alternative for an encounter-evasion differential game with integral constraints on the playersi controls. PMM 39 (3), 387--396, 1975. V. N. Ushakov. Extremal strategies in differential games with integral constraints. PMM , 36 (1), 15--23, 1972. B. N. Pshenichnii and Yu. N. Onopchuk. Linear differential games with integral constraints. Izvestige Akademii Nauk SSSR, Tekhnicheskaya Kibernetika , (1), 13--22, 1968. A. A. Azamov, B. Samatov. $\pi $-strategy. An elementary introduction to the theory of differential games. National University of Uzbekistan. Tashkent, Uzbekistan, 2000. G. I. Ibragimov. A game problem on a closed convex set. Siberian Advances in Mathematics . 12 (3), 16--31, 2002. G. I. Ibragimov. A problem of optimal pursuit in systems with distributed parameters. J. Appl. Math. Mech, 66 (5), 719--724, 2003. E. B. Lee and and L. Markus. Foundations of optimal control theory, John Wiley and Sons Inc., New York, 1967.

Correlated relaxed equilibria in nonzero-sum linear differential games

I. Zero-sum differential games: Theory and applications in worst-case controller design.- A Theory of Differential Games.- H?-Optimal Control of Singularly Perturbed Systems with Sampled-State Measurements.- New Results on Nonlinear H? Control Via Measurement Feedback.- Reentry Trajectory Optimization under Atmospheric Uncertainty as a Differential Game.- II. Zero-sum differential games: Pursuit-evasion games and numerical schemes.- Fully Discrete Schemes for the Value Function of Pursuit-Evasion Games.- Zero Sum Differential Games with Stopping Times: Some Results about its Numerical Resolution.- Singular Paths in Differential Games with Simple Motion.- The Circular Wall Pursuit.- III. Mathematical programming techniques.- Decomposition of Multi-Player Linear Programs.- Convergent Stepsizes for Constrained Min-Max Algorithms.- Algorithms for the Solution of a Large-Scale Single-Controller Stochastic Game.- IV. Stochastic games: Differential, sequential and Markov Games.- Stochastic Games with Average Cost Constraints.- Stationary Equilibria for Nonzero-Sum Average Payoff Ergodic Stochastic Games and General State Space.- Overtaking Equilibria for Switching Regulator and Tracking Games.- Monotonicity of Optimal Policies in a Zero Sum Game: A Flow Control Model.- V. Applications.- Capital Accumulation Subject to Pollution Control: A Differential Game with a Feedback Nash Equilibrium.- Coastal States and Distant Water Fleets Under Extended Jurisdiction: The Search for Optimal Incentive Schemes.- Stabilizing Management and Structural Development of Open-Access Fisheries.- The Non-Uniqueness of Markovian Strategy Equilibrium: The Case of Continuous Time Models for Non-Renewable Resources.- An Evolutionary Game Theory for Differential Equation Models with Reference to Ecosystem Management.- On Barter Contracts in Electricity Exchange.- Preventing Minority Disenfranchisement Through Dynamic Bayesian Reapportionment of Legislative Voting Power.- Learning by Doing and Technology Sharing in Asymmetric Duopolies.

In this chapter, a new optimal coordination control for the consensus problem of heterogeneous multi-agent differential graphical games by iterative ADP is developed. The main idea is to use iterative ADP technique to obtain the iterative control law which makes all the agents track a given dynamics and simultaneously makes the iterative performance index function reach the Nash equilibrium. In the developed heterogeneous multi-agent differential graphical games, the agent of each node is different from the one of other nodes. The dynamics and performance index function for each node depend only on local neighbor information. A cooperative policy iteration algorithm for graphical differential games is developed to achieve the optimal control law for the agent of each node, where the coupled Hamilton–Jacobi (HJ) equations for optimal coordination control of heterogeneous multi-agent differential games can be avoided. Convergence analysis is developed to show that the performance index functions of heterogeneous multi-agent differential graphical games can converge to the Nash equilibrium. Simulation results will show the effectiveness of the developed optimal control scheme.

В.И. Жуковский1, К.Н. Кудрявцев2, С.П. Самсонов1, М.И. Высокос3, Ю.А. Бельских3 1 Московский государственный университет им. М.В. Ломоносова, г. Москва, Российская Федерация 2 Южно-Уральский государственный университет, г. Челябинск, Российская Федерация 3 Государственный гуманитарно-технологический университет, г. Орехово-Зуево, Российская Федерация E-mail: kudrkn@gmail.co. V.I. Zhukovskiy1, K.N. Kudryavtsev2, S.P. Samsonov1, M.I. Vysokos3, Yu.A. Belskih3 1 Lomonosov Moscow State University, Moscow, Russian Federation 2 South Ural State University, Chelyabinsk, Russian Federation 3 State University of Humanities and Technology, Orekhovo-Zuyevo, Russian Federation

Plane Pursuit with Curvature Constraints

We consider a two-player nonzero-sum differential game in the case where players use nonanticipative strategies. We define the Nash equilibrium in this case and obtain a characterization of Nash equilibrium strategies. We show that a Nash equilibrium solution can be approximately realized by control-with-guide strategies.

Group control of connected and autonomous vehicles on automated highways is challenging for the advanced driver assistance systems (ADAS) and the automated driving systems (ADS). This paper investigates the differential game-based approach to autonomous convoy control with the aim of deployment on automated highways. Under the noncooperative differential games, the coupled vehicles make their decisions independently while their states are interdependent. The receding horizon Nash equilibrium of the linear-quadratic differential game provides the convoy a distributed state-feedback control strategy. This approach suffers a fundamental issue that neither a Nash equilibrium’s existence nor the uniqueness is guaranteed. We convert the individual dynamics-based differential game to a relative dynamics-based optimal control problem that carries all the features of the differential game. The existence of a unique Nash control under the differential game corresponds to a unique solution to the optimal control problem. The latter is shown, as well as the asymptotic stability of the closed-loop system. Simulations illustrate the effectiveness of the presented convey control scheme and how it well suits automated highway driving scenarios.

In this thesis, we study the problem of computing approximate equilibria in several classes of games. In particular, we study approximate Nash equilibria and approximate well-supported Nash equilibria in polymatrix and bimatrix games and approximate equilibria in Lipschitz games, penalty games and biased games. We construct algorithms for computing approximate equilibria that beat the cur- rent best algorithms for these problems. In Chapter 3, we present a distributed method to compute approximate Nash equilibria in bimatrix games. In contrast to previous approaches that analyze the two payoff matrices at the same time (for example, by solving a single LP that combines the two players’ payoffs), our algorithm first solves two independent LPs, each of which is derived from one of the two payoff matrices, and then computes an approximate Nash equilibrium using only limited communication between the players. In Chapter 4, we present an algorithm that, for every δ in the range 0 < δ ≤ 0.5, finds a (0.5+δ)-Nash equilibrium of a polymatrix game in time polynomial in the input size and 1 . Note that our approximation guarantee does not depend on δ the number of players, a property that was not previously known to be achievable for polymatrix games, and still cannot be achieved for general strategic-form games. In Chapter 5, we present an approximation-preserving reduction from the problem of computing an approximate Bayesian Nash equilibrium (e-BNE) for a two-player Bayesian game to the problem of computing an e-NE of a polymatrix game and thus show that the algorithm of Chapter 4 can be applied to two-player Bayesian games. Furthermore, we provide a simple polynomial-time algorithm for computing a 0.5-BNE. In Chapter 5, we study games with non-linear utility functions for the players. Our key insight is that Lipschitz continuity of the utility function allows us to provide algorithms for finding approximate equilibria in these games. We begin by studying Lipschitz games, which encompass, for example, all concave games with Lipschitz continuous payoff functions. We provide an efficient algorithm for computing approximate equilibria in these games. Then we turn our attention to penalty games, which encompass biased games and games in which players take risk into account. Here we show that if the penalty function is Lipschitz continuous, then we can provide a quasi-polynomial time approximation scheme. Finally, we study distance biased games, where we present simple strongly poly- nomial time algorithms for finding best responses in L1, L2, and L∞ biased games, and then use these algorithms to provide strongly polynomial algorithms that find 2/3, 5/7, and 2/3 approximations for these norms, respectively.

A linear-quadratic positional differential game of N persons is considered. The solution of a game in the form of Nash equilibrium has become widespread in the theory of noncooperative differential games. However, Nash equilibrium can be internally and externally unstable, which is a negative in its practical use. The consequences of such instability could be avoided by using Pareto maximality in a Nash equilibrium situation. But such a coincidence is rather an exotic phenomenon (at least we are aware of only three cases of such coincidence). For this reason, it is proposed to consider the equilibrium of objections and counterobjections. This article establishes the coefficient criteria under which in a differential positional linear-quadratic game of N persons there is Pareto equilibrium of objections and counterobjections and at the same time no Nash equilibrium situation; an explicit form of the solution of the game is obtained.

Equilibria, fixed points, and complexity classes

This paper is devoted to the differential positional coalitional games with non-transferable payoffs (games without side payments). We believe that the researches of the objection and counter-objection equilibrium for non-cooperative differential games that have been carried out over the last years allow to cover some aspects of non-transferable payoff coalitional games. In this paper we consider the issues of the internal and external stability of coalitions in the class of positional differential games. For a differential positional linear-quadratic six-player game with a two-coalitional structure, the coefficient constraints are obtained which provide an internal and external stability of the coalitional structure.

This paper tackles the problem of finding a Nash-equilibrium for a differential game with multiple scenarios or multiple models of the game. Player's dynamics is governed by an ordinary differential equation with unknown parameters (Multi-Model representation) from a given finite set. The problem consists in the designing of min-max strategies for each player which guarantee an equilibrium for the worst case scenario. Based on the Robust Maximum Principle necessary conditions for a game to be in Robust Nash Equilibrium are derived. The LQ differential games are considered in detail. It is shown that the initial min-max differential game may be converted in to a standard static game given in a multidimensional simplex. A numerical procedure for resolving the LQ differential game is designed.