Differential Game Problem Research Articles

Reinforcement learning for exploratory linear-quadratic two-person zero-sum stochastic differential games

In this paper, we study an entropy-regularized continuous-time linear-quadratic two-person zero-sum stochastic differential game problem from the perspective of reinforcement learning (RL). By the solvability of a discounted algebraic Riccati equation, we construct a Gaussian closed-loop optimal control pair for the problem, which achieves the best tradeoff between exploration and exploitation. Then, in this exploratory framework, we propose an RL algorithm that relies on only partial system information to solve a stochastic H∞ control problem. The corresponding convergence analysis and simulation examples are also provided to verify the efficiency of the proposed algorithm.

Pareto-based Stackelberg differential game for stochastic systems with multi-followers

This paper is concerned about a Pareto-based Stackelberg stochastic differential game with multi-followers, in which followers are cooperative relations. First of all, necessary and sufficient conditions for the existence of Pareto-based Stackelberg equilibrium are given for the general Pareto-based Stackelberg stochastic differential game problem. Next, we study the above game problem in the linear-quadratic (LQ) case and obtain a feedback form of Pareto-based Stackelberg equilibrium. The solvability of Riccati equations is guaranteed under some assumptions. Finally, the theoretical results are used to solve a national debt management problem.

A Nonlinear Finite-Time Robust Differential Game Guidance Law.

In this paper, a robust differential game guidance law is proposed for the nonlinear zero-sum system with unknown dynamics and external disturbances. First, the continuous-time nonlinear zero-sum differential game problem is transformed into solving the nonlinear Hamilton–Jacobi–Isaacs equation, a time-varying cost function is developed to reflect the fixed terminal time, and the robust guidance law is developed to compensate for the external disturbance. Then, a novel neural network identifier is designed to approximate the unknown nonlinear dynamics with online weight tuning. Subsequently, an online critic neural network approximator is presented to estimate the cost function, and time-varying activation functions are considered to deal with the fixed final time problem. An adaptive weight tuning law is given, where two additional terms are added to ensure the stability of the closed-loop nonlinear system and so as to meet the terminal cost at a fixed final time. Furthermore, the uniform ultimate boundedness of the closed-loop system and the critic neural network weights estimation error are proven based upon the Lyapunov approach. Finally, some simulation results are presented to demonstrate the effectiveness of the proposed robust differential game guidance law for nonlinear interception.

A representation theorem approach to probabilistic interpretation for viscosity solutions of Isaacs equations

The main objective of this paper is to show a certain representation theorem for generators of generalized backward stochastic differential equations (GBSDEs) and its application to probabilistic interpretation for viscosity solutions of Hamilton-Jacobi-Bellman-Isaacs equations with nonlinear Neumann boundary value problems. In the representation theorem a time change method is adopted to treat the difficulty brought from the random measure contained in GBSDEs. By means of the representation theorem the viscosity solution of Isaacs equations with nonlinear Neumann problems is proved to be the value function of a two-player zero-sum stochastic differential game problem with state constraints and recursive cost functionals.

Maximum principle for partially observed stochastic recursive optimal control problems involving impulse controls

AbstractThis article focuses on a stochastic recursive optimal control problem involving impulse control under partially observed information and obtains the related maximum principle. Unlike the classical literature, both the regular control (considering the nonconvexity of the domain) and the impulse control (where the domain is convex) are considered in the framework. In virtue of two types of variational methods, spike variation and convex variation, the Pontryagin maximum principle is established. In addition, to demonstrate the validity of the results, this article investigates a differential game problem as an application.

Stochastic Recursive Zero-sum Differential Games under Model Uncertainty

In this paper, we study stochastic recursive zero-sum differential game problem where the payoff function is described by the solutions of a class of backward stochastic differential equations with uncertainty parameter , which are used to represent different market conditions. In such case, the existence of saddle points for stochastic differential game problems above-mentioned is proved.

Convolutional neural network based simulation and analysis for backward stochastic partial differential equations

We develop a generic convolutional neural network (CNN) based numerical scheme to simulate the 2-tuple adapted strong solution to a unified system of backward stochastic partial differential equations (B-SPDEs) driven by Brownian motions, which can be used to model many real-world system dynamics such as optimal control and differential game problems. The dynamics of the scheme is modeled by a CNN through conditional expectation projection. It consists of two convolution parts: W layers of backward networks and L layers of reinforcement iterations. Furthermore, it is a completely discrete and iterative algorithm in terms of both time and space with mean-square error estimation and almost sure (a.s.) convergence supported by both theoretical proof and numerical examples. In doing so, we need to prove the unique existence of the 2-tuple adapted strong solution to the system under both conventional and Malliavin derivatives with general local Lipschitz and linear growth conditions.

Quantitative evaluation of machine learning capability based on a differential game problem

With the development of artificial intelligence, various machine learning algorithms have emerged, but in the past, it was very inefficient to use machine learning methods to solve problems such as optimal control and problems of game theory, so a method that can quantitatively evaluate the machine learning capability is proposed, which can significantly improve the efficiency of existing algorithms. In this paper, a differential game problem based on the sheep-dog two-dimensional motion model is established and five related questions are answered according to the constraints in the problem. Meanwhile, the algorithm is analyzed quantitatively and the corresponding results are obtained. The paper also verifies that several models, equations of motion and machine learning algorithms established in this paper are reliable through sensitivity analysis.

Nonzero-sum differential games of continuous-Time nonlinear systems with uniformly ultimately [formula omitted]-bounded by adaptive dynamic programming

In this paper, a single network adaptive dynamic programming (ADP) control method is presented to obtain the nearly optimal control policies for the non-zero sum (NZS) differential game problem of the autonomous nonlinear system. The Osgood condition, instead of the traditional Lipschitz condition, is firstly introduced to policy iteration to guarantee the existence and uniqueness of the solution of the dynamic nonlinear systems and to weaken the limited conditions of nonlinear dynamic functions f(x), g(x) and k(x). Moreover, this adaptive control pattern finds in real-time approximations of the optimal value and the non-zero sum Nash-equilibrium, while also ensuring the uniform ultimate ε-boundedness of the closed-loop system. Further, as the number of hidden-layer neurons tends to infinite, the approximation errors converge to zero. As a result, the closed-loop system is asymptotically stable. Finally, the effectiveness of the proposed near-optimal control pattern is verified by a simulation example.

Predictive Differential Game Guidance Approach for Hypersonic Target Interception Based on CQPSO

In the descent phase of the hypersonic target flight trajectory, the hypersonic speed of the target makes the reaction time shorter. Due to overload restrictions, it is challenging for the interceptor to achieve successful interception. To address this problem, a predictive differential game guidance approach based on dynamic optimization algorithm is proposed to relax the interceptor acceleration requirements. Two-dimensional kinematics and dynamics engagement mode for hypersonic target interception is formulated as a nonlinear differential control model in the presence of matched uncertainties. The nonlinear differential model is transformed into a differential game model by combining a performance index. The performance index is positively correlated with the Line-Of-Sight (LOS) rate and control effort of interceptor and negatively correlated with the target maneuver. The Chaotic Quantum Particle Swarm Optimization (CQPSO) algorithm is utilized to address the nonlinear differential game problem to generate the instantaneous and predicted guidance commands for the interceptor and target. Numerical examples are given to verify the performance of the proposed guidance law in various engagement scenarios, including different target maneuvers, initial heading angles and target overload. Moreover, the performance of the algorithm is validated comparing with traditional and state-of-the-art guidance laws.

An Evader Control Strategy in the Non-Linear Differential Game Problem with Terminal Limitations

An Evader Control Strategy in the Non-Linear Differential Game Problem with Terminal Limitations

Pursuit and Evasion Strategy of a Differential Game Based on Deep Reinforcement Learning.

Since the emergence of deep neural network (DNN), it has achieved excellent performance in various research areas. As the combination of DNN and reinforcement learning, deep reinforcement learning (DRL) becomes a new paradigm for solving differential game problems. In this study, we build up a reinforcement learning environment and apply relevant DRL methods to a specific bio-inspired differential game problem: the dog sheep game. The dog sheep game environment is set on a circle where the dog chases down the sheep attempting to escape. According to some presuppositions, we are able to acquire the kinematic pursuit and evasion strategy. Next, this study implements the value-based deep Q network (DQN) model and the deep deterministic policy gradient (DDPG) model to the dog sheep game, attempting to endow the sheep the ability to escape successfully. To enhance the performance of the DQN model, this study brought up the reward mechanism with a time-out strategy and the game environment with an attenuation mechanism of the steering angle of sheep. These modifications effectively increase the probability of escape for the sheep. Furthermore, the DDPG model is adopted due to its continuous action space. Results show the modifications of the DQN model effectively increase the escape probabilities to the same level as the DDPG model. When it comes to the learning ability under various environment difficulties, the refined DQN and the DDPG models have bigger performance enhancement over the naive evasion model in harsh environments than in loose environments.

On Nash-Coalitive Fuzzy Continuous Differential Games

This paper concentrates on providing a simple and effective technique to solve the Nash-coalitive fuzzy continuous differential games. These games are featured by having fuzzy coefficients, fuzzy controls, and fuzzy state functions. The proposed technique utilizes the α-cut set concept of fuzzy numbers to transform the problem under study into a corresponding interval problem. The latter one is sliced into two problems, lower problem and upper problem, to obtain the optimal solution for the Nash-coalitive fuzzy continuous differential games problem in a range form. The sufficient and necessary conditions for this problem are derived. A numerical example is provided to clarify the presented concepts. Keywords— fuzzy control, fuzzy differential games, fuzzy numbers, Nash-coalitive differential games

Pursuit differential game problem with integral and geometric constraints.

Abstract We study pursuit differential game problem in which a countable number of pursuers chase one evader. The problem is formulated in a Hilbert space l2 with pursuers’ motions described by nth order differential equations and that of the evader by mth order differential equation. The control functions of the pursuers and evader are subject to integral and geometric constraints respectively.Duration of the game is denoted by the positive number?. Pursuit is said to be completed if there exist strategies uj of the pursuers Pj such that for any admissible control v(·) of the evader E the inequality ky(?) ? xj (?)k ? rj is satisfied for some j ? {1, 2, . . .}. In this paper, sufficient condition for completion of pursuit were obtained and also strategies of the pursuers that ensure completion of pursuit are constructed.

Linear quadratic nonzero‐sum stochastic differential game of a partially observed Markov jump linear systems

AbstractIn this study, a partially observed linear quadratic (LQ) non‐cooperative stochastic differential game of Markov jump linear system (MJLS) is considered. First, a new proof of filtering theory is given, using the separation principle and filtering technology, partial observations can be transformed into complete observation. Second, nonzero‐sum stochastic differential game problem for MJLS are studied using the dynamic programming principle and completion square method. Then, the sufficient and necessary conditions for the Nash equilibrium are obtained. Finally, a numerical simulation is performed.

Game value for a pursuit-evasion differential game problem in a Hilbert space

<p style='text-indent:20px;'>We consider a pursuit-evasion differential game problem with countable number pursuers and one evader in the Hilbert space <inline-formula><tex-math id="M1">\begin{document}$ l_{2}. $\end{document}</tex-math></inline-formula> Players' dynamic equations described by certain <inline-formula><tex-math id="M2">\begin{document}$ n^{th} $\end{document}</tex-math></inline-formula> order ordinary differential equations. Control functions of the players subject to integral constraints. The goal of the pursuers is to minimize the distance to the evader and that of the evader is the opposite. The stoppage time of the game is fixed and the game payoff is the distance between evader and closest pursuer when the game is stopped. We study this game problem and find the value of the game. In addition to this, we construct players' optimal strategies.</p>

Risk-sensitive zero-sum stochastic differential game for jump–diffusions

This article is concerned with the infinite horizon risk-sensitive zero-sum stochastic differential game problem for a class of jump–diffusions controlled through the drift, and driven by a compensated Poisson process and a Wiener process. Under certain geometric stability assumption on the dynamics, we completely characterize all possible saddle point strategies in the class of stationary Markov strategies. We obtain our result by exploiting the stochastic representation of the principal eigenfunction of the associated Hamilton–Jacobi–Isaac (HJI) equation, which is a semilinear integro-partial differential equation.

On two pursuit differential game problems with state and geometric constraints in a Hilbert space

This paper concerns with the study of two pursuit differential game problems of many pursuers and many evaders on a nonempty closed convex subset of R^n. Throughout the period of the games, players must stay within the given closed convex set. Players’ laws of motion are defined by certain first order differential equations. Control functions of the pursuers and evaders are subject to geometric constraints. Pursuit is said to be completed if the geometric position of each of the evader coincides with that of a pursuer. We proved two theorems each of which is solution to a problem. Sufficient conditions for the completion of pursuit are provided in each of the theorems. Moreover, we constructed strategies of the pursuers that ensure completion of pursuit.

The Principle-Agent Conflict Problem in a Continuous-Time Delegated Asset Management Model

This paper considers the principle-agent conflict problem in a continuous-time delegated asset management model when the investor and the fund manager are all risk-averse with risk sensitivity coefficients γ f and γ m , respectively. Suppose that the investor entrusts his money to the fund manager. The return of the investment is determined by the manager’s effort level and incentive strategy, but the benefit belongs to the investor. In order to encourage the manager to work hard, the investor will determine the manager’s salary according to the terminal income. This is a stochastic differential game problem, and the distribution of income between the manager and the investor is a key point to be solved in the custody model. The uncertain form of the incentive strategy implies that the problem is different from the classical stochastic optimal control problem. In this paper, we first express the investor’s incentive strategy in term of two auxiliary processes and turn this problem into a classical one. Then, we employ the dynamic programming principle to solve the problem.

A Study on Nash-Collative Differential Game of N -Players for Counterterrorism

In this work, by using the Nash-collative approach for a differential game problem between N -governments and terrorist organizations, we study governments’ cooperation and the role of each government for counterterrorism. Furthermore, we discuss the intertemporal strategic interaction of governments and terrorist organizations, where all world governments have to cooperate to fight terrorism. Also, we study the necessary conditions for finding the optimal strategies for each government to fight terrorism; we discuss the existence of the solution of the formulated problem and the stability set of the first kind of the optimal strategies.