Nonzero-sum Games Research Articles

Dynamic event-triggered finite-horizon robust suboptimal control of multi-player systems with input disturbances

This paper presents an adaptive dynamic event-triggered robust finite-horizon suboptimal control scheme for multi-player nonzero-sum game of nonlinear systems with input disturbances based on dynamic programming (ADP) and integral sliding mode (ISM) control techniques. Firstly, dynamic event-triggered adaptive ISM controllers are designed to handle input disturbances that can force the system state to remain on the sliding manifold and relax the known upper bounds condition of input disturbances. Corresponding event-triggered condition is employed for ISM controller of each player to reduce the update frequency. Then, utilizing the obtained ISM dynamics, we employ ADP-based single critic neural networks to approximate the time-varying solution of the Hamilton–Jacobi equations. Thus, the event-triggered finite-horizon suboptimal controllers can be obtained, and corresponding state-based dynamic event-triggered condition is designed to improve the resource utilization. The stability property of the closed-loop system is demonstrated through the Lyapunov technique. Finally, the effectiveness of the proposed control scheme is confirmed through two simulation examples.

Integral reinforcement learning-based event-triggered optimal tracking control for modular robot manipulators via non-zero-sum game

Abstract Under an event-triggered mechanism, a non-zero-sum (NZS) game optimal tracking control method for modular robot manipulator (MRM) systems with input constraints is proposed using the adaptive dynamic programming (ADP) method based on integral reinforcement learning (IRL). First, a dynamic model of the MRM system is developed based on joint torque feedback technology, consisting of an n-joint subsystem related to interconnected dynamic coupling (IDC). Second, we design a robust compensation controller to handle the known model term and an optimal compensation controller to deal with the uncertainty term caused by the IDC and friction, respectively. In addition, a nonlinear disturbance observer is established to dispose of the negative effects caused by the uncertain sensor output disturbance. Third, based on differential game theory, we transform the optimal tracking control problem of the MRM system into an n-player NZS game problem. Then, the IRL-based ADP method is adopted, which relaxes the need for system partial unknown dynamic information, and only a critic neural network is used to solve the coupled Hamilton–Jacobi equation, so as to obtain the optimal control policy. Then, using Lyapunov theory, the tracking error of the MRM system is demonstrated to be uniformly ultimately bounded. Finally, the effectiveness and superiority of the proposed algorithm are verified through experiments.

Optimal Spin Polarization Control for the Spin-Exchange Relaxation-Free System Using Adaptive Dynamic Programming.

This work is the first to solve the 3-D spin polarization control (3DSPC) problem of atomic ensembles, which controls the spin polarization to achieve arbitrary states with the cooperation of multiphysics fields. First, a novel adaptive dynamic programming (ADP) structure is proposed based on the developed multicritic multiaction neural network (MCMANN) structure with nonquadratic performance functions, as a way to solve the multiplayer nonzero-sum game (MP-NZSG) problem in 3DSPC under the constraints of asymmetric saturation inputs. Then, we utilize the MCMANNs to implement the multicritic multiaction ADP (MCMA-ADP) algorithm, whose convergence is proven by the compression mapping principle. Finally, the MCMA-ADP is deployed in the spin-exchange relaxation-free (SERF) system to provide a set of control laws in 3DSPC that fully exploits the multiphysics fields to achieve arbitrary spin polarization states. Numerical simulations support the theoretical results.

Two-player nonlinear Stackelberg differential game via off-policy integral reinforcement learning

For the nonzero-sum games, players have taken different strategies to achieve the Nash equilibrium. However, regarding hierarchical optimization and asymmetric information, Nash equilibrium is ineffective. The Stackelberg game provides us with a leader–follower strategy to cope with this problem. This paper solved two-player continuous-time nonlinear Stackelberg differential games with unknown system dynamics using an off-policy integral reinforcement learning (IRL) technique. A two-level hierarchy optimal control problem is the Stackelberg differential game, and locating the open-loop Stackelberg equilibrium is equivalent to solving the coupled Hamiltonian-Jacobi (HJ) equations. First, the optimal strategies for the leader and the follower are derived. Then, the closed-loop system’s asymptotic stability of optimal control solution is proved. Based on the policy iteration (PI) algorithm, an off-policy IRL algorithm is developed to solve the Stackelberg game for completely unknown system dynamics. The control strategies and cost functions of the leader and follower are approximated using neural networks (NNs). It is demonstrated that NNs weight errors of off-policy algorithm are uniformly ultimately bounded (UUB). Lastly, two simulation examples demonstrate that the proposed learning algorithm is capable of obtaining two-player Stackelberg equilibrium strategies.

Sliding-mode surface-based adaptive optimal nonzero-sum games for saturated nonlinear multi-player systems with identifier-critic networks

This paper addresses the sliding-mode surface-based adaptive optimal nonzero-sum games control problem for continuous-time nonlinear systems with input constraints. By constructing a value function associated with the sliding-mode surface, the original nonzero-sum games problem can be transformed into constructing multiple optimal control policies. Then, an adaptive dynamic programming method under identifier-critic networks is introduced to deal with the Hamilton–Jacobi equation, where the identifier networks are applied to approximate the unknown dynamics. A remarkable feature is that the critic updating laws are designed through experience replay and gradient descent methodologies, such that the persistence of excitation condition can be excluded. Based on the Lyapunov stability theory, all signals of the close-loop system are verified to be uniformly ultimately bounded. Finally, simulation results are given to illustrate the effectiveness of the proposed optimal nonzero-sum games scheme.

Open-loop and closed-loop local and remote stochastic nonzero-sum game with inconsistent information structure

In this paper, the open-loop and closed-loop local and remote stochastic nonzero-sum game (LRSNG) problem is investigated. Different from previous works, the stochastic nonzero-sum game problem under consideration is essentially a special class of two-person nonzero-sum game problem, in which the information sets accessed by the two players are inconsistent. More specifically, both the local player and the remote player are involved in the system dynamics, and the information sets obtained by the two players are different, and each player is designed to minimise its own cost function. For the considered LRSNG problem, both the open-loop and closed-loop Nash equilibrium are derived. The contributions of this paper are given as follows. Firstly, the open-loop optimal Nash equilibrium is derived, which is determined in terms of the solution to the forward and backward stochastic difference equations (FBSDEs). Furthermore, by using the orthogonal decomposition method and the completing square method, the feedback representation of the optimal Nash equilibrium is derived for the first time. Finally, the effectiveness of our results is verified by a numerical example.

QESRL: Exploring Selfish Reinforcement Learning for Repeated Quantum Games

A novel unified learning algorithm that seamlessly applies to both classical and quantum non-zero sum games is presented. Building upon the exploring selfish reinforcement learning (ESRL) framework previously proposed in the context of classical games, we extend this approach to handle quantum games with imperfect information. A comparison is made between performance and fairness among agents learning using plain QRL vs. QESRL. The latter enables agents to explore and learn periodic policy strategies in quantum games, leveraging the quantization of games to uncover fairer results. By addressing the challenges posed by the expanded strategy space in quantum games, we test the algorithm’s scalability by increasing the number of agents. Empirical evidence is provided to showcase its performance and to compare classical and quantum game scenarios. The proposed learning algorithm represents a significant step towards understanding the convergence and optimality of strategies in non-zero sum games across classical and quantum settings, bringing us closer to harnessing the potential of independent reinforcement learning and quantum computing in game theory applications.

Nonzero-sum games using actor-critic neural networks: A dynamic event-triggered adaptive dynamic programming

This paper mainly investigates the nonzero-sum games of nonlinear systems with unmatched uncertainty by using actor-critic neural networks. To handle the unmatched components, an auxiliary system with a modified value function is constructed, which transforms the robust stabilization issue into the optimal control issue. Then, a novel dynamic event-triggering condition is designed to further save bandwidth via introducing a dynamic variable. In addition, the actor-critic algorithm is employed in adaptive dynamic programming to achieve Nash equilibrium, which is tuned together with the control policy. By constructing appropriate Lyapunov functions, a criterion is established to ensure that the considered system is uniformly ultimately bounded. Finally, the effectiveness of the developed strategy is demonstrated by an example.

Dynamic event-triggered robust optimal tracking control for multi-player nonzero-sum games with mismatched uncertainties and asymmetric constrained inputs

In this paper, a novel dynamic event-triggered robust optimal tracking control method is proposed for multi-player nonzero-sum game problem with mismatched uncertainties and asymmetric constrained inputs. First, the original asymmetric constrained-input multi-player system is transformed into a new multi-player system with symmetric constrained inputs implemented by the state-space transformation technique, and an augmented system is established based on the new system and the desired trajectory. Next, an auxiliary augmentation system is constructed to handle the robust tracking control problem. Then, a critic neural network with a novel modified weight updating law is established for each player to solve the event-triggered coupled Hamilton–Jacobi equations, which relaxes the restrictions of initial admissible control and persistence of excitation condition. Furthermore, efficient communication and computing are obtained by the newly established dynamic event-triggered mechanism, and the Zeno behavior can be excluded by introducing an exponential term. The uniformly ultimately bounded properties of all signals are proved. Finally, two numerical examples verify the effectiveness of the designed method.

Combination Therapy-Based Adaptive Control for Organism Using Medicine Dosage Regulation Mechanism.

In this article, the optimal control strategy for organism is investigated by using the adaptive dynamic programming (ADP) method under the architecture of nonzero-sum games (NZSGs). First, a tumor model is established to formulate the interaction relationships among normal cells, tumor cells, endothelial cells, and the concentrations of drugs. Then, the ADP-based method of single-critic network architecture is proposed to approximate the coupled Hamilton-Jacobi equations (HJEs) under the medicine dosage regulation mechanism (MDRM). According to the game theory, the approximate MDRM-based optimal strategy can be derived, which is of great practical significance. Owing to the proposed mechanism, the dosages of the chemotherapy and anti-angiogenic drugs can be regulated timely and necessarily. Furthermore, the stability of the closed-loop system with the obtained strategy is analyzed via the Lyapunov theory. Finally, a simulation experiment is conducted to verify the effectiveness of the proposed method.

Initial Excitation-Based Optimal Control for Continuous-Time Linear Nonzero-Sum Games

Initial Excitation-Based Optimal Control for Continuous-Time Linear Nonzero-Sum Games

Base on -Learning Pareto Optimality for Linear Itô Stochastic Systems With Markovian Jumps

This article investigate the cooperative differential game (CDG) for continuous-time linear Itô stochastic systems with markovian jumps (SSMJ) to obtain the Pareto solutions. Different from most existing works studying nonzero-sum games, this article studies the CDG on the quadratic infinite horizon for the Itô-type SSMJ with unknown system matrix and transition probability. A novel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning online algorithm is developed, which consists of that (i) the optimal control problem is equivalent to solving a stochastic algebraic Riccatic equation (ARE); (ii) the joint cost function is approximated by a critic neural network (NN) and Pareto efficient is approximated by two actor NNs. The rigorous stability analysis shows that the system state for SSMJ and the NN weight errors are uniformly ultimately bounded (UUB). Finally, the theory analysis is validated by a numerical example with detailed discussions. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In practical applications, many systems are often affected by the change of external environment or the failure of internal components, which leads to the random jump of system parameters. Markovian jump system can effectively describe the above problems. And when the system model is disturbed by the internal parameters, the system control input and external environment, the random errors of state measurement, and other random factors, the deterministic model can no longer accurately describe the controlled system. Therefore, the SSMJ can describe practical problems more accurately. By cooperation, in general, the cost one specific player incurs is not uniquely determined anymore. If all players decide, for example, to use their control variables to reduce the cost of player 1 as much as possible, a different minimum is attained for player 1 compared with that in the case where all players agree collectively to help a different player in minimizing his cost. So, depending on how the players choose to ‘divide’ their control efforts, a player incurs different ‘minima’. Therefore, we will design an online learning algorithm to obtain pareto solutions with different weights. On the other hand, in practice, it is difficult to obtain an accurate system model. In order to solve this problem, a novel scheme is designed by using <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning technology, which does not need system matrix.

Dynamic Event-Triggered Strategy-Based Optimal Control of Modular Robot Manipulator: A Multiplayer Nonzero-Sum Game Perspective.

Due to the limited computing and processing ability of modular robot manipulator (MRM) components, such as sensors and controllers, event-triggered mechanisms are considered a crucial communication paradigm shift in resource constrained applications. Dynamic event-triggered mechanism is developing into a new technology by reason of its higher resource utilization efficiency and more flexible system design requirements than traditional event-triggered. Therefore, an optimal control scheme of multiplayer nonzero-sum game based on dynamic event-triggered is developed for MRM systems with uncertain disturbances. First, dynamic model of the MRM is established according to joint torque feedback technique and model uncertainty is estimated by data-driven-based neural network identifier. In the framework of differential game, the tracking control problem of MRM system is transformed into the optimal control problem for multiplayer nonzero-sum game with the control input of each joint module as the player. Then, the static event-triggered control problem of MRM system is studied based on adaptive dynamic programming algorithm. On this basis, the internal dynamic variable describing the previous state of the system is introduced, and the characteristics of dynamic trigger rule and its relationship with static rule are revealed theoretically. By designing an exponential attenuation signal, the minimum sampling interval of the system is always positive, so that Zeno behavior is excluded. Lyapunov theory proves that the system is asymptotically stable and the experimental results verify the validity of the proposed method.

An Unknown Multiplayer Nonzero-Sum Game: Prescribed-Time Dynamic Event-Triggered Control via Adaptive Dynamic Programming

An Unknown Multiplayer Nonzero-Sum Game: Prescribed-Time Dynamic Event-Triggered Control via Adaptive Dynamic Programming

Nonzero-Sum Game-Based Voltage Recovery Consensus Optimal Control for Nonlinear Microgrids System.

Since most of the existing models based on the microgrids (MGs) are nonlinear, which could cause the controller oscillate, resulting in the excessive line loss, and the nonlinear could also lead to the controller design difficulty of MGs system. Therefore, this article researches the distributed voltage recovery consensus optimal control problem for the nonlinear MGs system with N -distributed generations (DGs), in the case of providing stringent real power sharing. First, based on the distributed cooperative control concept of multiagent systems and the critic neural networks (NNs), a novel distributed secondary voltage recovery consensus optimal control protocol is constructed via applying the backstepping technique and nonzero-sum (NZS) differential game strategy to realize the voltage recovery of island MGs. Meanwhile, the model identifier is established to reconstruct the unknown NZS games systems based on a three-layer NN. Then, a critic NN weight adaptive adjustment tuning law is proposed to ensure the convergence of the cost functions and the stability of the closed-loop system. Furthermore, according to Lyapunov stability theory, it is proven that all signals are uniform ultimate boundedness in the closed loop system and the voltage recovery synchronization error converges to an arbitrarily small neighborhood of the origin near. Finally, some simulation results in MATLAB illustrate the validity of the proposed control strategy.

An eavesdropping and jamming dilemma with sophisticated players

Wireless networks are susceptible to malicious attacks, especially those involving eavesdropping and jamming. In this paper, we consider a communication scenario involving a transmitter who wishes to communicate secretly and reliably with a receiver, while an adversary wants to obstruct this communication by means of either eavesdropping or jamming. The transmitter as well as the adversary wants to achieve its own goal in a manner that is as unpredictable as possible to its rival. We model this problem by a non-zero sum game. The expected throughput that is delivered secretly and non-jammed to the receiver is considered as the metric that reflects communication secrecy and reliability. The entropy of the player’s strategies is considered as the metric to reflect the player’s unpredictability. The equilibrium is found in closed form, and parameters of the transmitter’s utility supporting both goals are optimized via a proportional fairness approach.

Nonzero-Sum Pursuit-Evasion Game Control for Spacecraft Systems: A Q-Learning Method

In this paper, the nonzero-sum pursuit-evasion (PE) game control problem is studied for a class of linear spacecraft control systems subject to the complete information case and incomplete information case. The incomplete information includes the cost functions and the control inputs. In practical confrontation situations, due to the incomplete information constraints, it is impossible for the pursuer and the evader to build up the exact opposite cost function. Hence, a nonzero-sum game framework is utilized to describe the PE game problem of the double-spacecraft system. First of all, under the complete information case, the nonzero-sum PE game control strategy is designed by solving the coupled Riccati recursions. Then, aiming at the incomplete information case, a control gain estimator is established, which lays the foundation for the control strategy design of the pursuit spacecraft. On the basis of the estimated control gain, the pursuit control strategy is solved by using the standard discrete-time Riccati recursion. In order to further get rid of the system information, a Q-learning-based control gain is designed for the pursuit spacecraft. Finally, a numerical example on the PE spacecraft system is provided to verify the effectiveness of the proposed PE game control strategies.

Model-free optimal tracking policies for Markov jump systems by solving non-zero-sum games

This paper develops model-free optimal tracking policies for Markov jump systems by solving non-zero-sum games (NZSGs). First, coupled action and mode-dependent value functions (CAMDVFs) are built for solving a two-player NZSG and getting Nash equilibrium solutions. Second, we propose a value iteration (VI) algorithm to parallelly update policies under each mode by collecting data on different operation modes within each iterative window. Moreover, the iterative increasing convergence of the CAMDVFs is proved by introducing auxiliary functions between two adjacent iterations. It is worth pointing out that an influence function is introduced to remove abnormal data to improve the learning capability of the VI algorithm effectively. Finally, the tracking policies' validity, self-adaptability and application potential are verified by a numerical example and a generalized economic model.

A Stochastic Control Approach for Constrained Stochastic Differential Games with Jumps and Regimes

We develop an approach for two-player constraint zero-sum and nonzero-sum stochastic differential games, which are modeled by Markov regime-switching jump-diffusion processes. We provide the relations between a usual stochastic optimal control setting and a Lagrangian method. In this context, we prove corresponding theorems for two different types of constraints, which lead us to find real-valued and stochastic Lagrange multipliers, respectively. Then, we illustrate our results for a nonzero-sum game problem with the stochastic maximum principle technique. Our application is an example of cooperation between a bank and an insurance company, which is a popular, well-known business agreement type called Bancassurance.

Fuzzy logic nonzero-sum game-based distributed approximated optimal control of modular robot manipulators with human-robot collaboration

In this paper, a fuzzy logic nonzero-sum game-based distributed approximated optimal control scheme is presented for modular robot manipulators (MRMs) with human-robot collaboration (HRC) tasks. The MRM dynamic model is formulated by using joint torque feedback (JTF) technique. Based on the differential game strategy, the optimal control problem of HRC task-oriented MRM systems is transformed into a nonzero-sum game problem of multiple robotic subsystems. By taking advantage of the adaptive dynamic programming (ADP) algorithm, the distributed approximate optimal control policy under HRC tasks is developed by a novel fuzzy logic nonzero-sum game manner for solving the coupled Hamilton–Jacobian (HJ) equation. The trajectory tracking error under HRC task of the closed-loop MRM system is proved to be ultimately uniformly bounded (UUB) using the Lyapunov theory. Finally, experiment results have been presented, which demonstrate the advantage and effectiveness of the developed method.