Security consensus control for nonlinear stochastic multi-agent systems via reinforcement learning

The optimal control of nonlinear stochastic systems is considered in this paper. The central role played by the Fokker-Planck-Kolmogorov equation in the stochastic control problem is shown under the assumption of asymptotic stability. A computational approach for the problem is devised based on policy iteration/ successive approximations, and a finite dimensional approximation of the control parametrized diusion operator, i.e., the controlled Fokker-Planck operator. Several numerical examples are provided to show the ecacy of the proposed computational methodology. require the study of the time evolution of the Probability Density Function (PDF), p(t,x), corresponding to the state, x, of the relevant dynamic system. The pdf is given by the solution to the Fokker-Planck- Kolmogorov equation (FPE), which is a PDE in the pdf of the system, defined by the underlying dynamical system's parameters. In this paper, approximate solutions of the FPE are considered, and subsequently leveraged for the design of controllers for nonlinear stochastic dynamical systems. In the past few years, the authors have developed a generalized multi-resolution meshless FEM methodology, partition of unity FEM (PUFEM), utilizing the recently developed GLOMAP (Global Local Orthogonal MAPpings) methodology to provide the partition of unity functions 6,7 and the orthogonal local basis functions, for the solution of the FPE. The PUFEM is a Galerkin projection method and the solution is characterized in terms of an finite dimensional representation of the Fokker-Planck operator underlying the problem. The methodology is also highly amenable to parallelization. 8-12 Though the FPE is invaluable in quantifying the uncertainty evolution through nonlinear systems, per- haps its greatest benefit may be in the stochastic analysis, design and control of nonlinear systems. In the context of nonlinear stochastic control, Markov Decision Processes have long been one of the most widely used methods for discrete time stochastic control. However, the Dynamic Programming equations under- lying the MDPs suer from the curse of dimensionality. 13-15 Various approximate Dynamic Programming (ADP) methods have been proposed in the past several years for overcoming the curse of dimensionality, 15-19 and broadly can be categorized under the category of functional reinforcement learning. These methods are essentially model free method of approximating the optimal control policy in stochastic optimal control prob- lems. These methods generally fall under the category of value function approximation methods, 15 policy gradient/ approximation methods 17,18 and actor-critic methods. 16,19 These methods attempt to reduce the dimensionality of the DP problem through a compact parametrization of the value functions (with respect to

© 2017, Editorial Department of Journal of Drainage and Irrigation Machinery Engineering. All right reserved. Most nonlinear control systems are inevitably subject to random disturbances, such as systematic measurements and random noises(random vibrations or shocks) in practice, which affect the control of nonlinear systems. In this paper, a stochastic distributed control method is designed for nonlinear systems subject to random perturbations. In this method, the relationship between steady-state response probability density distribution and control target of a nonlinear stochastic system is studied. The control design is divided into two steps: firstly, the actual model with stochastic perturbation is transformed into the nonlinear system Hamiltonian model; then the output of the controlled system sa-tisfies with a prescribed probability density distribution by using a technique for solving exact stationary solution of a nonlinear stochastic system. The convergence of control system is achieved by introducing the Lyapelov function in which the output of a closed-loop nonlinear stochastic system can converge to a pre-defined steady PDF to ensure the closed-loop stability of the controlled system. The results show that the proposed method is effective and can make the controlled system be able to track a pre-defined target steady probability distribution.

With the development of science and technology, practical systems such as the power systems, traffic systems, robot manipulator systems, etc., have become more complex. Therefore, it is difficult to build practical systems by accurate models. Under the lack of accurate process models, using system data to improve system performance and learn optimal decisions becomes very important. Through the recent years, data-based learning control theories and technologies have widely been investigated, including adaptive dynamic programming, reinforcement learning, iterative learning control, and so on. Data-based methods require the system data instead of the accurate knowledge of system dynamics that can be considered as model-free learning control methods. The data-based methods are effective solutions for the optimal control of nonlinear systems, which motivate this special issue. This special issue aims to collect and present original research dealing with data-based learning and their applications for optimization and control problems. The first group of papers1-7 focuses on data-based control theory, approaches, and applications. A fuzzy model predictive control approach is proposed for stick-slip type piezoelectric actuator to realize the precise control of the end effector.1 A systematic online adaptive dynamic programming control framework is proposed for smart buildings control to ensure hard constraints to be satisfied.2 A multi-verse optimizer tuned PI-type active disturbance rejection generalized predictive control method is described for the motion control problems of ships.3 The sufficient optimality conditions for the optimal controls are established under some convexity assumptions.4 A receding-horizon reinforcement learning algorithm is proposed for near-optimal control of continuous-time systems under control constraints.5 In order to solve the interference compensation control problem of a class of nonlinear systems, a method based on memory data is introduced to suppress interference greatly.6 A new controller design method is proposed for the trajectory tracking problem of robots with imprecise dynamic properties and interference.7 The second group of papers8-12 considers iterative learning identification and iterative learning control. An iterative learning control approach is proposed for linear parabolic distributed parameter systems with multiple actuators and multiple sensors.8 The quantized data-based iterative learning tracking control problem is studied for nonlinear networked control systems with signals quantization and denial-of-service attacks.9 The output tracking problem is considered for a class of nonlinear parabolic distributed parameter systems with moving boundaries.10 A just-in-time learning based dual heuristic programming algorithm is proposed to optimize the control performance of autonomous wheeled mobile robots under faults or disturbances.11 A novel optimal constraint-following controller is proposed for uncertain mechanical systems.12 The third group of papers13-19 focuses on robustness on data-based optimal learning control. A novel Nash game-theoretical optimal adaptive robust control design approach is proposed to address the constraint-following control problem for the uncertain underactuated mechanical systems with fuzzy evidence theory.13 A partial model-free sliding mode control strategy is proposed for a class of disturbed systems.14 A new data-based adaptive dynamic programming algorithm is proposed to solve the optimal control policy for discrete-time systems with uncertainties.15 A method that applies event-triggered mechanism H ∞ $$ {\mathrm{H}}_{\infty } $$ control to continuous-time nonlinear systems with asymmetric constraints based on dual heuristic dynamic programming structure is proposed.16 A novel anti-disturbance inverse optimal controller design method is proposed for a class of high-dimensional chain structure systems with any disturbances, matched, or mismatched.17 A data-driven H ∞ $$ {\mathrm{H}}_{\infty } $$ controller design method is studied for continuous-time linear periodic systems.18 The problem of the post-stall pitching maneuver of an aircraft with lower deflection frequency of control actuator is studied by considering the unsteady aerodynamic disturbances.19 The fourth group of papers20-23 focuses on neural networks and deep neural networks learning methods for optimal control. An optimal tracking control problem for the injection flow front position arising in the filling process in the injection molding machine is considered, and an intelligent real-time optimal control method based on deep neural networks is developed for the online tracking of the flow front position to improve the efficient production process of the plastics.20 An efficient and systematic method is proposed for model-based predictive control synthesis.21 The decentralized control issues of nonlinear large-scale systems are investigated via critic-only adaptive dynamic programming learning methods.22 A singularity-free online neural network-based sliding mode control method is proposed to realize the fixed-wing perch maneuver.23 The fifth group of papers24-27 discusses data-based control for distributed control systems. A mission-driven control scheme, including a consensus-based near-optimal formation controller and a finite-time precise formation controller, is proposed aiming at different requirements of unmanned aerial vehicle swarm.24 The neural network adaptive formation control of a class of second-order nonlinear systems with unmodeled dynamics is investigated, where the control law merely depends on the relative bearings between neighboring agents.25 The neighbor Q-learning based consensus control algorithm is developed for discrete-time multiagent systems.26 The fault-tolerate containment control problem is considered for stochastic nonlinear multiagent systems in the presence of input saturation and sensor faults.27 The sixth group of papers28-30 considers applications of data-based learning methods to industrial processes. A stochastic gradient algorithm based on the minimum Shannon entropy is proposed to identify a type of Hammerstein system with random noise.28 A predictive control strategy based on Hammerstein–Wiener inverse model compensation is proposed aiming at the nonlinearity and large lag of the pH change in wet flue gas desulfurization process.29 An algorithm called the kernel entropy regression is proposed to enhance the interpretability between the fault and the key performance indicator.30 The seventh group of papers31-36 focuses on machine learning, data mining, and practical applications in automation. The performance of a Takagi–Sugeno fuzzy-model-based observer is enhanced by proposing a featured multi-instant united switch-type observer.31 The reinforcement learning theory with deep Q-network is applied for the mobile robot to achieve a collision-free path in an unknown dynamic environment.32 An energy-saving velocity planning algorithm is proposed for rail transit train with running and computation delays.33 A novel COVID-19 transmission model is established by introducing traditional susceptible–exposed–infected–removed disease transmission models into complex network.34 A novel collaborative diagnosis method is presented by combining variational modal decomposition and stochastic configuration network for incipient faults of rolling bearing.35 The linear dependence graph associated with a finite-dimensional vector space is studied.36 In summary, this special issue provides an opportunity to review the most recent developments in data-based learning control for optimization of nonlinear systems, by considering theory, algorithms, and applications.

Observer-Based Adaptive Fuzzy Tracking Control for Stochastic Nonlinear Multi-Agent Systems with Dead-Zone Input

This article focuses on the state-feedback ℋ∞ control problem for the stochastic nonlinear systems with state and disturbance-dependent noise and time-varying state delays. Based on the maxmin optimisation approach, both the delay-independent and the delay-dependent Hamilton–Jacobi-inequalities (HJIs) are developed for synthesising the state-feedback ℋ∞ controller for a general type of stochastic nonlinear systems. It is shown that the resulting control system achieves stochastic stability in probability and the prescribed disturbance attenuation level. For a class of stochastic affine nonlinear systems, the delay-independent as well as delay-dependent matrix-valued inequalities are proposed; the resulting control system satisfies global asymptotic stability in the mean-square sense and the required disturbance attenuation level. By modelling the nonlinearities as uncertainties in corresponding stochastic time-delay systems, the sufficient conditions in terms of a linear matrix inequality (LMI) and a bilinear matrix inequality (BMI) are derived to facilitate the design of the state-feedback ℋ∞ controller. Finally, two numerical examples are provided to illustrate the effectiveness of the proposed methods.

An optimal bounded control strategy for smart structure systems as controlled Hamiltonian systems with random excitations and noised observations is proposed. The basic dynamic equations for a smart structure system with smart sensors and actuators are firstly given. The nonlinear stochastic control system with noised observations is then obtained from the simplified smart structure system, and the system is expressed by generalized Hamiltonian equations with control, random excitation and dissipative forces. The optimal control problem for nonlinear stochastic systems with noised observations includes two parts: optimal state estimation and optimal response control based on estimated states, which are coupled each other. The probability density of optimally estimated systems has generally infinite dimensions based on the separation theorem. The proposed optimal control strategy gives an approximate separate solution. First, the optimally estimated system state is determined by the observations based on the extended Kalman filter, and the estimated nonlinear system with controls and stochastic excitations is obtained which has finite-dimensional probability density. Second, the dynamical programming equation for the estimated system is determined based on the stochastic dynamical programming principle. The control boundedness due to actuator saturation is considered, and the optimal bounded control law is obtained by the programming equation with the bounded control constraint. The optimal control depends on the estimated system state which is determined by noised observations. The proposed optimal bounded control strategy is finally applied to a single-degree-of-freedom nonlinear stochastic system with control and noised observation. The remarkable vibration control effectiveness is illustrated with numerical results. Thus the proposed optimal bounded control strategy is promising for application to nonlinear stochastic smart structure systems with noised observations.

In this article, a prescribed-time state-feedback stabilization design strategy is proposed for a class of p-norm stochastic nonlinear strict feedback systems. In previous work on prescribed-time stabilization of stochastic systems, only stochastic nonlinear systems with fractional power less than or equal to one are considered. To overcome this problem, we improve the existing method and discuss the issue of prescribed-time stabilization of stochastic nonlinear systems with fractional power is arbitrary positive odd rational number. First, a prescribed-time controller is designed by combining the Lyapunov function with adding a power integrator technique. It should be pointed out that the homogeneous domination approach is adopted when dealing with the nonlinear terms of the system. Then, according to the stochastic prescribed-time stability theorem, it is proved that the designed controller can ensure the closed-loop system is prescribed-time mean-square stable. Finally, three simulation examples are given to investigate the validity of the presented method, in which the last one is an electromechanical system example.

In this article, the leaderless adaptive fuzzy consensus control problem is studied for a class of stochastic nonlinear multiagent systems with unknown measurement sensitivity under false data injection attacks. Unknown measurement sensitivity and false data injection attacks can prevent sensors from obtaining right state information and make it difficult to design controllers and adaptive laws. The existing works considered only one of these cases for deterministic systems. In this article, the coexistence of both cases is considered with the help of Nussbaum functions and fuzzy logic systems, and the corresponding controllers and auxiliary variables are not only codesigned to solve the problem, but also extended to stochastic multiagent systems. Then, an improved switching threshold event-triggered mechanism is proposed to reduce the communication burden of the control channel. Furthermore, the leaderless asymptotic consensus control scheme for stochastic multiagent systems is proposed. The boundedness of all signals and leaderless asymptotic consensus control performance are guaranteed via the Lyapunov stability theorem. Finally, two simulation examples are given to verify the effectiveness of the proposed control scheme.

Distributed containment output-feedback control for a general class of stochastic nonlinear multi-agent systems

A NARMAX model-based state-space self-tuning control for nonlinear stochastic hybrid systems

Optimized distributed formation control using identifier–critic–actor reinforcement learning for a class of stochastic nonlinear multi-agent systems

This paper presents a new state-space self-tuning control scheme for adaptive digital control of continuous-time multivariable nonlinear stochastic and chaotic systems, which have unknown system parameters, system and measurement noises, and inaccessible system states. Instead of using the moving average (MA)-based noise model commonly used for adaptive digital control of linear discrete-time stochastic systems in the literature, an adjustable auto-regressive moving average (ARMA)-based noise model with estimated states is constructed for state-space self-tuning control of nonlinear continuous-time stochastic systems. By taking advantage of a digital redesign methodology, which converts a predesigned high-gain analog tracker/observer into a practically implementable low-gain digital tracker/observer, and by taking the non-negligible computation time delay and a relatively longer sampling period into consideration, a digitally redesigned predictive tracker/observer has been newly developed in this paper for adaptive chaotic orbit tracking. The proposed method enables the development of a digitally implementable advanced control algorithm for nonlinear stochastic and chaotic hybrid systems.

Sufficient conditions for the controllability of nonlinear stochastic fractional boundary control systems are established. The equivalent integral equations are derived for both linear and nonlinear systems, and the control function is given in terms of the pseudoinverse operator. The Banach contraction mapping theorem is used to obtain the result. A controllability result for nonlinear stochastic fractional integrodifferential systems is also attained. Examples are included to illustrate the theory.

Some recent advances on the filtering and control problems for nonlinear stochastic complex systems with incomplete information are surveyed. The incomplete information under consideration mainly includes missing measurements, randomly varying sensor delays, signal quantization, sensor saturations, and signal sampling. With such incomplete information, the developments on various filtering and control issues are reviewed in great detail. In particular, the addressed nonlinear stochastic complex systems are so comprehensive that they include conventional nonlinear stochastic systems, different kinds of complex networks, and a large class of sensor networks. The corresponding filtering and control technologies for such nonlinear stochastic complex systems are then discussed. Subsequently, some latest results on the filtering and control problems for the complex systems with incomplete information are given. Finally, conclusions are drawn and several possible future research directions are pointed out.

In this paper, an adaptive neural network control for stochastic nonlinear systems with uncertain disturbances is proposed. The neural network is considered to approximate an uncertain function in a nonlinear system. And computational burden in operation is reduced by handling the norm of the neural-network vector. However, it will arise chattering issue, which is a challenge to avoid it from the symbolic operation. Further, traditional schemes often view error of estimate as bounded constant, but it is a time-varying function exactly, which may lead control schemes cannot conform to practical situation and guarantee stability of systems. Thus, backstepping technology and the neural network technology combined to stabilize stochastic nonlinear systems together to handle the aforementioned issues. It is proved that the proposed control scheme can guarantee the satisfactory asymptotic convergence performance and predetermined transient tracking error performance. From simulation results, the proposed control scheme is verified that can guarantee the satisfactory effectiveness.