Discrete-time Affine Nonlinear Systems Research Articles

Discounted Near-Optimal Control of Affine Systems via a Progressive Cost Evolution Formulation

In this essay, we develop a combined value iteration (CVI) framework to address discounted optimal control problems for discrete-time affine nonlinear systems. First, generated by novel value iteration (NVI), the admissibility is investigated for the iterative control policy. Note that the relaxation factor leads to the adjustable convergence speed. Second, the constraint condition for the discount factor is established to guarantee the admissibility of the iterative control policy derived from stabilizing value iteration (SVI). In addition, the monotonicity is discussed for the iterative cost function sequence. More importantly, CVI is constructed based on NVI and SVI. Third, produced by CVI, the system stability under the evolving control sequence is ensured by the introduction of the attraction region. In the end, a numerical example is involved to confirm the related theoretical results.

Safe reinforcement learning for affine nonlinear systems with state constraints and input saturation using control barrier functions

This paper provides a novel safe reinforcement learning (RL) control algorithm to solve safe optimal problems for discrete-time affine nonlinear systems, while the safety and convergence of the control algorithm are proven. The algorithm is proposed based on an adjusted policy iteration (PI) framework using only the measured data along the system trajectories in the environment. The adjusted PI algorithm combines with the system predictive information. Unlike most PI algorithms, an effective method of obtaining an initial safe and stable control policy is given here. In addition, control barrier functions (CBFs) and an input constraint function are introduced to augment reward functions. And the monotonically nonincreasing property of the iterative value function maintains the safe set forward invariant in the PI framework. Moreover, the safety and convergence of the proposed algorithm are proven in theory. Then, the design and implementation of the proposed algorithm are presented based on the identifier-actor-critic structure, where neural networks are employed to approximate the system dynamics, the iterative control policy, and the iterative value function, respectively. Finally, the simulation results illustrate the effectiveness and safety of the proposed algorithm.

A New Approach to Finite-Horizon Optimal Control for Discrete-Time Affine Nonlinear Systems via a Pseudolinear Method

In this article, a new time-varying adaptivedynamic programming (ADP) algorithm is developed to solve finite-horizon optimal control problems for a class of discrete-time affine nonlinear systems. Inspired by the pseudolinear method, the nonlinear system can be approximated by a series of time-varying linear systems. In each iteration of the time-varying ADP algorithm, the optimal control law for the time-varying linear system is obtained. For an arbitrary initial state, it is proven that states of the time-varying linear systems converge to the states of discrete-time affine nonlinear systems. It is also shown that the iterative value functions and the iterative control laws converge to the optimal value function and the optimal control law, respectively. Finally, numerical results are presented to verify the effectiveness of the present method.

Adaptive Interleaved Reinforcement Learning: Robust Stability of Affine Nonlinear Systems With Unknown Uncertainty.

This article investigates adaptive robust controller design for discrete-time (DT) affine nonlinear systems using an adaptive dynamic programming. A novel adaptive interleaved reinforcement learning algorithm is developed for finding a robust controller of DT affine nonlinear systems subject to matched or unmatched uncertainties. To this end, the robust control problem is converted into the optimal control problem for nominal systems by selecting an appropriate utility function. The performance evaluation and control policy update combined with neural networks approximation are alternately implemented at each time step for solving a simplified Hamilton-Jacobi-Bellman (HJB) equation such that the uniformly ultimately bounded (UUB) stability of DT affine nonlinear systems can be guaranteed, allowing for all realization of unknown bounded uncertainties. The rigorously theoretical proofs of convergence of the proposed interleaved RL algorithm and UUB stability of uncertain systems are provided. Simulation results are given to verify the effectiveness of the proposed method.

Convergence of Recurrent Neuro-Fuzzy Value-Gradient Learning With and Without an Actor

In recent years, a gradient of the n-step temporal-difference [TD(λ)] learning has been developed to present an advanced adaptive dynamic programming (ADP) algorithm, called value-gradient learning [VGL(λ)]. In this paper, we improve the VGL(λ) architecture, which is called the “single adaptive actor network [SNVGL(λ)]” because it has only a single approximator function network (critic) instead of dual networks (critic and actor) as in VGL(λ). Therefore, SNVGL(λ) has lower computational requirements when compared to VGL(λ). Moreover, in this paper, a recurrent hybrid neuro-fuzzy (RNF) and a first-order Takagi-Sugeno RNF (TSRNF) are derived and implemented to build the critic and actor networks. Furthermore, we develop the novel study of the theoretical convergence proofs for both VGL(λ) and SNVGL(λ) under certain conditions. In this paper, mobile robot simulation model (model based) is used to solve the optimal control problem for affine nonlinear discrete-time systems. Mobile robot is exposed various noise levels to verify the performance and to validate the theoretical analysis.

Comment on “$$H_2$$ Filtering for Discrete-Time Affine Nonlinear Descriptor Systems”

The objective of Aliyu and Perrier (Circuits Syst Signal Process 31(3):927–943, 2012) is to give a $$H_2$$ filter design method for discrete-time affine nonlinear descriptor systems, where both singular and normal filters were considered. A sufficient condition for the solvability of this problem was given in terms of discrete-time Hamilton–Jacobi–Bellman equations, and in some special cases this condition was reduced to some linear matrix inequalities. However, this linear matrix inequality condition is not correct, which is pointed out in this note.

A neural network-based approach for solving quantized discrete-time H∞ optimal control with input constraint over finite-horizon

In this paper, an online neural network (NN) approach for solving H∞ optimal control problem is proposed for unknown affine nonlinear discrete-time systems with input quantization over finite-horizon. Different from value and policy iteration of traditional approximation dynamic programming (ADP) technology which always requires adequate number of iterations and more than one iteration loops to guarantee stability of the controlled systems and convergence of system states and control laws, an online NN-based finite-horizon H∞ constrained-input optimal control method is presented using actor-critic-disturbance structure which can be applied as time goes forward. The terminal cost function has been considered, though the value of system state converges to zero in the regulation problem over finite-horizon. Additionally, an input quantization has been implemented to eliminate the quantization error by using the dynamic quantizer in the control process. Moreover, an NN identification strategy is presented to obviate the dependance of the system input dynamics. The stability analysis of the proposed control algorithm is provided by using Lyapunov stability theorem. Finally, a simulation example is given to verify the feasibility and effectiveness of designed control algorithm.

Off-Policy Interleaved Q -Learning: Optimal Control for Affine Nonlinear Discrete-Time Systems.

In this paper, a novel off-policy interleaved Q-learning algorithm is presented for solving optimal control problem of affine nonlinear discrete-time (DT) systems, using only the measured data along the system trajectories. Affine nonlinear feature of systems, unknown dynamics, and off-policy learning approach pose tremendous challenges on approximating optimal controllers. To this end, on-policy Q-learning method for optimal control of affine nonlinear DT systems is reviewed first, and its convergence is rigorously proven. The bias of solution to Q-function-based Bellman equation caused by adding probing noises to systems for satisfying persistent excitation is also analyzed when using on-policy Q-learning approach. Then, a behavior control policy is introduced followed by proposing an off-policy Q-learning algorithm. Meanwhile, the convergence of algorithm and no bias of solution to optimal control problem when adding probing noise to systems are investigated. Third, three neural networks run by the interleaved Q-learning approach in the actor-critic framework. Thus, a novel off-policy interleaved Q-learning algorithm is derived, and its convergence is proven. Simulation results are given to verify the effectiveness of the proposed method.

Iterative Learning Control for discrete nonlinear systems with randomly iteration varying lengths

This note proposes ILC for discrete-time affine nonlinear systems with randomly iteration varying lengths. No prior information on the probability distribution of random iteration length is required prior for controller design. The conventional P-type update law is used with a modified tracking error because of randomly iteration varying lengths. A novel technical lemma is proposed for the strict convergence analysis in pointwise sense. An illustrative example verifies the theoretical results.

Neural network-based online H∞ control for discrete-time affine nonlinear system using adaptive dynamic programming

In this paper, the problem of H∞ control design for affine nonlinear discrete-time systems is addressed by using adaptive dynamic programming (ADP). First, the nonlinear H∞ control problem is transformed into solving the two-player zero-sum differential game problem of the nonlinear system. Then, the critic, action and disturbance networks are designed by using neural networks to solve online the Hamilton–Jacobi–Isaacs (HJI) equation associating with the two-player zero-sum differential game. When novel weight update laws for the critic, action and disturbance networks are tuned online by using data generated in real-time along the system trajectories, it is shown that the system states, all neural networks weight estimation errors are uniformly ultimately bounded by using Lyapunov techniques. Further, it is shown that the output of the action network approaches the optimal control input with small bounded error and the output of the disturbance network approaches the worst disturbance with small bounded error. At last, simulation results are presented to demonstrate the effectiveness of the new ADP-based method.

Adaptive Output Trajectory Tracking Control for a Class of Affine Nonlinear Discrete-Time Systems

In this paper, we develop an adaptive output trajectory tracking control (AOTTC) method for a class of affine nonlinear discrete-time systems. The controller designed by this AOTTC method can make real-system output track the given expected trajectory asymptotically. Our method has some advantages: 1) it requires relatively few assumptions about the system model; 2) it can simplify the control problem by dynamically linearizing the system model and designing and adjusting the feedback gain matrix to adaptively stabilize the system online; and 3) it can estimate the system parameters by using an optimization scheme without using vast amounts of sampled data. This designing and adjusting procedure of the feedback gain matrix has a clear physical meaning and is easily applied. We also give out the convergence condition of the AOTTC method. Then, a voltage-controllable permanent magnet linear motor system is employed for computer simulation, whose results demonstrate the feasibility of our control method.

A Krein Space Approach to $H_{\infty}$ Filtering of Discrete-Time Nonlinear Systems

In this paper, a Krein space approach to finite horizon H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> filtering is proposed for a class of affine nonlinear discrete-time systems. It is shown that the problem of H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> nonlinear filtering can be converted into a minimum of an indefinite quadratic form. Hence, a relationship between H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> nonlinear filter in Hilbert space and nonlinear estimation in Krein space is established. By using first-order Taylor approximation and Krein space projection, a sufficient and necessary condition for the minimum is derived. Moreover, a feasible solution of the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> nonlinear filter can be obtained by recursively computing Riccati recursions. Finally, a numerical example and one kind of integration filter are used to demonstrate the effectiveness of the proposed method.

Online adaptive policy learning algorithm for H∞ state feedback control of unknown affine nonlinear discrete-time systems.

The problem of H∞ state feedback control of affine nonlinear discrete-time systems with unknown dynamics is investigated in this paper. An online adaptive policy learning algorithm (APLA) based on adaptive dynamic programming (ADP) is proposed for learning in real-time the solution to the Hamilton-Jacobi-Isaacs (HJI) equation, which appears in the H∞ control problem. In the proposed algorithm, three neural networks (NNs) are utilized to find suitable approximations of the optimal value function and the saddle point feedback control and disturbance policies. Novel weight updating laws are given to tune the critic, actor, and disturbance NNs simultaneously by using data generated in real-time along the system trajectories. Considering NN approximation errors, we provide the stability analysis of the proposed algorithm with Lyapunov approach. Moreover, the need of the system input dynamics for the proposed algorithm is relaxed by using a NN identification scheme. Finally, simulation examples show the effectiveness of the proposed algorithm.

Zero-Sum Two-Player Game Theoretic Formulation of Affine Nonlinear Discrete-Time Systems Using Neural Networks

In this paper, the nearly optimal solution for discrete-time (DT) affine nonlinear control systems in the presence of partially unknown internal system dynamics and disturbances is considered. The approach is based on successive approximate solution of the Hamilton-Jacobi-Isaacs (HJI) equation, which appears in optimal control. Successive approximation approach for updating control and disturbance inputs for DT nonlinear affine systems are proposed. Moreover, sufficient conditions for the convergence of the approximate HJI solution to the saddle point are derived, and an iterative approach to approximate the HJI equation using a neural network (NN) is presented. Then, the requirement of full knowledge of the internal dynamics of the nonlinear DT system is relaxed by using a second NN online approximator. The result is a closed-loop optimal NN controller via offline learning. A numerical example is provided illustrating the effectiveness of the approach.

Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm

In this paper, we solve the zero-sum game problems for discrete-time affine nonlinear systems with known dynamics via iterative adaptive dynamic programming algorithm. First, a greedy heuristic dynamic programming iteration algorithm is developed to solve the zero-sum game problems, which can be used to solve the Hamilton–Jacobi–Isaacs equation associated with H∞ optimal regulation control problems. The convergence analysis in terms of value function and control policy is provided. To facilitate the implementation of the algorithm, three neural networks are used to approximate the control policy, the disturbance policy, and the value function, respectively. Then, we extend the algorithm to H∞ optimal tracking control problems through system transformation. Finally, two simulation examples are presented to demonstrate the effectiveness of the proposed scheme.

Optimal control for discrete-time affine non-linear systems using general value iteration

In this study, the authors propose a novel adaptive dynamic programming scheme based on general value iteration (VI) to obtain near optimal control for discrete-time affine non-linear systems with continuous state and control spaces. First, the selection of initial value function is different from the traditional VI, and a new method is introduced to demonstrate the convergence property and convergence speed of value function. Then, the control law obtained at each iteration can stabilise the system under some conditions. At last, an error-bound-based condition is derived considering the approximation errors of neural networks, and then the error between the optimal and approximated value functions can also be estimated. To facilitate the implementation of the iterative scheme, three neural networks with Levenberg-Marquardt training algorithm are used to approximate the unknown system, the value function and the control law. Two simulation examples are presented to demonstrate the effectiveness of the proposed scheme.

Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update.

In this paper, the Hamilton-Jacobi-Bellman equation is solved forward-in-time for the optimal control of a class of general affine nonlinear discrete-time systems without using value and policy iterations. The proposed approach, referred to as adaptive dynamic programming, uses two neural networks (NNs), to solve the infinite horizon optimal regulation control of affine nonlinear discrete-time systems in the presence of unknown internal dynamics and a known control coefficient matrix. One NN approximates the cost function and is referred to as the critic NN, while the second NN generates the control input and is referred to as the action NN. The cost function and policy are updated once at the sampling instant and thus the proposed approach can be referred to as time-based ADP. Novel update laws for tuning the unknown weights of the NNs online are derived. Lyapunov techniques are used to show that all signals are uniformly ultimately bounded and that the approximated control signal approaches the optimal control input with small bounded error over time. In the absence of disturbances, an optimal control is demonstrated. Simulation results are included to show the effectiveness of the approach. The end result is the systematic design of an optimal controller with guaranteed convergence that is suitable for hardware implementation.

ℋ ∞ filtering for discrete-time affine non-linear descriptor systems

In this study, the authors consider the ℋ∞-filtering problem for discrete-time affine non-linear descriptor systems. Two types of filters are discussed; namely, (i) singular, (ii) normal and sufficient conditions for the solvability of the problem in terms of disrete-time Hamilton–Jacobi–Isaacs equations (DHJIEs) are presented. The results are also specialised to linear systems, in which case the DHJIEs reduce to a system of linear-matrix-inequalities (LMIs). Examples are also presented to illustrate the results.

$\mathcal{H}_{2}$ Filtering for Discrete-Time Affine Nonlinear Descriptor Systems

In this paper, we consider the Kalman (or \(\mathcal{H}_{2}\))-filtering problem for discrete-time affine nonlinear descriptor systems. Two types of filter are discussed, namely, (i) singular; and (ii) normal. Sufficient conditions for the solvability of the problem in terms of discrete-time Hamilton–Jacobi–Bellman equations (DHJBEs) are presented. The results are also specialized to linear systems in which case the DHJBEs reduce to a system of linear matrix-inequalities (LMIs). Examples are also presented to illustrate the results.

Online optimal control of nonlinear discrete-time systems using approximate dynamic programming

In this paper, the optimal control of a class of general affine nonlinear discrete-time (DT) systems is undertaken by solving the Hamilton Jacobi-Bellman (HJB) equation online and forward in time. The proposed approach, referred normally as adaptive or approximate dynamic programming (ADP), uses online approximators (OLAs) to solve the infinite horizon optimal regulation and tracking control problems for affine nonlinear DT systems in the presence of unknown internal dynamics. Both the regulation and tracking controllers are designed using OLAs to obtain the optimal feedback control signal and its associated cost function. Additionally, the tracking controller design entails a feedforward portion that is derived and approximated using an additional OLA for steady state conditions. Novel update laws for tuning the unknown parameters of the OLAs online are derived. Lyapunov techniques are used to show that all signals are uniformly ultimately bounded and that the approximated control signals approach the optimal control inputs with small bounded error. In the absence of OLA reconstruction errors, an optimal control is demonstrated. Simulation results verify that all OLA parameter estimates remain bounded, and the proposed OLA-based optimal control scheme tunes itself to reduce the cost HJB equation.