Discrete-time Control Problem Research Articles

Discrete-time hybrid control with risk-sensitive discounted costs

AbstractThis paper studies discrete-time hybrid control problems where the controller’s payoff function follows a risk-sensitive discounted model. The associated discount factor can depend on state and action variables, which can even take values of zero or one. We prove the existence of optimal control policies under two different hypotheses. The following technique is the dynamic programming method, where we characterize the minimum cost as the solution of the so-called dynamic programming equations and find the above optimal policies through these equations. We illustrate our theory with practical examples involving inventory-manufacturing systems, and pollution management.

Formation control of high-order linear discrete-time multiple unmanned aerial vehicles systems in noisy environments

Formation control of high-order linear discrete-time multiple unmanned aerial vehicles (multi-UAVs) systems with multiplicative noises is studied in this article. Because of the existence of multiplicative noises, classical methods for discrete-time system control problems should be improved. Here, we develop a new discrete-time formation protocol for multi-UAVs systems. The stochastic Lyapunov stability theory and the positive definiteness theory of a generalized algebraic Riccati equation are then used to derive sufficient conditions for formation feasibility.

Advanced Optimal Tracking Control With Stability Guarantee via Novel Value Learning Formulation.

In this article, to solve the optimal tracking control problem (OTCP) for discrete-time (DT) nonlinear systems, general value iteration (GVI) scheme and online value iteration (VI) algorithms with novel value function are discussed. First, the disadvantage of the traditional value function for the OTCP is presented and the novel value function is introduced. Second, we analyze the monotonicity and convergence of GVI and establish the admissibility condition of GVI to evaluate the admissibility of the current iterative control. Note that a novel approach is introduced to analyze the admissibility. Third, based on the attraction domain, improved control policies with online VI can be obtained by judging the location of the current tracking error and reference point. Finally, the stability of the online VI-based control system is guaranteed. Besides, we provide two simulation examples to show the performance of the proposed methods.

Dissipativity in Infinite Horizon Optimal Control and Dynamic Programming

In this paper we extend dynamic programming techniques to the study of discrete-time infinite horizon optimal control problems on compact control invariant sets with state-independent best asymptotic average cost. To this end we analyse the interplay of dissipativity and optimal control, and propose novel recursive approaches for the solution of so called shifted Bellman Equations.

UKF-Based Optimal Tracking Control for Uncertain Dynamic Systems With Asymmetric Input Constraints.

To enhance system robustness in the face of uncertainty and achieve adaptive optimization of control strategies, a novel algorithm based on the unscented Kalman filter (UKF) is developed. This algorithm addresses the finite-horizon optimal tracking control problem (FHOTCP) for nonlinear discrete-time (DT) systems with uncertainty and asymmetric input constraints. An augmented system is constructed with asymmetric control constraints being considered. The augmented problem is addressed with a DT Hamilton-Jacobi-Bellman equation (DTHJBE). By analyzing convergence with regard to the cost function and control law, the UKF-based iterative adaptive dynamic programming (ADP) algorithm is proposed. This algorithm approximates the solution of the DTHJBE, ensuring that the cost function converges to its optimal value within a bounded range. To execute the UKF-based iterative ADP algorithm, the actor-estimator-critic framework is built, in which the estimator refers to system state estimation through the application of UKF. Ultimately, simulation examples are presented to show the performance of the proposed method.

Decentralized Output-Feedback Control With Asymmetric One-Step Delayed Information

This paper considers the decentralized LQG control problem for discrete-time decentralized system controlled by two players. In this scenario, player 1 shares a unit delayed observations and control inputs with the controller of player 2, whereas due to the limiting capacity, the controller of player 1 cannot obtain the observations and control inputs of player 2 which leads to the asymmetric one-step delay information. It should be emphasized that this structure makes the classical separation principle fail. Under the assumption of linear control strategies, we derive the optimal estimators based on asymmetric information. In virtue of the Pontryagin's maximum principle, the optimal explicit decentralized controllers are obtained by solving the forward and backward stochastic difference equations (FBSDEs). It is noted that the control gains are coupled with the estimation gains. Moreover, the estimation gains satisfy the forward Riccati equations and the control gains follow the backward Riccati equations which cannot be solved simultaneously. To this end, we present iterative solutions to the coupled forward and backward Riccati equations. Finally, a sufficient condition for the stabilization problem is given in terms of the coupled algebraic Riccati equations. Numerical examples are illustrated to show the effectiveness of the proposed algorithm.

Optimal guaranteed cost control of discrete-time uncertain systems subjected to state saturation nonlinearity

This paper investigates the optimal guaranteed cost (GC) control problem for uncertain discrete-time (DT) systems with state saturation nonlinearities via static-state feedback. The parameter uncertainties in the system state and input matrix are considered to be norm bounded. Utilizing Lyapunov theory and sector-based characterization of saturation nonlinearities, a novel criterion for the existence of GC controllers is proposed. The criterion is expressed in the framework of linear matrix inequalities (LMIs) and is thus computationally tractable. The proposed GC controllers make the closed-loop (CL) system asymptotically stable and ensure an adequate level of performance for all admissible parameter uncertainties. To design an optimum GC controller, a convex optimization problem with LMI constraints is formulated. The effectiveness of the proposed approach is illustrated by examples.

Discrete time adaptive neural network control for WME and compression refrigeration systems

In this paper, we study the discrete-time control problem for the refrigeration system with unmodeled dynamics. This paper proposes discrete adaptive neural network controllers for two refrigeration systems, including the water membrane evaporator cooling and compression refrigeration systems. The influence of model uncertainty on system performance can be eliminated effectively by designing the neural network and the corresponding discrete-time adaptive updating strategy. Thus, the model parameters in the refrigeration system are not required in our control algorithms. Simulation results show that the proposed strategy can effectively adjust the refrigeration system temperatures. Compared with the traditional PID control strategy, the system response overshoot under our method can be reduced by 0.3%, and the system settling time is reduced by at least 94.3%.

Sensitivity updates for linear-quadratic optimization problems in multi-step model predictive control

The paper discusses how parametric sensitivity analysis can be used in certain model predictive control (MPC) schemes. The sensitivity analysis will be performed with regard to the initial state measurement and update schemes will be derived that speed-up the computations. Throughout we restrict the discussion to linear-quadratic optimal control problems in discrete time, which frequently arise in tracking tasks with MPC. The derived tools from sensitivity analysis can be embedded into MPC schemes with a prediction step and multi-step MPC schemes with re-optimization and prediction step. Numerical experiments illustrating the sensitivity analysis are presented.

Premium control with reinforcement learning

AbstractWe consider a premium control problem in discrete time, formulated in terms of a Markov decision process. In a simplified setting, the optimal premium rule can be derived with dynamic programming methods. However, these classical methods are not feasible in a more realistic setting due to the dimension of the state space and lack of explicit expressions for transition probabilities. We explore reinforcement learning techniques, using function approximation, to solve the premium control problem for realistic stochastic models. We illustrate the appropriateness of the approximate optimal premium rule compared with the true optimal premium rule in a simplified setting and further demonstrate that the approximate optimal premium rule outperforms benchmark rules in more realistic settings where classical approaches fail.

Mean Field Markov Decision Processes

We consider mean-field control problems in discrete time with discounted reward, infinite time horizon and compact state and action space. The existence of optimal policies is shown and the limiting mean-field problem is derived when the number of individuals tends to infinity. Moreover, we consider the average reward problem and show that the optimal policy in this mean-field limit is varepsilon -optimal for the discounted problem if the number of individuals is large and the discount factor close to one. This result is very helpful, because it turns out that in the special case when the reward does only depend on the distribution of the individuals, we obtain a very interesting subclass of problems where an average reward optimal policy can be obtained by first computing an optimal measure from a static optimization problem and then achieving it with Markov Chain Monte Carlo methods. We give two applications: Avoiding congestion an a graph and optimal positioning on a market place which we solve explicitly.

Invariant ideals and their applications to the turnpike theory

AbstractIn this paper, the turnpike property is established for a nonconvex optimal control problem in discrete time. The functional is defined by the notion of the ideal convergence and can be considered as an analogue of the terminal functional defined over infinite-time horizon. The turnpike property states that every optimal solution converges to some unique optimal stationary point in the sense of ideal convergence if the ideal is invariant under translations. This kind of convergence generalizes, for example, statistical convergence and convergence with respect to logarithmic density zero sets.

Nonlinear Impulsive Control Design for Biologically Grounded NSM Tumor Growth Model Using Exact Linearized Mapping

This paper investigates the problem of personalized chemotherapy scheduling using a system-modeling approach based on a nonlinear impulsive control technique. In particular, this work deals with the so-called Norton-Simon-Massagué (NSM) model, a minimal biologically grounded model for controlled tumor growth, which has been accepted in clinical oncology for breast cancer treatment. In this paper, we continue this line of research, to advance the concept of closed-loop control a step further towards widespread clinical use. In the present work, the problem of chemotherapy drug scheduling is approached using an impulsive control strategy combined with a state-space observer. The technical challenges of the nonlinearity of the controlled system and the impulsive nature of control input are handled via the utilization of a novel mapping that transforms the control problem into a “simplified” linear discrete-time control problem. Theoretical guarantees regarding the sign and invertibility of the proposed transformation are demonstrated. Then, an observer-based state feedback controller is proposed for chemotherapy drug scheduling as a proof-of-concept control strategy. Simulation results showing the performance of the designed closed-loop controlled chemotherapy schedules in reducing and eradicating tumor volume are provided using model's parameters derived using real tumor data and Doxirubicin drug.

Optimal Consensus Control Design for Multiagent Systems With Multiple Time Delay Using Adaptive Dynamic Programming.

In this article, a novel data-based adaptive dynamic programming (ADP) method is presented to solve the optimal consensus tracking control problem for discrete-time (DT) multiagent systems (MASs) with multiple time delays. Necessary and sufficient conditions of the corresponding equivalent time-delay system are provided on the basis of the causal transformations. Benefitting from the construction of tracking error dynamics, the optimal tracking problem can be transformed into settling the Nash-equilibrium in the graphical game, which can be completed by solving the coupled Hamilton-Jacobi (HJ) equations. An error estimator is introduced to construct the tracking error of the MASs only using the input and output (I/O) data. Therefore, the designed data-based ADP algorithm can minimize the cost functions and ensure the consensus of MASs without the knowledge of system dynamics. Finally, a numerical example is given to demonstrate the effectiveness of the proposed method.

Linear programming based optimality conditions and approximate solution of a deterministic periodic optimal control problem in discrete time

This paper is focused on a discrete time periodic optimal control problem. We establish linear programming based optimality conditions and present a method for the construction of near optimal controls. A numerical example is included to demonstrate the results.

Input–Output Finite-Time Stabilization of Linear Time-Varying Discrete-Time Systems

This article deals with the input–output finite-time stability (IO-FTS) of linear discrete-time systems. In current control science, discrete-time systems play a very important role in many engineering contexts. Moreover, many discrete-time control problems are defined over a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">finite</i> -interval of time; therefore, the development and application of finite-time control methodologies is of particular relevance. The first contribution of this article is a pair of necessary and sufficient conditions for IO-FTS. The former involves the solution of a set of generalized difference Lyapunov equations; the latter allows one to establish the feasibility of an optimization problem by solving a set of difference linear matrix inequalities (DLMIs). The second contribution of this article is a theorem for IO finite-time stabilization via state feedback, followed by a more general one for stabilization via output feedback. Both conditions are necessary and sufficient, and lead to optimization problems cast in the form of DLMIs. The applicability of the devised results is illustrated through a numerical example.

Sensitivity analysis in parametric multiobjective discrete-time control via Fréchet subdifferential calculus of the frontier map

This paper studies the first-order behavior of the frontier map of a vector parametric control problem with nonconvex objective functions, nonlinear state equations and control constraints. By establishing an abstract result on the Fréchet subdifferential of the frontier map of a vector parametric optimization problem, we derive a formula for computing the Fréchet subdifferential of the frontier map to a parametric multiobjective discrete-time control problem. Examples are given to illustrate the obtained results.

Maximum principle for discrete-time stochastic control problem of mean-field type

In this work, a discrete-time mean-field type stochastic optimal control problem is studied. The goal is to derive the stochastic maximum principle with convex control domains. L-derivative is applied to handle the mean-field term and a technique of adjoint operator is used to overcome the difficulties of obtaining adjoint equations and duality relation. Then, the stochastic maximum principle for discrete-time mean-field type stochastic optimal control problem is established. Finally, as an illustration of our stochastic maximum principle, a discrete-time mean–variance portfolio selection problem is solved with the decoupling technique which is different from the continuous-time case.

Fast Online Optimization for Terrain-Blind Bipedal Robot Walking With a Decoupled Actuated SLIP Model.

We present an online optimization algorithm which enables bipedal robots to blindly walk over various kinds of uneven terrains while resisting pushes. The proposed optimization algorithm performs high-level motion planning of footstep locations and center-of-mass height variations using the decoupled actuated spring-loaded inverted pendulum (aSLIP) model. The decoupled aSLIP model simplifies the original aSLIP with linear inverted pendulum (LIP) dynamics in horizontal states and spring dynamics in the vertical state. The motion planning can be formulated as a discrete-time model predictive control (MPC) problem and solved at a frequency of 1 kHz. The output of the motion planner is fed into an inverse-dynamics–based whole body controller for execution on the robot. A key result of this controller is that the feet of the robot are compliant, which further extends the robot’s ability to be robust to unobserved terrain variations. We evaluate our method in simulation with the bipedal robot SLIDER. The results show that the robot can blindly walk over various uneven terrains including slopes, wave fields, and stairs. It can also resist pushes of up to 40 N for a duration of 0.1 s while walking on uneven terrains.

Optimal control for multistage uncertain random dynamic systems with multiple time delays

A complex system exhibiting uncertainty as well as randomness is an uncertain random dynamic system. A novel idea of an uncertain random matrix is introduced as an interaction between uncertainty and randomness. Thereby a time-delayed uncertain random dynamic system is formulated. And then the corresponding time-delayed optimal control problem is first presented in this paper. To tackle a problem like that, a method is proposed to translate it into an equivalent optimization problem without time delay. The relevant recursion equations are developed with the dynamic programming technique to solve the converted optimization problem. Solving the related recursion equations yields the answer to such an optimization problem. The presented approach offers a unified framework for tackling the discrete-time uncertain random optimal control problem involving time delays. In this framework, three algorithms are successfully devised in doing with time-delayed optimal control problems for dynamic systems regarding linear, quadratic, and cubic controls. Meanwhile, the optimal solutions for numerical examples are found by the raised algorithms. Finally, a problem of optimal control for a two-spool turbofan engine is investigated as an application of our results.