State-feedback Policy Research Articles

Controlled descent training

SummaryIn this work, a novel and model‐based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g., stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally optimal and convergent training behavior. The novel algorithm, Controlled Descent Training (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

Policy Iteration-Based Learning Design for Linear Continuous-Time Systems Under Initial Stabilizing OPFB Policy.

Policy iteration (PI), an iterative method in reinforcement learning, has the merit of interactions with a little-known environment to learn a decision law through policy evaluation and improvement. However, the existing PI-based results for output-feedback (OPFB) continuous-time systems relied heavily on an initial stabilizing full state-feedback (FSFB) policy. It thus raises the question of violating the OPFB principle. This article addresses such a question and establishes the PI under an initial stabilizing OPFB policy. We prove that an off-policy Bellman equation can transform any OPFB policy into an FSFB policy. Based on this transformation property, we revise the traditional PI by appending an additional iteration, which turns out to be efficient in approximating the optimal control under the initial OPFB policy. We show the effectiveness of the proposed learning methods through theoretical analysis and a case study.

Explicit model predictive control through robust optimization

AbstractA strategy that calculates an explicit state feedback policy to regulate constrained uncertain discrete‐time uncertain linear systems is presented. We consider uncertain processes, affected by box‐bounded multiplicative uncertainty as well as bounded additive uncertainty with linear state and inputs constraints. The proposed method includes (i) the calculation of a terminal set constraint and (ii) the robust reformulation of state constraints in the prediction horizon. These features allow the derivation of the desired policy by solving a single multiparametric quadratic programming problem that guarantees feasible operation in the presence of uncertainty. Additionally, we employ variable and constraint elimination approaches to enhance the computational performance of the strategy. We demonstrate the steps and benefits of these developments with a numerical example and a chemical engineering case study.

A posteriori control densities: Imitation learning from partial observations

This paper treats a special case of the Imitation from Observations (IfO) problem. IfO is a generalisation of Imitation Learning from state-only demonstrations. Our treatment of IfO considers the case of feature-only demonstrations. This means that the full state is inaccessible for inference, and imitation must occur on the basis of a limited set of features. We refer to this setting as Imitation from Partial Observations (IfPO). This scenario has the advantage of allowing to address a wider variety of demonstrations, as well as solving the problem of heteromorphic student and teacher. We set out for policy learning methods that extract an executable state-feedback policy, directly from those features, which in the literature is known as Behavioural Cloning. In this theoretical work, we formalize the rational inference model of the student decision maker, devoted to imitation, as a controlled Hidden Markov Model. The IfPO problem is then reformulated as a Maximum Likelihood Estimation problem and treated using Expectation-Maximization. We name the resulting fixed point iterations A Posteriori Control Densities. We compare the presented approach to existing methods in the field and identify potential directions for further development, such as an extension to unknown transition and emission models.

On the Relationship of Optimal State Feedback and Disturbance Response Controllers

This paper studies the relationship between state feedback policies and disturbance response policies for the standard Linear Quadratic Regulator (LQR). For open-loop stable plants, we establish a simple relationship between the optimal state feedback controller ut = K*xt and the optimal disturbance response controller ut = L(H)*;1wt-1 +...+ L(H)*;1wt-H with H-order. Here xt, wt, ut stands for the state, disturbance, control action of the system, respectively. Our result shows that L(H)*,1 is a good approximation of K* and the approximation error ∥K* — L(H)*;1∥ decays exponentially with H. We further extend this result to LQR for open-loop unstable systems, when a pre-stabilizing controller K0 is available.

Implications of Regret on Stability of Linear Dynamical Systems

The setting of an agent making decisions under uncertainty and under dynamic constraints is common for the fields of optimal control, reinforcement learning, and recently also for online learning. In the online learning setting, the quality of an agent's decision is often quantified by the concept of regret, comparing the performance of the chosen decisions to the best possible ones in hindsight. While regret is a useful performance measure, when dynamical systems are concerned, it is important to also assess the stability of the closed-loop system for a chosen policy. In this work, we show that for linear state feedback policies and linear systems subject to adversarial disturbances, linear regret implies asymptotic stability in both time-varying and time-invariant settings. Conversely, we also show that bounded input bounded state stability and summability of the state transition matrices imply linear regret.

Unified Control Parameterization Approach for Finite-Horizon Feedback Control With Trajectory Shaping

This article presents control parameterization as a unifying framework for designing a linear feedback control law that achieves the finite-time transfer of output as well as trajectory shaping. Representing control input as a linear combination of independent basis functions allows wide variability in the resultant feedback control laws through the selection of the number and types of basis functions. Given an array of basis functions that meets the trajectory shaping necessities, the unified design approach proceeds with the determination of the coefficients so that the predicted trajectory attains the desired output at the final time. The input evaluated with the coefficients found at each instance essentially turns out to be a linear state feedback policy with an additional feedforward term and time-dependent gains, which is appropriate for practical use. The unified control parameterization approach lends itself well to missile guidance applications with the expandability and direct trajectory shaping capability that it provides. To emphasize the expandability of the framework, this study revisits the trajectory shaping guidance laws from the control parameterization viewpoint and shows how the notion of specifying input basis functions not only generalizes various existing methods but also enables further extensions. Furthermore, an application to integrated guidance and control illustrates the strength of the design process in handling the shaping requirements more directly through the construction of appropriate basis.

Binary search tree-based explicit MPC controller design with Kalman filter for vehicular adaptive cruise system

The adaptive cruise control (ACC) system has received significant attention due to traffic safety improvement, traffic throughput increment, and energy conservation. Model Predictive Control (MPC) has been successfully applied in the control of multi-objective vehicular ACC. However, as a state-feedback policy, MPC requires full state measurement. Meanwhile, the real-time performance of MPC is intractable. This paper proposes to estimate the state value and disturbance value with an extended state Kalman filter to deal with measurement uncertainty. The Kalman filter is based on an augmented state-space model which takes the disturbance term as a new state. To improve real-time performance, this paper suggests employing an explicit MPC (EMPC) based on binary search tree to move the online computational burden of MPC to offline computation by multi-parametric quadratic programming (MPQP). An improved algorithm to solve the MPQP problem offline is proposed, which is initialized discarding the requirement of parameters range, while previous methods need. In the simulated measurement process, the extended state Kalman filter can effectively reduce noise and accurately estimate the value of state and disturbance in the car-following model. Simulations in different scenarios are performed to test the effectiveness of the proposed ACC controller. Results show that the proposed EMPC for the ACC system can improve the real-time performance of the MPC with little loss of performance. On average, the EMPC via binary search is 95.8 times faster than the MPC with the same parameters as EMPC for the studied ACC system. And it has better overall performance compared with the ACC with collision avoidance (CA-ACC) method.

Stochastic Model Predictive Control Using Simplified Affine Disturbance Feedback for Chance-Constrained Systems

This letter covers the model predictive control of linear discrete-time systems subject to stochastic additive disturbances and chance constraints on their state and control input. We propose a simplified control parameterization under the framework of affine disturbance feedback, and we show that our method is equivalent to parameterization over the family of state feedback policies. Using our method, associated finite-horizon optimization can be computed efficiently, with a slight increase in conservativeness compared with conventional affine disturbance feedback parameterization.

On Optimal Control of Discounted Cost Infnite-Horizon Markov Decision Processes Under Local State Information Structures

This paper investigates a class of optimal control problems associated with Markov processes with local state information. The decision-maker has only a local access to a subset of a state vector information as often encountered in decentralized control problems in multi-agent systems. Under this information structure, part of the state vector cannot be observed. We leverage ab initio principles and find a new form of Bellman equations to characterize the optimal policies of the control problem under local information structures. The dynamic programming solutions feature a mixture of dynamics associated unobservable state components and the local state-feedback policy based on the observable local information. We further characterize the optimal local-state feedback policy using linear programming methods. To reduce the computational complexity of the optimal policy, we propose an approximate algorithm based on virtual beliefs to find a sub-optimal policy. We show the performance bounds on the sub-optimal solution and corroborate the results with numerical case studies.

Design and Analysis of Several State-Feedback Fault-Tolerant Control Strategies for Semi-Active Suspensions

Abstract The design of State-Feedback Fault Tolerant control for Semi-Active Suspension Systems is considered in this work, that exploits diverse simple-to-implement approaches. The suspension damper is assumed to undergo multiplicative (time-varying) faults, that can be estimated by (four) modular fault estimation observers. With these fault estimations, active fault tolerant control (FTC) schemes can be synthesized, based on the reconfiguration of nominal State-Feedback policies. Seven approaches are discussed: i) Direct fault compensation; ii) Pole placement compensation; iii) Fault-dependent pole placement; iv) Linear-fault-dependent Linear Quadratic Regulator (LQR) design; v) Polynomially-fault-dependent LQR parameters; vi) LQR with Fault-dependent controlled outputs; vii) Heuristic (vehicle-oriented) fault-dependent LQR synthesis. The performances of these methods are analysed and compared through realistic and high-fidelity simulations. Results show the overall good operation of the latter approaches to compensate fault events and maintain performances.

Stability of Fluid Queueing Systems With Parallel Servers and Stochastic Capacities

This note introduces a piecewise-deterministic queueing (PDQ) model to study the stability of traffic queues in parallel-link transportation systems facing stochastic capacity fluctuations. The saturation rate (capacity) of the PDQ model switches between a finite set of modes according to a Markov chain, and link inflows are controlled by a state-feedback policy. A PDQ system is stable only if a lower bound on the time-average link inflows does not exceed the corresponding time-average saturation rate. Furthermore, a PDQ system is stable if the following two conditions hold: the nominal mode's saturation rate is high enough that all queues vanish in this mode, and a bilinear matrix inequality involving an underestimate of the discharge rates of the PDQ in individual modes is feasible. The stability conditions can be strengthened for two-mode PDQs. These results can be used for design of routing policies that guarantee stability of traffic queues under stochastic capacity fluctuations.

Fast Evaluation of Quadratic Control-Lyapunov Policy

The evaluation of a control-Lyapunov policy, with quadratic Lyapunov function, requires the solution of a quadratic program (QP) at each time step. For small problems this QP can be solved explicitly; for larger problems an online optimization method can be used. For this reason the control-Lyapunov control policy is considered a computationally intensive control law, as opposed to an “analytical” control law, such as conventional linear state feedback, linear quadratic Gaussian control, or H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> , too complex or slow to be used in high speed control applications. In this note we show that by precomputing certain quantities, the control-Lyapunov policy can be evaluated extremely efficiently. We will show that when the number of inputs is on the order of the square-root of the state dimension, the cost of evaluating a control-Lyapunov policy is on the same order as the cost of evaluating a simple linear state feedback policy, and less (in order) than the cost of updating a Kalman filter state estimate. To give an idea of the speeds involved, for a problem with 100 states and 10 inputs, the control-Lyapunov policy can be evaluated in around 67 μs, on a 2 GHz AMD processor; the same processor requires 40 μs to carry out a Kalman filter update.

Affine and predictive control policies for a class of nonlinear systems

The input-state linear horizon (ISLH) for a nonlinear discrete-time system is defined as the smallest number of time steps it takes the system input to appear nonlinearly in the state variable. In this paper, we employ the latter concept and show that the class of constraint admissible N -step affine state-feedback policies is equivalent to the associated class of constraint admissible disturbance-feedback policies, provided that N is less than the system’s ISLH. The result generalizes a recent result in [Goulart, P. J., Kerrigan, E. C., Maciejowski, J. M. (2006). Optimization over state feedback policies for robust control with constraints. Automatica, 42(4), 523–533] and is significant because it enables one: (i) to determine a constraint admissible state-feedback policy by employing well-known convex optimization techniques; and (ii) to guarantee robust recursive feasibility of a class of model predictive control (MPC) policies by imposing a suitable terminal constraint. In particular, we propose an input-to-state stabilizing MPC policy for a class of nonlinear systems with bounded disturbance inputs and mixed polytopic constraints on the state and the control input. At each time step, the proposed MPC policy requires the solution of a single convex quadratic programme parameterized by the current system state.

Constrained linear system with disturbance: Convergence under disturbance feedback

This paper proposes a disturbance-based control parametrization under the Model Predictive Control framework for constrained linear discrete time systems with bounded additive disturbances. The proposed approach has the same feasible domain as that obtained from parametrization over the family of time-varying state feedback policies. In addition, the closed-loop system is stable in the sense that the state converges to a bounded set that has a characterization determined by a feedback gain.

Robust control of constrained max‐plus‐linear systems

AbstractMax‐plus‐linear (MPL) systems are a class of nonlinear systems that can be described by models that are ‘linear’ in the max‐plus algebra. We provide here solutions to the three types of finite‐horizon min–max control problems for uncertain MPL systems, depending on the nature of the control input over which we optimize: open‐loop input sequences, disturbances feedback policies, and state feedback policies. We assume that the uncertainty lies in a bounded polytope and that the closed‐loop input and state sequence should satisfy a given set of linear inequality constraints for all admissible disturbance realizations. Despite the fact that the controlled system is nonlinear, we provide sufficient conditions that allow one to preserve convexity of the optimal value function and its domain. As a consequence, the min–max control problems can be either recast as a linear program or solved via N parametric linear programs, where N is the prediction horizon. In some particular cases of the uncertainty description (e.g. interval matrices), by employing results from dynamic programming, we show that a min–max control problem can be recast as a deterministic optimal control problem. Copyright © 2008 John Wiley &amp; Sons, Ltd.

Optimization over state feedback policies for robust control with constraints

This paper is concerned with the optimal control of linear discrete-time systems subject to unknown but bounded state disturbances and mixed polytopic constraints on the state and input. It is shown that the class of admissible affine state feedback control policies with knowledge of prior states is equivalent to the class of admissible feedback policies that are affine functions of the past disturbance sequence. This implies that a broad class of constrained finite horizon robust and optimal control problems, where the optimization is over affine state feedback policies, can be solved in a computationally efficient fashion using convex optimization methods. This equivalence result is used to design a robust receding horizon control (RHC) state feedback policy such that the closed-loop system is input-to-state stable (ISS) and the constraints are satisfied for all time and all allowable disturbance sequences. The cost to be minimized in the associated finite horizon optimal control problem is quadratic in the disturbance-free state and input sequences. The value of the receding horizon control law can be calculated at each sample instant using a single, tractable and convex quadratic program (QP) if the disturbance set is polytopic, or a tractable second-order cone program (SOCP) if the disturbance set is given by a 2-norm bound.

RELATIONSHIPS BETWEEN AFFINE FEEDBACK POLICIES FOR ROBUST CONTROL WITH CONSTRAINTS

This paper is concerned with the analysis of a recently-proposed robust control policy for linear discrete-time systems subject to bounded state disturbances with mixed constraints on the states and inputs, which parameterizes the input as an affine function of the past disturbance sequence. The paper shows that this disturbance feedback policy is equivalent to the class of affine state feedback policies with memory of prior states, and thus subsumes the wellknown classes of open-loop and pre-stabilising control policies. Furthermore, the parameterization transforms the non-convex problem of finding an admissible state feedback policy to an equivalent and tractable convex problem.

H infintity control of large-scale jump linear systems via averaging and aggregation

This paper studies, under state feedback policies, the H infinity control design for large-scale jump linear systems where the form process admits strong and weak interactions. Through an analysis that covers both finite and infinite horizon cases and using averaging and aggregation techniques, an aggregate jump linear system of considerably smaller order has been obtained, along with a corresponding (compatible) cost function. This reduced-order (aggregate) problem is another piecewise-deterministic H infinity control problem, and, on the basis of the solution of this problem, we obtain the asymptotic limit of the optimal performance level for the full-order system, as well as an approximate controller that can asymptotically achieve any desired performance level for the full-order system. A by-product of this analysis is a similar decomposition for the piecewise-deterministic linear-regulator problem.

State feedback control of condition/event systems

We consider the problem of synthesizing state feedback control policies for condition/event (C/E) systems where the control signals can be both conditions, which enable and inhibit state transitions, and events, which force state transitions. To represent the causal structure of condition and event feedback, a new feedback composition is defined. An algorithm is presented for synthesizing state feedback policies that keep the state of the C/E system out of a given set of forbidden states. The problem formulation and solution is illustrated for a process control example.