Optimal Feedback Control Policy Research Articles

A generalization of the Riccati recursion for equality‐constrained linear quadratic optimal control

AbstractThis paper introduces a generalization of the well‐known Riccati recursion for solving the discrete‐time equality‐constrained linear quadratic optimal control problem. The recursion can be used to compute problem solutions as well as optimal feedback control policies. Unlike other tailored approaches for this problem class, the proposed method does not require restrictive regularity conditions on the problem. This allows its use in nonlinear optimal control problem solvers that use exact Lagrangian Hessian information. We demonstrate that our approach can be implemented in a highly efficient algorithm that scales linearly with the horizon length. Numerical tests show a significant speed‐up of about one order of magnitude with respect to state‐of‐the‐art general‐purpose sparse linear solvers. Based on the proposed approach, faster nonlinear optimal control problem solvers can be developed that are suitable for more complex applications or for implementations on low‐cost or low‐power computational platforms. The implementation of the proposed algorithm is made available as open‐source software.

Hierarchical Sliding-Mode Surface-Based Adaptive Critic Tracking Control for Nonlinear Multiplayer Zero-Sum Games via Generalized Fuzzy Hyperbolic Models

This paper investigates the hierarchical sliding-mode surface (HSMS)-based adaptive critic tracking control problem for nonlinear multiplayer zero-sum games (ZSGs). First, a generalized fuzzy hyperbolic models-based identifier is employed to approximate the unknown nonlinear functions. According to the derived data-driven model, a static control policy is proposed to transform the considered optimal tracking control problem into an optimal regulation control problem of an induced error system. Subsequently, based on sliding mode control technology and the concept of hierarchical design, a novel optimal feedback control policy is developed to regulate the tracking error by minimizing a performance index function related to the HSMS. The solution to the Hamilton-Jacobi-Isaacs equation of nonlinear multiplayer ZSGs is obtained via the HSMS-based critic neural network. Furthermore, the historical stored data is utilized to conquer the difficulty associated with the persistence of excitation conditions. Based on Lyapunov stability theory, it is strictly proven that the tracking error and the critic neural network weight are uniformly ultimately bounded. Finally, two examples are presented to demonstrate the effectiveness of the proposed control scheme.

Observer-based optimal control method combination with event-triggered strategy for hypersonic morphing vehicle

A morphing-attitude coupling control method based on event-triggered adaptive dynamic programming (ETADP) and a finite-time fuzzy disturbance observer (FTFDO) is proposed for a class of hypersonic morphing vehicle (HMV). Firstly, an attitude-morphing coupled dynamic model is established, and the morphing variable is treated as the state for integrated control with attitude. The control law is updated by an event-triggered mechanism, which is designed in two parts: feedforward control and feedback control. Based on the proposed FTFDO, the matched and mismatched uncertainties are estimated and compensated in finite time to solve the feedforward control input. Based on an adaptive dynamic programming algorithm, Critic-Actor dual networks are constructed to implement policy iteration for an approximate optimal feedback control policy by offline training with exploration. The Critic-Actor dual networks then adopt the ETADP algorithm to tune the weight of the networks online when triggering criteria for policy improvement are met. Event-triggered conditions are developed such that the feedforward control and feedback control are both updated by aperiodic sampling, reducing the number of controller updates required. Zeno behavior is avoided by determining the minimum sampling interval time. The stability of the closed-loop system is demonstrated theoretically. Simulation results verify the control efficiency and robustness of the proposed method.

Reinforcement learning and cooperative [formula omitted] output regulation of linear continuous-time multi-agent systems

This paper proposes a novel control approach to solve the cooperative H∞ output regulation problem for linear continuous-time multi-agent systems (MASs). Different from existing solutions to cooperative output regulation problems, a distributed feedforward-feedback controller is developed to achieve asymptotic tracking and reject both modeled and unmodeled disturbances. The feedforward control policy is computed via solving regulator equations, and the optimal feedback control policy is obtained through handling a zero-sum game. Instead of relying on the knowledge of system matrices in the state equations of the followers’ dynamics and initial stabilizing feedback control gains, a value iteration (VI) algorithm is proposed to learn the optimal feedback control gain and feedforward control gain using online data. To the best of our knowledge, this paper is the first to show that the proposed VI algorithm can approximate the solution to continuous-time game algebraic Riccati equations with guaranteed convergence. Finally, the numerical analysis is provided to show the effectiveness of the proposed approach.

Constrained Decoupling Adaptive Dynamic Programming for A Partially Uncontrollable Time-Delayed Model of Energy Systems

The lack of feedback-loop design and uncontrollable state reconstruction, such as load demand and renewable generation, are key factors to hinder the development of energy control. Adaptive dynamic programming is an efficient method to break through the bottleneck. However, most current works assume that the evolution of uncontrollable state satisfies Markovian property. This paper proposes a time-delayed adaptive dynamic programming framework for optimal energy control, utilizing more comprehensive historical data for state reconstruction. The uncontrollable state is reconstructed by an attention mechanism-based Transformer network based on online measured data. Then, a self-learning iterative algorithm is developed to obtain the optimal state feedback control policy under a discounted performance index function. The problem of solving iterative control policy, which is a mixed-integer nonlinear programming problem with an enormous computational burden, is decomposed into three sub-problems, i.e. nonlinear programming with no discrete variable, which is solved by the Projected Newton method, mixed-integer quadratic programming, and quadratic programming. Numerical results illustrate that the developed control algorithm minimizes the electricity cost in the long term and avoids peak load.

Optimal feedback control of batch self-assembly processes using dynamic programming

This paper reviews a previously-reported methodology for establishing feedback control of self-assembly. The methodology combines dimension reduction, supervised learning, and dynamic programming to obtain an optimal feedback control policy for reaching a desired assembled state. Sampled data are used in calculating the optimal feedback policy; this data can be generated using a predictive model (i.e. “simulated data”) or using experimental data. The control strategy is demonstrated, with both simulation and experimental results, for two applications: control of colloidal assembly (to produce perfect colloidal crystals) and control of crystallization from solution (to produce crystals of desired average size).

Control of Self-Assembly with Dynamic Programming

Abstract This conference paper updates a previously-reported methodology for establishing feedback control of self-assembly (Griffin et al. (2016b)). The methodology combines dimension reduction, supervised learning, and dynamic programming to obtain an optimal feedback control policy for reaching a desired assembled state. The strategy is further demonstrated, with both simulation and experimental results, for two applications: control of colloidal assembly (to produce perfect colloidal crystals) and control of crystallization from solution (to produce crystals of desired average size).

A mean-field game model for homogeneous flocking.

Empirically derived continuum models of collective behavior among large populations of dynamic agents are a subject of intense study in several fields, including biology, engineering, and finance. We formulate and study a mean-field game model whose behavior mimics an empirically derived nonlocal homogeneous flocking model for agents with gradient self-propulsion dynamics. The mean-field game framework provides a non-cooperative optimal control description of the behavior of a population of agents in a distributed setting. In this description, each agent's state is driven by optimally controlled dynamics that result in a Nash equilibrium between itself and the population. The optimal control is computed by minimizing a cost that depends only on its own state and a mean-field term. The agent distribution in phase space evolves under the optimal feedback control policy. We exploit the low-rank perturbative nature of the nonlocal term in the forward-backward system of equations governing the state and control distributions and provide a closed-loop linear stability analysis demonstrating that our model exhibits bifurcations similar to those found in the empirical model. The present work is a step towards developing a set of tools for systematic analysis, and eventually design, of collective behavior of non-cooperative dynamic agents via an inverse modeling approach.

Optimal coupled and decoupled perimeter control in one-region cities

Perimeter controllers, located at a regional border, can manipulate the transfer flows across the border to optimize the regional operational performance. The macroscopic fundamental diagram (MFD), that relates average flow with accumulation, is used to model the traffic flow dynamics. In this paper, two cases of perimeter control inputs are considered: coupled and decoupled control. For both cases, the explicit formulations of the optimal feedback control policies and proofs of optimality are provided for three criteria. The proofs are based on the modified Krotov-Bellman sufficient conditions of optimality, where the upper and lower bounds of state variables are calculated.

A bottom‐up approach to identifying the maximum operational adaptive capacity of water resource systems to a changing climate

AbstractMany water resource systems have been designed assuming that the statistical characteristics of future inflows are similar to those of the historical record. This assumption is no longer valid due to large‐scale changes in the global climate, potentially causing declines in water resource system performance, or even complete system failure. Upgrading system infrastructure to cope with climate change can require substantial financial outlay, so it might be preferable to optimize existing system performance when possible. This paper builds on decision scaling theory by proposing a bottom‐up approach to designing optimal feedback control policies for a water system exposed to a changing climate. This approach not only describes optimal operational policies for a range of potential climatic changes but also enables an assessment of a system's upper limit of its operational adaptive capacity, beyond which upgrades to infrastructure become unavoidable. The approach is illustrated using the Lake Como system in Northern Italy—a regulated system with a complex relationship between climate and system performance. By optimizing system operation under different hydrometeorological states, it is shown that the system can continue to meet its minimum performance requirements for more than three times as many states as it can under current operations. Importantly, a single management policy, no matter how robust, cannot fully utilize existing infrastructure as effectively as an ensemble of flexible management policies that are updated as the climate changes.

Nonlinear and Adaptive Suboptimal Control of Connected Vehicles: A Global Adaptive Dynamic Programming Approach

This paper studies the cooperative adaptive cruise control (CACC) problem of connected vehicles with unknown nonlinear dynamics. Different from the present literature on CACC, data-driven feedforward and optimal feedback control policies are developed by global adaptive dynamic programming (GADP). Due to the presence of nonvanishing disturbance, a modified version of GADP is presented. Interestingly, the developed policy is guaranteed to globally stabilize the vehicular platoon system, and is robust to unmeasurable nonvanishing disturbance. Numerical simulation results are presented to validate the effectiveness of the developed approach.

Stochastic Control of Event-Driven Feedback in Multiantenna Interference Channels

Spatial interference avoidance is a simple and effective way of mitigating interference in multi-antenna wireless networks. The deployment of this technique requires channel-state information (CSI) feedback from each receiver to all interferers, resulting in substantial network overhead. To address this issue, this paper proposes the method of distributive control that intelligently allocates CSI bits over multiple feedback links and adapts feedback to channel dynamics. For symmetric channel distributions, it is optimal for each receiver to equally allocate the average sum-feedback rate for different feedback links, thereby decoupling their control. Using the criterion of minimum sum-interference power, the optimal feedback-control policy is shown using stochastic-optimization theory to exhibit opportunism. Specifically, a specific feedback link is turned on only when the corresponding transmit-CSI error is significant or interference-channel gain large, and the optimal number of feedback bits increases with this gain. For high mobility and considering the sphere-cap-quantized-CSI model, the optimal feedback-control policy is shown to perform water-filling in time, where the number of feedback bits increases logarithmically with the corresponding interference-channel gain. Furthermore, we consider asymmetric channel distributions with heterogeneous path losses and high mobility, and prove the existence of a unique optimal policy for jointly controlling multiple feedback links. Given the sphere-cap-quantized-CSI model, this policy is shown to perform water-filling over feedback links. Finally, simulation demonstrates that feedback-control yields significant throughput gains compared with the conventional differential-feedback method.

Event-Driven Optimal Feedback Control for Multiantenna Beamforming

Transmit beamforming is a simple multi-antenna technique for increasing throughput and the transmission range of a wireless communication system. The required feedback of channel state information (CSI) can potentially result in excessive overhead especially for high mobility or many antennas. This work concerns efficient feedback for transmit beamforming and establishes a new approach of controlling feedback for maximizing net throughput, defined as throughput minus average feedback cost. The feedback controller using a stationary policy turns CSI feedback on/off according to the system state that comprises the channel state and transmit beamformer. Assuming channel isotropy and Markovity, the controller's state reduces to two scalars. This allows the optimal control policy to be efficiently computed using dynamic programming. Consider the perfect feedback channel free of error, where each feedback instant pays a fixed price. The corresponding optimal feedback control policy is proved to be of the threshold type. This result holds regardless of whether the controller's state space is discretized or continuous. Under the threshold-type policy, feedback is performed whenever a state variable indicating the accuracy of transmit CSI is below a threshold, which varies with channel power. The practical finite-rate feedback channel is also considered. The optimal policy for quantized feedback is proved to be also of the threshold type. The effect of CSI quantization is shown to be equivalent to an increment on the feedback price. Moreover, the increment is upper bounded by the expected logarithm of one minus the quantization error. Finally, simulation shows that feedback control increases net throughput of the conventional periodic feedback by up to 0.5 bit/s/Hz without requiring additional bandwidth or antennas.

Optimal Production Planning in a Stochastic N-machine Flowshop with Long-Run Average Cost

This paper is concerned with the problem of production planning in a stochastic manufacturing system with serial machines that are subject to breakdown and repair. The machine capacities are modeled as Markov chains. Since the number of parts in the internal buffers between any two the problem is inherently a state constrained problem. The objective is choose the input rates at the various machines over time in order to meet formulated as a stochastic dynamic programming problem. We prove a verification theorem and derive the optimal feedback control policy in terms of the directional derivatives of the potential function.

Efficient robust optimization for robust control with constraints

This paper proposes an efficient computational technique for the optimal control of linear discrete-time systems subject to bounded disturbances with mixed linear constraints on the states and inputs. The problem of computing an optimal state feedback control policy, given the current state, is non-convex. A recent breakthrough has been the application of robust optimization techniques to reparameterize this problem as a convex program. While the reparameterized problem is theoretically tractable, the number of variables is quadratic in the number of stages or horizon length N and has no apparent exploitable structure, leading to computational time of $${\mathcal{O}}(N^{6})$$ per iteration of an interior-point method. We focus on the case when the disturbance set is ∞-norm bounded or the linear map of a hypercube, and the cost function involves the minimization of a quadratic cost. Here we make use of state variables to regain a sparse problem structure that is related to the structure of the original problem, that is, the policy optimization problem may be decomposed into a set of coupled finite horizon control problems. This decomposition can then be formulated as a highly structured quadratic program, solvable by primal-dual interior-point methods in which each iteration requires $${\mathcal{O}}(N^3)$$ time. This cubic iteration time can be guaranteed using a Riccati-based block factorization technique, which is standard in discrete-time optimal control. Numerical results are presented, using a standard sparse primal-dual interior point solver, that illustrate the efficiency of this approach.

A DECENTRALIZED INVENTORY CONTROL POLICY FOR SUPPLY CHAINS

The paper proposes a linear model with distributed delays to describe a multistage supply chain. The product structure determines the organisation of the network of enterprises that cooperate to manufacture end-products from raw materials. Autonomy of production units imposes structural constraints to the production and inventory policy. For the long-term time-horizon, an optimal global state feedback control policy is proposed to meet stationary stochastic demands for end-products. It is shown that this policy naturally satisfies the structural constraints of the decentralized problem and reduces to a sequence of local base-stock policies. This policy is then naturally extended to the short-term time-horizon with the objectives of reaching a target set while meeting the demand and respecting the constraints.

AN EFFICIENT DECOMPOSITION-BASED FORMULATION FOR ROBUST CONTROL WITH CONSTRAINTS

This paper is concerned with the development of an efficient computational solution method for control of linear discrete-time systems subject to bounded disturbances with mixed polytopic constraints on the states and inputs. It is shown that the non-convex problem of computing an optimal affine state feedback control policy can be solved through reparameterization to an equivalent convex problem, and that if the disturbance set is the linear map of a hypercube, then this problem may be decomposed into a coupled set of finite horizon control problems. If the problem involves the minimization of a quadratic cost, then this yields a highly structured quadratic program in a tractable number of variables, which can be solved with great efficiency.

Existence of Optimal Feedback Production Plans in Stochastic Flowshops with Limited Buffers

We consider an N-machine flowshop with unreliable machines and bounds on work-in-process. Machine capacities and demand processes are finite-state Markov chains. The problem is to choose the rates of production on the machines over time to minimize the expected discounted costs of production and inventory/backlog. We show that the value function of the problem is locally Lipschitz and is a solution to a dynamic programming equation with a certain boundary condition. We provide a verification theorem, and derive the optimal feedback control policy in terms of the directional derivatives of the value function. © 1997 Elsevier Science Ltd.

Optimal feedback production planning in a stochastic N-machine flowshop

We consider a production planning problem in an N-machine flowshop subject to breakdown and repair of machines and to non-negativity constraints on work-in-process. The machine capacities and demand processes are assumed to be finite-state Markov chains. The problem is to choose the rates of production on the N machines over time to minimize the expected discounted cost of production and inventory/backlog over an infinite horizon. It is formulated as a stochastic dynamic programming problem. We show that the value function of the problem is locally Lipschitz and is a solution to a dynamic programming equation together with a certain boundary condition. We provide an interpretation of the boundary condition. We also prove a verification theorem and derive the optimal feedback control policy in terms of the directional derivatives of the value function. Finally, we obtain a deterministic optimal control problem that is equivalent to the stochastic production planning problem under consideration.

Optimal feedback production planning in a stochastic two-machine flowshop

In this paper, we consider a production planning problem in a two-machine flowshop subject to breakdown and repair of machines and subject to non-negativity constraints on work-in-process. The machine capacities and demand processes are assumed to be finite state Markov chains. The problem is to choose the rate of production over time so as to minimize the expected discounted cost of production and inventory/backlog over an infinite horizon. The problem is formulatred as a stochastic dynamic programming problem. It is shown that the value function of the problem is locally Lipschitz and is a viscosity solution to the dynamic programming equation together with certain boundary conditions. Optimal feedback control policies are obtained in terms of the ‘partial derivatives’ of the value function. It is shown that the derived control policy is feasible on the boundary of the state space. The results establish the theoretical framework within which further analyses and/or numerical solutions of the problem can be attempted.