Quadratic Value Function Research Articles

Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

In this paper, a dynamic compensator-based near-optimal control approach for unknown nonaffine nonlinear systems is developed by using integral reinforcement learning. Since system dynamics is unknown, it is difficult to obtain the optimal control policy via neuro-dynamic programming. To address this problem, a general dynamic compensator is introduced as the virtual control input to augment the unknown nonaffine nonlinear system as a partially unknown affine system. For the augmented system, a novel quadratic value function is designed with the system states, the actual control input and the virtual control input. The optimal control of the augmented system can be regarded as the near-optimal control for the original system since the novel optimal value function is an upper bound of the original optimal value function. In order to avoid the identification of system dynamics, the integral reinforcement learning framework is utilized to derive the optimal control based on the solution of Hamilton–Jacobi–Bellman equation via the critic-only structure. Meanwhile, the weight learning rule of the critic neural network is presented with the experience replay technique to relax the persistence of excitation condition. Moreover, the uniform ultimate boundedness of weight estimation errors and the stability of the closed-loop system are guaranteed by using the Lyapunov’s direct method. Finally, simulation results of two examples demonstrate the effectiveness of the developed dynamic compensator-based near-optimal control method.

Matrix Manifold-Based Performance Monitoring of Automatic Control Systems

A “smart” automatic feedback control system is supposed to be aware of its operational performance throughout the service time. Driven by this desire, this article addresses the problem of monitoring performance variations caused by multiplicative factors, such as components’ faults, repairing, or replacement. Differing from the conventional performance indices (e.g., the quadratic value function), it detects the performance variation information from symmetric positive-definite (SPD) kernel matrices. As a carrier of performance variation information, the SPD kernel matrix is identified online. Furthermore, the performance variation is given a fresh insight in the sense that it drives the sliding of the SPD kernel matrix on a Riemannian manifold. Thus, performance variation monitoring is achieved by quantizing the geodesic between SPD kernel matrices directly on the Riemannian manifold. At last, the performance variation is visualized on the Riemannian manifold and the proposed schemes are verified via a simulation study.

Construction of Discontinuous Piecewise Quadratic Value Functions in a Target Control Problem

Construction of Discontinuous Piecewise Quadratic Value Functions in a Target Control Problem

Discrete-Time Non-Zero-Sum Games With Completely Unknown Dynamics.

In this article, off-policy reinforcement learning (RL) algorithm is established to solve the discrete-time N -player nonzero-sum (NZS) games with completely unknown dynamics. The N -coupled generalized algebraic Riccati equations (GARE) are derived, and then policy iteration (PI) algorithm is used to obtain the N -tuple of iterative control and iterative value function. As the system dynamics is necessary in PI algorithm, off-policy RL method is developed for discrete-time N -player NZS games. The off-policy N -coupled Hamilton-Jacobi (HJ) equation is derived based on quadratic value functions. According to the Kronecker product, the N -coupled HJ equation is decomposed into unknown parameter part and the system operation data part, which makes the N -coupled HJ equation solved independent of system dynamics. The least square is used to calculate the iterative value function and N -tuple of iterative control. The existence of Nash equilibrium is proved. The result of the proposed method for discrete-time unknown dynamics NZS games is indicated by the simulation examples.

Approximate Solution of the Target Control Problem with a Nonlinearity Depending on One State Variable

We consider a target control problem for a system of differential equations of a special form in which the nonlinear terms depend on a single state variable. The solution is based on the passage from the original problem to an auxiliary problem for a piecewise linear system as well as on an application of the comparison principle and “smoothed” piecewise quadratic value functions. We also propose a feedback control and derive an a priori estimate for the deviation from the target set in the closed-loop original system. In addition, we present a numerical method whose efficiency is confirmed by two sample computations.

An ADDHP-based Q-learning algorithm for optimal tracking control of linear discrete-time systems with unknown dynamics

This paper investigates a novel Q-learning algorithm based on action dependent dual heuristic programming (ADDHP) to solve the infinite-time domain linear quadratic tracker (LQT) for unknown linear discrete-time systems. The proposed method is conducted based on only system data without requiring the knowledge of the system matrices. After the reference system is determined, an augmented system composed of the original system and the reference system is constructed, and it is proved that the value function of LQT is quadratic concerning the state of the augmented system. Using the quadratic value function, the augmented algebraic Riccati equation (ARE) is derived to solve the LQT. Due to the difficulty of directly solving the augmented ARE, a Q-learning algorithm based on ADDHP structure is used to solve this problem. With unknown system matrices, a model neural network is developed to reconstruct system dynamics incorporating stability analysis. The estimated system matrices are employed to the proposed algorithm to calculate the optimal control by policy iteration. Moreover, the convergence of the algorithm is proved. Two simulation examples are used to validate the performance of the method, where all results have demonstrated the effectiveness of the proposed ADDHP-based Q-learning method without a priori knowledge of system matrices for LQT.

An optimal control design for bidirectional inductive power transfer system using dynamics identification

In this article, we address the tracking control problem for the bidirectional inductive power transfer systems (BIPTS) in the view of optimal performance. To achieve the optimal destination with respect to a quadratic value function, an identified linear model for the BIPTS is first developed, and it is proved to be convergent to the true dynamic model when the persistent excitation condition is satisfied. And then, based on the identified model, the desired system states and control inputs are derived according to the reference transferring power, and be utilized as the feedback signals and feedforward compensation, respectively, for the optimal controller. A hybrid Lyapunov function is developed to guarantee the closed-loop stability and the asymptotically tracking convergence. A numerical simulation is carried out to illustrate the satisfactory control performance.

Relations between Model Predictive Control and Reinforcement Learning

In this paper relations between model predictive control and reinforcement learning are studied for discrete-time linear time-invariant systems with state and input constraints and a quadratic value function. The principles of model predictive control and reinforcement learning are reviewed in a tutorial manner. From model predictive control theory it is inferred that the optimal value function is piecewise quadratic on polyhedra and that the optimal policy is piecewise affine on polyhedra. Various ideas for exploiting the knowledge on the structure and the properties of the optimal value function and the optimal policy in reinforcement learning theory and practice are presented. The ideas can be used for deriving stability and feasibility criteria and for accelerating the learning process which can facilitate reinforcement learning for systems with high order, fast dynamics, and strict safety requirements.

Hybrid IS-VWAP Dynamic Algorithmic Trading via LQR

For optimal trade execution, static models are useful for pre-trade estimation or benchmarking, but real trading must be based on dynamic decision making or dynamic programming (DP). DP algorithms can adjust execution automatically to real-time market price waves as well as inventory positions, and thus are more responsive and realistic. Development of DP models however faces a few notorious obstacles including the curse of dimensionality, absence of closed-form solutions for general formulations, and difficulty in incorporating hard constraints. The DP approach based on linear-quadratic regulators (LQR) was first introduced to algorithmic trading by Hora (2006). In Dynamic Control Theory, LQR models allow closed-form solutions thanks to the Bellman equations and quadratic value functions. The resulted optimal policies are affine functions of the state variables. The current work improves the LQR model of Hora (2006) in several important areas, including (a) the unconditional stability (or convexity), (b) retiring of the second-order small term of stagewise risk aversion, (c) quasi risk aversion achieved by time-varying delay costs, (d) incorporation of the bid-offer spreads, and (e) hybridizing the implementation shortfall (IS) algorithm with the VWAP approach to softly enforce the constraint of completion. The impact models and price dynamics still follow the earlier framework of Huberman and Stanzl (2001). The motivations behind every component of the improved model are elaborated, and the closed-form LQR solutions are derived (Theorem 1 and 2). Numerical simulation of the LQR trading paths is presented, which confirms several desirable properties of the hybrid IS-VWAP model.

Lifted Evaluation of mp-MIQP Solutions

This note presents an efficient approach for the evaluation of multi-parametric mixed integer quadratic programming (mp-MIQP) solutions, occurring for instance in control problems involving discrete time hybrid systems with quadratic cost. Traditionally, the online evaluation requires a sequential comparison of piecewise quadratic value functions. We introduce a lifted parameter space in which the piecewise quadratic value functions become piecewise affine and can be merged to a single value function defined over a single polyhedral partition without any overlaps. This enables efficient point location approaches using a single binary search tree. Numerical experiments with a power electronics application demonstrate an online speedup up to an order of magnitude. We also show how the achievable online evaluation time can be traded off against the offline computational time.

On the construction of nonconvex approximations to reach sets of piecewise linear systems

We solve the problem of constructing external and internal approximations to reach sets for hybrid systems with piecewise linear dynamics. As a rule, such sets are not convex and have complicated structure. For their estimation, we suggest to use the comparison principle and piecewise quadratic value functions of special form. To define the parameters of such functions, we use solutions of a family of systems of Riccati matrix differential equations and some additional ordinary differential equations.

Algebraic Activities Aid Discovery Lessons

Three graphing activities lead students to discover the shapes and properties of the graphs for linear, quadratic, and absolute value functions and inequalities.

Optimal multi-period mean–variance policy under no-shorting constraint

We consider in this paper the mean–variance formulation in multi-period portfolio selection under no-shorting constraint. Recognizing the structure of a piecewise quadratic value function, we prove that the optimal portfolio policy is piecewise linear with respect to the current wealth level, and derive the semi-analytical expression of the piecewise quadratic value function. One prominent feature of our findings is the identification of a deterministic time-varying threshold for the wealth process and its implications for market settings. We also generalize our results in the mean–variance formulation to utility maximization with no-shorting constraint.

Optimal Multiperiod Mean-Variance Policy Under No-Shorting Constraint

We consider in this paper the mean-variance formulation in multi-period portfolio selection under no-shorting constraint. Recognizing the structure of a piecewise quadratic value function, we prove that the optimal portfolio policy is piecewise linear with respect to the current wealth level, and derive the semi-analytical expression of the piecewise quadratic value function. One prominent feature of our findings is the identification of a deterministic time-varying threshold for the wealth process and its implications for market settings. We also generalize our results in the mean-variance formulation to utility maximization under no-shorting constraint.

Optimal Estimation and Regulator: Risk-Sensitive Method for Systems of First Degree

The algorithms for the optimal filter and control have been obtained for polynomial systems of first grade. For the filter, two cases are presented: systems with disturbances in L2 and systems with Brownian motion and parameter ∈ multiplying the diffusion term, in state and observations equations. The performance of this algorithms is verified and compared with the optimal Kalman-Bucy filter through an example. Besides the solution to the optimal control Risk-Sensitive problem for stochastic system, taking quadratic value function as solution of PDE HJB is obtained. These Risk-Sensitive control algorithms are compared with the L-Q control algorithms through a numerical example, using quadratic-exponential cost function to be minimized. The optimal risk-sensitive filter and control algorithms show better performance for large values of the parameter ∈.

Parametric Approximation of Piecewise Quadratic Value Functions for the Control of Complex Systems

We present a new technique for approximate dynamic programming that is suitable for control of large-scale systems with complex dynamics. The approach improves closed-loop performances from a starting control policy by incrementally updating a value function on-line based on the Bellman's optimality equation. The value function is approximated as a map between the state and the associated cost-to-go value to circumvent the “curse-of-dimensionality.” The approximation method uses piecewise quadratic representations of the value function, and can considerably reduce the computational requirement compared to the instance-based methods, which store all the historical data and retrieve them for calculating optimal control actions. Hybrid and nonlinear examples are included to demonstrate the applicability of the approach.

A Verification Theorem and Application to the Linear Quadratic Regulator for Minimax Control Problems

Sufficient conditions for a function to be an optimal minimax control are given for the general problem. Then attention is turned to the control problem with a minimax quadratic value function [formula] and linear state equation dξ/ dτ = A(τ) ξ(τ) + B(τ) ζ(τ), ξ( t) = x ∈ R n . The Bellman equation for the function V is derived and a lower bound for the value function is given. Finally, the scalar case is completely solved with some numerical results given.