Dual Heuristic Dynamic Programming Research Articles

Wide-Area Measurement Based Dynamic Stochastic Optimal Power Flow Control for Smart Grids With High Variability and Uncertainty

To achieve a high penetration level of intermittent renewable energy, the operation and control of power systems need to account for the associated high variability and uncertainty. Power system stability and security need to be ensured dynamically as the system operating condition continuously changes. A wide-area measurement based dynamic stochastic optimal power flow (DSOPF) control algorithm using the adaptive critic designs (ACDs) is presented in this paper. The proposed DSOPF control replaces the traditional AGC and secondary voltage control, and provides a coordinated AC power flow control solution to the smart grid operation in an environment with high short-term uncertainty and variability. The ACD technique, specifically the dual heuristic dynamic programming (DHP), is used to provide nonlinear optimal control, where the control objective is explicitly formulated to incorporate power system economy, stability and security considerations. The proposed DSOPF controller dynamically drives the power system to its optimal operating point by continuously adjusting the steady-state set points sent by the traditional OPF algorithm. A 12-bus test power system is used to demonstrate the development and effectiveness of the proposed DSOPF controller.

Transformation Invariant On-Line Target Recognition

Transformation invariant automatic target recognition (ATR) has been an active research area due to its widespread applications in defense, robotics, medical imaging and geographic scene analysis. The primary goal for this paper is to obtain an on-line ATR system for targets in presence of image transformations, such as rotation, translation, scale and occlusion as well as resolution changes. We investigate biologically inspired adaptive critic design (ACD) neural network (NN) models for on-line learning of such transformations. We further exploit reinforcement learning (RL) in ACD framework to obtain transformation invariant ATR. We exploit two ACD designs, such as heuristic dynamic programming (HDP) and dual heuristic dynamic programming (DHP) to obtain transformation invariant ATR. We obtain extensive statistical evaluations of proposed on-line ATR networks using both simulated image transformations and real benchmark facial image database, UMIST, with pose variations. Our simulations show promising results for learning transformations in simulated images and authenticating out-of plane rotated face images. Comparing the two on-line ATR designs, HDP outperforms DHP in learning capability and robustness and is more tolerant to noise. The computational time involved in HDP is also less than that of DHP. On the other hand, DHP achieves a 100% success rate more frequently than HDP for individual targets, and the residual critic error in DHP is generally smaller than that of HDP. Mathematical analyses of both our RL-based on-line ATR designs are also obtained to provide a sufficient condition for asymptotic convergence in a statistical average sense.

Optimality and convergence of adaptive optimal control by reinforcement synthesis

Adaptive Optimal Control (AOC) by reinforcement synthesis is proposed to facilitate the application of optimal control theory in feedback controls. Reinforcement synthesis uses the critic–actor architecture of reinforcement learning to carry out sequential optimization. Optimality conditions for AOC are formulated using the discrete minimum principle. A proof of the convergence conditions for the reinforcement synthesis algorithm is presented. As the final time extends to infinity, the reinforcement synthesis algorithm is equivalent to the Dual Heuristic dynamic Programming (DHP) algorithm, a version of approximate dynamic programming. Thus, formulating DHP with the AOC approach has rigorous proofs of optimality and convergence. The efficacy of AOC by reinforcement synthesis is demonstrated by solving a linear quadratic regulator problem.

Automatic train regulation for metro lines using dual heuristic dynamic programming

Issues regarding capacity and service quality of metro lines have been receiving attention in worldwide transportation circles recently. Because investment in metro infrastructure is costly, increasing capacity utilization of existing systems while maintaining service quality, particularly in environments where disturbance occurs frequently, is a challenge. For highly homogeneous railway systems such as metro lines, the robustness of the train regulation system against disturbances is an important factor affecting capacity utilization and service quality. Train regulation in metro lines is essentially a highly non-linear, stochastic, and constrained optimization problem. This article proposes an automatic train regulation (ATR) design using dual heuristic dynamic programming (DHP). The ATR system makes easier the handling of the non-linear and stochastic characteristics of metro line with infrastructure constraint; and near-optimal regulation can be obtained more rapidly and accurately. The verification of the DHP-based ATR referring to practical operation data of Taipei metro is presented, and a comparison with a design using linear quadratic regulator is also provided, especially regarding robustness against disturbance.

Adaptive Critic Designs for Discrete-Time Zero-Sum Games With Application to $H_{\infty}$ Control

In this correspondence, adaptive critic approximate dynamic programming designs are derived to solve the discrete-time zero-sum game in which the state and action spaces are continuous. This results in a forward-in-time reinforcement learning algorithm that converges to the Nash equilibrium of the corresponding zero-sum game. The results in this correspondence can be thought of as a way to solve the Riccati equation of the well-known discrete-time H(infinity) optimal control problem forward in time. Two schemes are presented, namely: 1) a heuristic dynamic programming and 2) a dual-heuristic dynamic programming, to solve for the value function and the costate of the game, respectively. An H(infinity) autopilot design for an F-16 aircraft is presented to illustrate the results.

New External Neuro-Controller for Series Capacitive Reactance Compensator in a Power Network

The controllable capacitive reactance can be used as the input variable for the external controller of a series capacitive reactance compensator (SCRC) to improve the damping of low-frequency oscillations of the rotor angle and active power in a power system. Conventional linear PI controllers are tuned for best performance at one specific operating point of the nonlinear power system. At other operating point its performance degrades. Nonlinear optimal neuro-controllers are able to overcome this degradation. In this paper, the dual heuristic dynamic programming (DHP) optimization algorithm is applied to design an external nonlinear optimal neuro-controller for the SCRC. Simulation studies using PSCAD/EMTDC/spl reg/ software are presented.