Direct HDP Research Articles

Virtual inertia control parameter regulator of doubly fed induction generator based on direct heuristic dynamic programming

The most widely applied doubly-fed induction generator (DFIG) leads to insufficient frequency regulation ability of power system. The virtual inertia control can adjust the active power of DFIG according to frequency fluctuation. However, the frequent changes in power flow and operating point bring great challenges to traditional virtual inertia controller with fixed parameters. Direct Heuristic Dynamic Programming (Direct HDP) based control parameter regulator is proposed in this paper to obtain the optimal control parameters when the operation condition changes. Compared with the initial parameters, the response has been significantly improved after adaptive optimization by the proposed method when the system operating point changes.

Adaptive Modulation for DFIG and STATCOM With High-Voltage Direct Current Transmission.

This paper develops an adaptive modulation approach for power system control based on the approximate/adaptive dynamic programming method, namely, the goal representation heuristic dynamic programming (GrHDP). In particular, we focus on the fault recovery problem of a doubly fed induction generator (DFIG)-based wind farm and a static synchronous compensator (STATCOM) with high-voltage direct current (HVDC) transmission. In this design, the online GrHDP-based controller provides three adaptive supplementary control signals to the DFIG controller, STATCOM controller, and HVDC rectifier controller, respectively. The mechanism is to observe the system states and their derivatives and then provides supplementary control to the plant according to the utility function. With the GrHDP design, the controller can adaptively develop an internal goal representation signal according to the observed power system states, therefore, to achieve more effective learning and modulating. Our control approach is validated on a wind power integrated benchmark system with two areas connected by HVDC transmission lines. Compared with the classical direct HDP and proportional integral control, our GrHDP approach demonstrates the improved transient stability under system faults. Moreover, experiments under different system operating conditions with signal transmission delays are also carried out to further verify the effectiveness and robustness of the proposed approach.

A boundedness result for the direct heuristic dynamic programming

Approximate/adaptive dynamic programming (ADP) has been studied extensively in recent years for its potential scalability to solve large state and control space problems, including those involving continuous states and continuous controls. The applicability of ADP algorithms, especially the adaptive critic designs has been demonstrated in several case studies. Direct heuristic dynamic programming (direct HDP) is one of the ADP algorithms inspired by the adaptive critic designs. It has been shown applicable to industrial scale, realistic and complex control problems. In this paper, we provide a uniformly ultimately boundedness (UUB) result for the direct HDP learning controller under mild and intuitive conditions. By using a Lyapunov approach we show that the estimation errors of the learning parameters or the weights in the action and critic networks remain UUB. This result provides a useful controller convergence guarantee for the first time for the direct HDP design.

Performance Evaluation of Direct Heuristic Dynamic Programming using Control-Theoretic Measures

Approximate dynamic programming (ADP) has been widely studied from several important perspectives: algorithm development, learning efficiency measured by success or failure statistics, convergence rate, and learning error bounds. Given that many learning benchmarks used in ADP or reinforcement learning studies are control problems, it is important and necessary to examine the learning controllers from a control-theoretic perspective. This paper makes use of direct heuristic dynamic programming (direct HDP) and three typical benchmark examples to introduce a unique analytical framework that can be applied to other learning control paradigms and other complex control problems. The sensitivity analysis and the linear quadratic regulator (LQR) design are used in the paper for two purposes: to quantify direct HDP performances and to provide guidance toward designing better learning controllers. The use of LQR however does not limit the direct HDP to be a learning controller that addresses nonlinear dynamic system control issues. Toward this end, applications of the direct HDP for nonlinear control problems beyond sensitivity analysis and the confines of LQR have been developed and compared whenever appropriate to an LQR design.

Direct Heuristic Dynamic Programming for Damping Oscillations in a Large Power System

This paper applies a neural-network-based approximate dynamic programming method, namely, the direct heuristic dynamic programming (direct HDP), to a large power system stability control problem. The direct HDP is a learning- and approximation-based approach to addressing nonlinear coordinated control under uncertainty. One of the major design parameters, the controller learning objective function, is formulated to directly account for network-wide low-frequency oscillation with the presence of nonlinearity, uncertainty, and coupling effect among system components. Results include a novel learning control structure based on the direct HDP with applications to two power system problems. The first case involves static var compensator supplementary damping control, which is used to provide a comprehensive evaluation of the learning control performance. The second case aims at addressing a difficult complex system challenge by providing a new solution to a large interconnected power network oscillation damping control problem that frequently occurs in the China Southern Power Grid.

Supplementary Damping Controller Design using Direct Heuristic Dynamic Programming in Complex Power Systems

In modern, large scale interconnected power grids, low-frequency oscillation is a key roadblock to improved power transmission capacity. Supplementary generator control, flexible AC transmission system (FACTS), and high voltage direct currents (HVDC) are engineered devices designed to damp such low frequency swings. In this paper a neural network-based approximate dynamic programming method, namely direct heuristic dynamic programming (direct HDP), is applied to power system stability enhancement. Direct HDP is a learning and approximation based approach to addressing nonlinear system control problems under uncertainty, and it is also a model-free design strategy. The action and critic networks of the direct HDP are implemented using multi-layer perceptrons; learning is carried out based on the interactions between the controller and the power system. For this design approach, real time system responses are provided through wide-area measurement system (WAMS). The controller learning objective is formulated as a reward function that reflects global characteristics of the power system under low frequency oscillation, as well as tight coupling effects among system components. Direct HDP control design is illustrated by case studies, which are also used to demonstrate the learning control performance. The proposed direct HDP learning control is also developed as a new solution to a large scale system coordination problem by using the China Southern Power Grid as a major test bed.