Continuous Action Reinforcement Learning Automata Research Articles

A Reinforcement-learning-based Bidding Strategy for Power Suppliers with Limited Information

The power market is a typical imperfectly competitive market where power suppliers gain higher profits through strategic bidding behaviors. Most existing studies assume that a power supplier is accessible to the sufficient market information to derive an optimal bidding strategy. However, this assumption may not be true in reality, particularly when a power market is newly launched. To help power suppliers bid with the limited information, a modified continuous action reinforcement learning automata algorithm is proposed. This algorithm introduces the discretization and Dyna structure into continuous action reinforcement learning automata algorithm for easy implementation in a repeated game. Simulation results verify the effectiveness of the proposed learning algorithm.

Optimal Incentive Strategy in Cloud-Edge Integrated Demand Response Framework for Residential Air Conditioning Loads

In the residential demand response area, currently the incentive-based method (e.g., direct load control, DLC) may impair users’ comfort and autonomy, while the price-based method can hardly guarantee users’ engagements. This paper proposes an edge-cloud integrated demand response framework to achieve an effect-predictable residential demand response without harming users’ benefits. First, we combine the cloud-computing resource (cloud) and the home-installed smart thermostats (edges) to formulate an efficient, cost-effective, and data-secured infrastructure to implement the demand response program. Then, we model the demand response problem between the load aggregator and its served residential users as a bi-level optimization problem, and the key is for the load aggregator to find the optimal incentive strategy. To solve this problem, we introduce an RL algorithm, i.e., Continuous Action Reinforcement Learning Automata, to quickly obtain the optimal incentive strategy under an incomplete information scenario. Simulation results based on 136 real-world residential users in Austin area demonstrate that the proposed CEI-DR framework can increase the social welfare by about $8.6/h compared to the traditional DLC method during a normal DR event.

Multivariable System Identification Method Based on Continuous Action Reinforcement Learning Automata

In this work, a closed-loop identification method based on a reinforcement learning algorithm is proposed for multiple-input multiple-output (MIMO) systems. This method could be an attractive alternative solution to the problem that the current frequency-domain identification algorithms are usually dependent on the attenuation factor. With this method, after continuously interacting with the environment, the optimal attenuation factor can be identified by the continuous action reinforcement learning automata (CARLA), and then the corresponding parameters could be estimated in the end. Moreover, the proposed method could be applied to time-varying systems online due to its online learning ability. The simulation results suggest that the presented approach can meet the requirement of identification accuracy in both square and non-square systems.

A novel dynamics model of ball-screw feed drives based on theoretical derivations and deep learning

High fidelity models of feed drive are critical factors to increase positioning accuracy and decrease contour error. To predict feed drives dynamics, this paper reports a novel method for modeling dynamics of feed drive by combining advantages of theoretical derivations and deep learning. First, the paper derives a rigid-flexible-combined dynamics model (RFCDM) for feed drive from classical dynamics theory. Then parameters identification of RFCDM is accomplished by referring product manuals and conducting constant velocity experiment with different feed rates. Continuous action reinforcement learning automata (CARLA) is adopted to tune all parameters of RFCDM simultaneously. A simulation error estimation model (SEEM) is applied to approximate simulation error between models simulation position and worktables actual position. The hybrid dynamics model (HDM) of feed drives which integrates RFCDM with SEEM is validated by experiments with various trajectories. Experimental results show that the gap between HDMs prediction position and worktables actual position is on the order of magnitude of 0.01 mm which is about 1/10 of the tracking error, indicating the HDM can predict the dynamics of feed drives with safe accuracy.

A reinforcement learning approach to coordinate exploration with limited communication in continuous action games

AbstractLearning automata are reinforcement learners belonging to the class of policy iterators. They have already been shown to exhibit nice convergence properties in a wide range of discrete action game settings. Recently, a new formulation for a continuous action reinforcement learning automata (CARLA) was proposed. In this paper, we study the behavior of these CARLA in continuous action games and propose a novel method for coordinated exploration of the joint-action space. Our method allows a team of independent learners, using CARLA, to find the optimal joint action in common interest settings. We first show that independent agents using CARLA will converge to a local optimum of the continuous action game. We then introduce a method for coordinated exploration which allows the team of agents to find the global optimum of the game. We validate our approach in a number of experiments.

Combination of Continuous Action Reinforcement Learning Automata & PSO to Design a PID Controller for AVR System

Combination of Continuous Action Reinforcement Learning Automata & PSO to Design a PID Controller for AVR System

Power System Dynamic Stability Enhancement Using Optimum Design of PSS and Static Phase Shifter Based Stabilizer

Power system stability enhancement via individual and coordinated optimum design of power system stabilizer and static phase shifter based stabilizer is thoroughly investigated in this paper. The coordinated design problem of excitation and static phase shifter-based stabilizer at nominal loading condition over a wide range of uncertainties and system parameter variations is formulated as an optimization problem with an eigenvalue–based objective function. The reinforcement learning automata based optimization algorithm called Combinatorial Discrete and Continuous Action Reinforcement Learning Automata (CDCARLA) is employed to search for the optimal setting of the proposed controller parameters. The participation factors method is used to identify the poorly damped electromechanical mode which is used in the evaluation of the eigenvalue–based objective function. This study also presents a singular value decomposition–based approach to assess and measure the controllability of the identified electromechanical mode by different control inputs over a wide range of loading conditions. For evaluation of the effectiveness and the robustness of the proposed stabilizers, a weakly connected power system with different disturbances, loading conditions, and system parameter variations is considered. The nonlinear simulation results and the eigenvalues analysis indicate a high performance of the proposed stabilizers and their ability to provide efficient damping of the low frequency oscillations.

APPLICATION OF NOVEL REINFORCEMENT LEARNING AUTOMATA APPROACH IN POWER SYSTEM REGULATION

In this paper, a novel efficient optimization method based on reinforcement learning automata (RLA) for optimum parameters setting of conventional proportional-integral-derivative (PID) controller for AVR system of power synchronous generator is proposed. The proposed method is Combinatorial Discrete and Continuous Action Reinforcement Learning Automata (CDCARLA) which is able to explore and learn to improve control performance without the knowledge of the analytical system model. This paper demonstrates the full details of the CDCARLA technique and compares its performance with Particle Swarm Optimization (PSO) as an efficient evolutionary optimization method. The proposed method has been applied to PID controller design. The simulation results show the superior efficiency and robustness of the proposed method.

Parameter optimization of multimachine power system conventional stabilizers using CDCARLA method

In this paper, a novel Combinatorial Discrete and Continuous Action Reinforcement Learning Automata (CDCARLA) based approach for optimal design of multimachine power system stabilizers (PSSs) is presented. The proposed CDCARLA based design approach is a combined procedure of two optimization stages in discrete and continuous spaces for fast convergence and high optimization efficiency. The potential of the proposed approach in seeking the optimal settings of the widely used conventional lead-lag PSSs’ parameters is investigated and assessed in multimachine power systems. The performance and robustness of the proposed CDCARLA based PSS is evaluated under different power system disturbances. The performance of the proposed stabilizer is also compared with other stabilizers reported in the literature including the multi-band PSSs for a two-area four-machine power system. Simulation results show the effectiveness and robustness of the proposed CDCARLA PSS in damping local and inter area oscillation modes under various disturbances, and confirm its superiority in comparison with other types of PSSs.

Continuous action reinforcement learning automata and their application to adaptive digital filter design

In the design of adaptive IIR filters, the multi-modal nature of the error surfaces can limit the use of gradient-based and other iterative search methods. Stochastic learning automata have previously been shown to have global optimisation properties making them suitable for the optimisation of filter coefficients. Continuous action reinforcement learning automata are presented as an extension to the standard automata which operate over discrete parameter sets. Global convergence is claimed, and demonstrations are carried out via a number of computer simulations.

On-line PID tuning for engine idle-speed control using continuous action reinforcement learning automata

PID systems are widely used to apply control without the need to obtain a dynamic model. However, the performance of controllers designed using standard on-line tuning methods, such as Ziegler–Nichols, can often be significantly improved. In this paper the tuning process is automated through the use of continuous action reinforcement learning automata (CARLA). These are used to simultaneously tune the parameters of a three term controller on-line to minimise a performance objective. Here the method is demonstrated in the context of engine idle-speed control; the algorithm is first applied in simulation on a nominal engine model, and this is followed by a practical study using a Ford Zetec engine in a test cell. The CARLA provides marked performance benefits over a comparable Ziegler–Nichols tuned controller in this application.