Actor-critic Reinforcement Learning Research Articles

Adaptive Optimal Tracking Control of an Underactuated Surface Vessel Using Actor-Critic Reinforcement Learning.

In this article, we present an adaptive reinforcement learning optimal tracking control (RLOTC) algorithm for an underactuated surface vessel subject to modeling uncertainties and time-varying external disturbances. By integrating backstepping technique with the optimized control design, we show that the desired optimal tracking performance of vessel control is guaranteed due to the fact that the virtual and actual control inputs are designed as optimized solutions of every subsystem. To enhance the robustness of vessel control systems, we employ neural network (NN) approximators to approximate uncertain vessel dynamics and present adaptive control technique to estimate the upper boundedness of external disturbances. Under the reinforcement learning framework, we construct actor-critic networks to solve the Hamilton-Jacobi-Bellman equations corresponding to subsystems of surface vessel to achieve the optimized control. The optimized control algorithm can synchronously train the adaptive parameters not only for actor-critic networks but also for NN approximators and adaptive control. By Lyapunov stability theorem, we show that the RLOTC algorithm can ensure the semiglobal uniform ultimate boundedness of the closed-loop systems. Compared with the existing reinforcement learning control results, the presented RLOTC algorithm can compensate for uncertain vessel dynamics and unknown disturbances, and obtain the optimized control performance by considering optimization in every backstepping design. Simulation studies on an underactuated surface vessel are given to illustrate the effectiveness of the RLOTC algorithm.

An efficient and lightweight off-policy actor–critic reinforcement learning framework

In the framework of current off-policy actor–critic methods, the state–action pairs in an experience replay buffer (called historical behaviors) cannot be used to improve the policy, and the target network and the clipped double Q-learning techniques need to be used to evaluate the policy. The framework limits the policy learning capability in complex environments, and needs to maintain four critic networks. As a result, we propose an efficient and lightweight off-policy actor–critic (EL-AC) framework. In the policy improvement, an efficient off-policy likelihood ratio policy gradient algorithm with historical behaviors reusing (PG-HBR) is proposed, which promotes the agent to learn an approximately optimal policy by using the historical behaviors. Moreover, a theoretically interpretable universal critic network is designed. It can approximate the action-value and the state-value functions simultaneously, so as to obtain the advantage function in PG-HBR. In the policy evaluation, we develop the algorithms of low-pass filtering for target state-values and adaptive controlling algorithm for overestimation bias, which can evaluate the policy efficiently and accurately using only one universal critic network. Extensive evaluation results indicate that EL-AC outperforms the state-of-the-art off-policy actor–critic methods in terms of approximately optimal policy learning and neural network storage space occupation, and it is more suitable for policy learning in complex environments.

Robust-optimization-guiding deep reinforcement learning for chemical material production scheduling

This study presents a novel guiding framework embedded with robust optimization (RO) for the training phase of reinforcement learning (RL), specifically tailored for dynamic scheduling of a single-stage multi-product chemical reactor in an uncertain environment. The proposed framework addresses the challenge of local optima in policy gradient methods by integrating optimization methods, enhancing both the objective value and learning stability of the RL. We further enhance the robustness of the proposed model against parameter distortions by incorporating RO as a guiding engine. A numerical study of chemical material production scheduling is conducted to validate the proposed model and the results demonstrate the effectiveness to address demand volatility with several metrics including sensitivity analysis, solution quality analysis, computational time, compared to a typical actor-critic RL.

Actor-Critic Reinforcement Learning Algorithms for Mean Field Games in Continuous Time, State and Action Spaces

Actor-Critic Reinforcement Learning Algorithms for Mean Field Games in Continuous Time, State and Action Spaces

Expert knowledge data-driven based actor–critic reinforcement learning framework to solve computationally expensive unit commitment problems with uncertain wind energy

With the expansion of power grid, unaffordable computational cost and time will pose serious challenges of time-efficient scheduling in unit commitment problem (UCP). However, existing optimization methods, i.e., mathematical programming methods and meta-heuristic algorithms, are powerless and time-consuming to handle computationally expensive UCP (CEUCP). Thus, reinforcement learning methods with strong inference and time-saving performances are motivated to solve the computationally expensive challenges in tackling CEUCPs. In this paper, a novel expert knowledge data-driven based actor–critic (AC) reinforcement learning methodology is proposed for solving CEUCPs. Specifically, in the proposed AC reinforcement learning methodology, expert knowledge, data-driven surrogate model, and improved meta-heuristic algorithm are integrated for further performance enhancement. Firstly, a novel action selection mechanism (based on the expert knowledge of thermal units characteristic) is integrated into AC to improve the efficiency of network training. Secondly, an improved extreme learning machine (ELM) data-driven surrogate model is proposed to build reward function in AC. In detail, original cost function in reward is replaced by a lightweight ELM model. Shape distance is integrated into ELM for enhancing accuracy. Finally, original marine predators algorithm (MPA) is improved for obtaining optimal dispatching decisions and rewards of AC method quickly and correctly. Original search pattern is replaced by quantum based representation for boosting convergence. The excellent performances of the proposed AC framework are verified by simulations of 10-units, 100-units, and 100-units with wind energy test systems.

Controlling estimation error in reinforcement learning via Reinforced Operation

In Value-based and Actor-Critic reinforcement learning (RL) methods, both inaccuracy and instability of value estimation will detrimentally affect their performance. Some typical RL methods, such as Maxmin Q-learning and QMD3, are plagued by the underestimation problem while failing to trade off the estimation bias and variance jointly. We propose the Reinforced Operation (RO) to address these shortcomings, which selects the closest median among multiple Q-function. RO is applicable to any model-free RL method. Theoretically, we introduce the Mean Square Error (MSE) to jointly analyze the estimation bias and variance of value estimation methods. We also demonstrate the superiority of RO in MSE reduction and give an upper bound for the estimation bias of value estimation methods with arbitrary distribution to guide the calculation of the estimation bias of value estimation methods. Based on RO, we propose the variants of Q-learning and TD3, Reinforced Q-learning (RQ) and Reinforced Delayed Deep Deterministic policy gradient (RD3), respectively, to tackle different tasks. We empirically demonstrate that our method can reduce estimation error and achieve superior performance on discrete and continuous benchmark tasks.

Energy management strategies based on soft actor critic reinforcement learning with a proper reward function design based on battery state of charge constraints

The environmental concerns have pushed our society to find solutions to reduce the emission of transportation systems. One promising solution is the fuel cell hybrid electric vehicle (FCHEV), with a proton exchange membrane fuel cell (PEMFC) and a lithium-ion battery. The presence of two different energy sources requires a smart energy management strategy (EMS) which must not only optimize energy consumption (conventional approaches), but also mitigate component’s degradation (New approaches : Health-conscious strategies). For that, a wide range of techniques has been proposed in the literature, and the most recent techniques are based on machine learning. Reinforcement learning (RL) is usually the most chosen strategy, but as in the optimization techniques, a strong design of the objective function (called reward function in RL) is required. The main challenge in the reward function is to encourage a proper battery state of charge (SOC) management, while reducing the energy consumption and avoiding the factors that accelerate the FC aging. The current advancement in the literature shows satisfactory results, but the reward function design have great improvement potential in the chosen approach. Indeed, most approaches in RL for SOC management with continuous actions are based on the reference SOC principle, which reduces the EMS ability to optimize the other part of the reward function. This paper reveals the challenges associated with the introduction of SOC limits in the reward function and proposes an approach, based on SOC boundaries in the reward function. The contribution allows to better consider the parts to optimize, as it gives more freedom than previous reference SOC techniques all reducing the consumption by 12.9%.

Reinforcement learning in building controls: A comparative study of algorithms considering model availability and policy representation

Reinforcement Learning (RL) has presented considerable potential as an advanced control technique in building controls to enable buildings operating more energy-efficient. As various types of RL algorithms have been studied on their performance of building controls, benchmarking these algorithms across the entire spectrum of features is essential to provide an overview and deepen the understanding of RL applications. Therefore, this study aims to compare and analyze the effectiveness of various RL algorithms, encompassing the entire RL categories featured by value-based, policy gradient, actor-critic and model-based RL considering model availability and policy representation. To provide a comprehensive analysis, in addition to the control performance quantified by the cumulative rewards based on the cost function of RL, data demand and robustness of hyperparameter tuning were investigated. The open-source Gym-Eplus framework was selected as the virtual environment to train and test different RL agents. The results showed that both model-free and model-based RL agents outperformed the baseline rule-based control in terms of energy consumption and thermal comfort, and RL agents were capable of evaluating both short-term and long-term rewards to achieve adaptive control optimization continuously along with the online control process. Model-based RL agent improved the data sampling efficiency but presented a relatively sacrificed control performance during the tested summer days.

Development of an Attention Mechanism for Task-Adaptive Heterogeneous Robot Teaming

The allure of team scale and functional diversity has led to the promising adoption of heterogeneous multi-robot systems (HMRS) in complex, large-scale operations such as disaster search and rescue, site surveillance, and social security. These systems, which coordinate multiple robots of varying functions and quantities, face the significant challenge of accurately assembling robot teams that meet the dynamic needs of tasks with respect to size and functionality, all while maintaining minimal resource expenditure. This paper introduces a pioneering adaptive cooperation method named inner attention (innerATT), crafted to dynamically configure teams of heterogeneous robots in response to evolving task types and environmental conditions. The innerATT method is articulated through the integration of an innovative attention mechanism within a multi-agent actor–critic reinforcement learning framework, enabling the strategic analysis of robot capabilities to efficiently form teams that fulfill specific task demands. To demonstrate the efficacy of innerATT in facilitating cooperation, experimental scenarios encompassing variations in task type (“Single Task”, “Double Task”, and “Mixed Task”) and robot availability are constructed under the themes of “task variety” and “robot availability variety.” The findings affirm that innerATT significantly enhances flexible cooperation, diminishes resource usage, and bolsters robustness in task fulfillment.

A reinforcement learning approach for selecting infill drilling locations considering long-term production planning in mining complexes with supply uncertainty

Simultaneous stochastic optimisation frameworks provide a method for optimising long-term production schedules in mining complexes that aim to maximise net present value and manage risk related to supply uncertainty. The uncertainty and local variability related to the quality and quantity of material in the mineral deposits are modelled with a set of stochastic orebody simulations, an input into the simultaneous stochastic optimisation framework. Infill drilling provides opportunities to collect additional information associated with the mineral deposits, which can inform future production scheduling decisions. A framework is developed for optimising infill drilling locations with a criterion that seeks areas that directly affect long-term planning decisions and requires the use of geostatistical simulations. Actor-critic reinforcement learning is applied to identify infill drilling locations in a copper mining complex using this criterion. The case study demonstrates that adapting production scheduling decisions given additional information has the potential to improve the associated production and financial forecasts and identifies a stable area for infill drilling.

An effective deep actor-critic reinforcement learning method for solving the flexible job shop scheduling problem

An effective deep actor-critic reinforcement learning method for solving the flexible job shop scheduling problem

Reinforcement learning-based adaptive tracking control for mismatched non-affine nonlinear systems with coupled uncertainties

This paper studies a reinforcement learning-based adaptive non-affine tracking control method for a class of uncertain mismatched non-affine nonlinear systems. The considered system is not only affected by external mismatched disturbances and internal uncertainties, but also influenced by the non-affine control structures. Firstly, an auxiliary integral system is developed for the purpose of isolating the non-affine control input. Secondly, by designing the actor-critic networks to evaluate the system control performance and generate the reinforcement signal, the unknown internal uncertainties can be handled. Thirdly, based on the output of reinforcement learning network, several disturbance compensation laws are constructed to address the adverse impact of matched and mismatched disturbances. As a result, a novel intelligent adaptive non-affine controller is proposed by integrating actor-critic reinforcement learning framework, disturbance compensation and adaptive laws. It has been proved that closed-loop system are stable and the tracking errors are bounded. The numerical simulation results show the effectiveness and superiority of the proposed method.

An experimental application of machine learning algorithms to optimize the FEL lasing via beam trajectory tuning at Dalian Coherent Light Source

The lasing optimization of Free-Electron Laser (FEL) facilities is a time-consuming and challenging task. Instead of operating manually by experienced operators, implementation of machine learning algorithms offers a rapid and adaptable approach for FEL lasing optimization. Recently, such an experiment has been conducted at the vacuum ultraviolet FEL facility - Dalian Coherent Light Source (DCLS). Four algorithms, namely the standard and the neural network-based genetic algorithms, the deep deterministic policy gradient and the soft actor critic reinforcement learning algorithms, have been employed to enhance the FEL intensity by optimizing the electron beam trajectory. These algorithms have shown notable efficacy in enhancing the FEL lasing, especially the reinforcement learning ones which achieved convergence within only approximately 400 iterations. This study demonstrates the validity of machine learning algorithms for FEL lasing optimization, providing a forward-looking perspective on the automatic operation of DCLS.

Mobile User Interface Adaptation Based on Usability Reward Model and Multi-Agent Reinforcement Learning

Today, reinforcement learning is one of the most effective machine learning approaches in the tasks of automatically adapting computer systems to user needs. However, implementing this technology into a digital product requires addressing a key challenge: determining the reward model in the digital environment. This paper proposes a usability reward model in multi-agent reinforcement learning. Well-known mathematical formulas used for measuring usability metrics were analyzed in detail and incorporated into the usability reward model. In the usability reward model, any neural network-based multi-agent reinforcement learning algorithm can be used as the underlying learning algorithm. This paper presents a study using independent and actor-critic reinforcement learning algorithms to investigate their impact on the usability metrics of a mobile user interface. Computational experiments and usability tests were conducted in a specially designed multi-agent environment for mobile user interfaces, enabling the implementation of various usage scenarios and real-time adaptations.

Optimizing Energy Resources in WSNs: ARIMA Feature Selection Meets Adaptive Reinforcement Learning

ABSTRACT Efficient energy management in WSNs is pivotal for prolonged network lifetime and sustained performance. This research introduces a novel approach to energy optimization through the integration of an ARIMA-driven feature selection process and an Actor-Critic Reinforcement Learning model. The Intel Lab dataset, encompassing data from 54 nodes distributed over a month, serves as the basis for experimentation. In the proposed methodology, the experimental setup ensures the validity of the study, leveraging the realism of the Intel Lab dataset. The data collection process captures the intricacies of WSN dynamics, with temperature records collected from strategically positioned nodes. The ARIMA-driven feature selection method refines the dataset, capturing temporal dependencies critical for energy prediction. The Actor-Critic model, a hybrid of policy and value-based reinforcement learning, dynamically adapts energy allocation strategies based on learned policies and offers a dynamic solution to the challenges posed by WSNs’ ever-changing environments. Comparative analysis with methods reveals lower energy consumption of 0.32 mJ, an extended network lifetime of 1501 rounds, and higher prediction accuracy of 98%. The study’s implications extend beyond the specific algorithms, suggesting a shift toward adaptive learning models in WSNs. The findings open avenues for future research in the integration of machine learning models for sustainable and efficient WSN deployments, emphasizing the growing importance of dynamic adaptation in sensor networks. Moreover, limitations, such as simulation realism and computational complexity, are acknowledged, prompting avenues for future research.

Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences.

Soft actor-critic (SAC) is an off-policy actor-critic (AC) reinforcement learning (RL) algorithm, essentially based on entropy regularization. SAC trains a policy by maximizing the trade-off between expected return and entropy (randomness in the policy). It has achieved the state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. SAC works in an off-policy fashion where data are sampled uniformly from past experiences (stored in a buffer) using which the parameters of the policy and value function networks are updated. We propose certain crucial modifications for boosting the performance of SAC and making it more sample efficient. In our proposed improved SAC (ISAC), we first introduce a new prioritization scheme for selecting better samples from the experience replay (ER) buffer. Second we use a mixture of the prioritized off-policy data with the latest on-policy data for training the policy and value function networks. We compare our approach with the vanilla SAC and some recent variants of SAC and show that our approach outperforms the said algorithmic benchmarks. It is comparatively more stable and sample efficient when tested on a number of continuous control tasks in MuJoCo environments.

Power System Frequency Dynamics Modeling, State Estimation, and Control using Neural Ordinary Differential Equations (NODEs) and Soft Actor-Critic (SAC) Machine Learning Approaches

With the global energy transition of the electric power system, grid control, supervision, and protection is becoming more challenging. With the increasing integration of renewable energy sources (RES), the system dynamics are changing, causing traditional power system dynamic modeling with swing equation-based modeling approaches to fail. Additionally, the converter-dominated power grid is decreasing the system inertia, making the power system more fragile to the frequency swings. This paper first investigates and compares the application of a model-based Kalman filter state estimation approach with (i) a model-free machine learning approach --- neural ordinary differential equations (NODEs) --- and (ii) a data-driven system identification (SysId) approach to model and infer critical state values of the power system frequency dynamics. Then a model predictive control (MPC) framework is compared to a model-free Soft Actor-Critic (SAC) reinforcement learning (RL) control algorithm in providing efficient fast frequency response (FFR) to the power system frequency dynamics. The approaches are compared in terms of their performance goals as well as their per-timestep computational efficiency. The comparative study for state estimation shows that for the model-free requirement, both NODEs and SysId can provide accurate state estimates; however, with increasing model complexity, NODEs can be a better choice for model identification. Similarly, the results from the FFR comparative study show that the SAC RL-based FFR, once trained, outperforms MPC with better control signals and faster computation time, making the SAC RL-based FFR better option for providing FFR to the power system.

One-shot sim-to-real transfer policy for robotic assembly via reinforcement learning with visual demonstration

AbstractReinforcement learning (RL) has been successfully applied to a wealth of robot manipulation tasks and continuous control problems. However, it is still limited to industrial applications and suffers from three major challenges: sample inefficiency, real data collection, and the gap between simulator and reality. In this paper, we focus on the practical application of RL for robot assembly in the real world. We apply enlightenment learning to improve the proximal policy optimization, an on-policy model-free actor-critic reinforcement learning algorithm, to train an agent in Cartesian space using the proprioceptive information. We introduce enlightenment learning incorporated via pretraining, which is beneficial to reduce the cost of policy training and improve the effectiveness of the policy. A human-like assembly trajectory is generated through a two-step method with segmenting objects by locations and iterative closest point for pretraining. We also design a sim-to-real controller to correct the error while transferring to reality. We set up the environment in the MuJoCo simulator and demonstrated the proposed method on the recently established The National Institute of Standards and Technology (NIST) gear assembly benchmark. The paper introduces a unique framework that enables a robot to learn assembly tasks efficiently using limited real-world samples by leveraging simulations and visual demonstrations. The comparative experiment results indicate that our approach surpasses other baseline methods in terms of training speed, success rate, and efficiency.

Reinforcement learning-based optimal control of uncertain nonlinear systems

This paper considers the optimal control of a second-order nonlinear system with unknown dynamics. A new reinforcement learning based approach is proposed with the aid of direct adaptive control. By the new approach, actor-critic reinforcement learning algorithms are proposed with neural network approximation. Simulation results are presented to show the effectiveness of the proposed algorithms.

Computing Task Offloading in Vehicular Edge Network via Deep Reinforcement Learning

Background: In recent years, with the development of the Internet of Vehicles, a variety of novel in-vehicle application devices have surfaced, exhibiting increasingly stringent requirements for time delay. Vehicular edge networks (VEN) can fully use network edge devices, such as roadside units (RSUs), for collaborative processing, which can effectively reduce latency. background: In recent years, with the development of the field of internet of vehicles, a variety of novel in-vehicle application devices have surfaced, exhibiting increasingly stringent requirements for time delay. Vehicular edge network (VEN) can make full use of network edge devices, such as road side unit (RSU) for collaborative processing, which can effectively reduce the latency. Objective: Most extant studies, including patents, assume that RSU has sufficient computing resources to provide unlimited services. But in fact, its computing resources will be limited with the increase in processing tasks, which will restrict the delay-sensitive vehicular applications. To solve this problem, a vehicle-to-vehicle computing task offloading method based on deep reinforcement learning is proposed in this paper, which fully considers the remaining available computational resources of neighboring vehicles to minimize the total task processing latency and enhance the offloading success rate. objective: A vehicle-to-vehicle computing task offloading method based on deep reinforce-ment learning is proposed in this paper, which fully considers the remaining available computa-tional resources of neighboring vehicles with the objective of minimizing the total task processing latency and enhancing the offloading success rate. Methods: In the multi-service vehicle scenario, the analytic hierarchy process (AHP) was first used to prioritize the computing tasks of user vehicles. Next, an improved sequence-to-sequence (Seq2Seq) computing task scheduling model combined with an attention mechanism was designed, and the model was trained by an actor-critic (AC) reinforcement learning algorithm with the optimization goal of reducing the processing delay of computing tasks and improving the success rate of offloading. A task offloading strategy optimization model based on AHP-AC was obtained on this basis. Results: The average latency and execution success rate are used as performance metrics to compare the proposed method with three other task offloading methods: only-local processing, greedy strategy- based algorithm, and random algorithm. In addition, experimental validation in terms of CPU frequency and the number of SVs is carried out to demonstrate the excellent generalization ability of the proposed method. result: The average latency and execution success rate are used as performance metrics to compare the proposed method with three other task offloading methods: only-local processing, greedy strate-gy-based algorithm and random algorithm. In addition, experimental validation in terms of both CPU frequency and the number of SVs is carried out to demonstrate the good generalization abil-ity of the proposed method. Conclusion: The simulation results reveal that the proposed method outperforms other methods in reducing the processing delay of tasks and improving the success rate of task offloading, which solves the problem of limited execution of delay-sensitive tasks caused by insufficient computational resources.