Actor-critic Algorithm Research Articles

RLfOLD: Reinforcement Learning from Online Demonstrations in Urban Autonomous Driving

Reinforcement Learning from Demonstrations (RLfD) has emerged as an effective method by fusing expert demonstrations into Reinforcement Learning (RL) training, harnessing the strengths of both Imitation Learning (IL) and RL. However, existing algorithms rely on offline demonstrations, which can introduce a distribution gap between the demonstrations and the actual training environment, limiting their performance. In this paper, we propose a novel approach, Reinforcement Learning from Online Demonstrations (RLfOLD), that leverages online demonstrations to address this limitation, ensuring the agent learns from relevant and up-to-date scenarios, thus effectively bridging the distribution gap. Unlike conventional policy networks used in typical actor-critic algorithms, RLfOLD introduces a policy network that outputs two standard deviations: one for exploration and the other for IL training. This novel design allows the agent to adapt to varying levels of uncertainty inherent in both RL and IL. Furthermore, we introduce an exploration process guided by an online expert, incorporating an uncertainty-based technique. Our experiments on the CARLA NoCrash benchmark demonstrate the effectiveness and efficiency of RLfOLD. Notably, even with a significantly smaller encoder and a single camera setup, RLfOLD surpasses state-of-the-art methods in this evaluation. These results, achieved with limited resources, highlight RLfOLD as a highly promising solution for real-world applications.

Adaptive Control for Virtual Synchronous Generator Parameters Based on Soft Actor Critic.

This paper introduces a model-free optimization method based on reinforcement learning (RL) aimed at resolving the issues of active power and frequency oscillations present in a traditional virtual synchronous generator (VSG). The RL agent utilizes the active power and frequency response of the VSG as state information inputs and generates actions to adjust the virtual inertia and damping coefficients for an optimal response. Distinctively, this study incorporates a setting-time term into the reward function design, alongside power and frequency deviations, to avoid prolonged system transients due to over-optimization. The soft actor critic (SAC) algorithm is utilized to determine the optimal strategy. SAC, being model-free with fast convergence, avoids policy overestimation bias, thus achieving superior convergence results. Finally, the proposed method is validated through MATLAB/Simulink simulation. Compared to other approaches, this method more effectively suppresses oscillations in active power and frequency and significantly reduces the setting time.

Multi-Objective Advantage Actor-Critic Algorithm for Hybrid Disassembly Line Balancing with Multi-Skilled Workers

The scheduling of disassembly lines is of great importance to achieve optimized productivity. In this paper, we address the Hybrid Disassembly Line Balancing Problem that combines linear disassembly lines and U-shaped disassembly lines, considering multi-skilled workers, and targeting profit and carbon emissions. In contrast to common approaches in reinforcement learning that typically employ weighting strategies to solve multi-objective problems, our approach innovatively incorporates non-dominated ranking directly into the reward function. The exploration of Pareto frontier solutions or better solutions is moderated by comparing performance between solutions and dynamically adjusting rewards based on the occurrence of repeated solutions. The experimental results show that the multi-objective Advantage Actor-Critic algorithm based on Pareto optimization exhibits superior performance in terms of metrics superiority in the comparison of six experimental cases of different scales, with an excellent metrics comparison rate of 70%. In some of the experimental cases in this paper, the solutions produced by the multi-objective Advantage Actor-Critic algorithm show some advantages over other popular algorithms such as the Deep Deterministic Policy Gradient Algorithm, the Soft Actor-Critic Algorithm, and the Non-Dominated Sorting Genetic Algorithm II. This further corroborates the effectiveness of our proposed solution.

Deep reinforcement learning navigation via decision transformer in autonomous driving.

In real-world scenarios, making navigation decisions for autonomous driving involves a sequential set of steps. These judgments are made based on partial observations of the environment, while the underlying model of the environment remains unknown. A prevalent method for resolving such issues is reinforcement learning, in which the agent acquires knowledge through a succession of rewards in addition to fragmentary and noisy observations. This study introduces an algorithm named deep reinforcement learning navigation via decision transformer (DRLNDT) to address the challenge of enhancing the decision-making capabilities of autonomous vehicles operating in partially observable urban environments. The DRLNDT framework is built around the Soft Actor-Critic (SAC) algorithm. DRLNDT utilizes Transformer neural networks to effectively model the temporal dependencies in observations and actions. This approach aids in mitigating judgment errors that may arise due to sensor noise or occlusion within a given state. The process of extracting latent vectors from high-quality images involves the utilization of a variational autoencoder (VAE). This technique effectively reduces the dimensionality of the state space, resulting in enhanced training efficiency. The multimodal state space consists of vector states, including velocity and position, which the vehicle's intrinsic sensors can readily obtain. Additionally, latent vectors derived from high-quality images are incorporated to facilitate the Agent's assessment of the present trajectory. Experiments demonstrate that DRLNDT may achieve a superior optimal policy without prior knowledge of the environment, detailed maps, or routing assistance, surpassing the baseline technique and other policy methods that lack historical data.

Adaptive critic design with weight allocation for intelligent learning control of wastewater treatment plants

With the deepening of modernization and industrialization, the issues of water pollution and scarcity have become more pressing. To address these issues, many wastewater treatment factories have been built to improve the reuse of water resources. However, the control of the wastewater treatment process (WWTP) is a complex task due to the highly nonlinear and strongly coupled nature. It is challenging to develop the accurate mechanism models of the wastewater treatment system. The improvement of the efficiency for the WWTP is crucial to safeguard the urban ecological environment. In this paper, adaptive critic with weight allocation (ACWA) is developed to address the optimal control problem in the WWTP. Different from the previous methods of the WWTP, system modeling is not adopted in this paper, which meets the actual physical background of the wastewater treatment system to a great extent. In addition, the actor-critic algorithm in reinforcement learning is used as the basic structure in the ACWA. It is worth noting that a novel weighted action-value function and the advantage function are introduced in the weight updating process of the action network and the critic network. The experimental results show that the control accuracy of the ACWA is greatly improved compared with the previous control methods.

An applied deep reinforcement learning approach to control active networked microgrids in smart cities with multi-level participation of battery energy storage system and electric vehicles

This study proposed an intelligent energy management strategy for islanded networked microgrids (NMGs) in smart cities considering the renewable energy sources uncertainties and power fluctuations. Energy management of active power and frequency control approach is based on the intelligent probabilistic wavelet fuzzy neural network-deep reinforcement learning algorithm (IPWFNN-DRLA). The control strategy is formulated with deep reinforcement learning approach based on the Markov decision process and solved by the soft actor-critic algorithm. The NMG local controller (NMGLC) provides information such as the frequency, active power, power generation data, and status of the electric vehicle's battery energy storage system to the NMG central controller (NMGCC). Then the NMGCC calculates active power and frequency support based on the IPWFNN-DRLA approach and sends the results to the NMGLC. The proposed model is developed in a continuous problem-solving space with two structures of offline training and decentralized distributed operation. For this purpose, each NMG has a control agent (NMGCA) based on the IPWFNN algorithm, and the NMGCA learning model is formulated based the online back-propagation learning algorithm. The proposed approach demonstrates a computation accuracy exceeding 98%, along with a 7.82% reduction in computational burden and a 61.1% reduction in computation time compared to alternative methods.

An Optimized Advantage Actor-Critic Algorithm for Disassembly Line Balancing Problem Considering Disassembly Tool Degradation

The growing emphasis on ecological preservation and natural resource conservation has significantly advanced resource recycling, facilitating the realization of a sustainable green economy. Essential to resource recycling is the pivotal stage of disassembly, wherein the efficacy of disassembly tools plays a critical role. This work investigates the impact of disassembly tools on disassembly duration and formulates a mathematical model aimed at minimizing workstation cycle time. To solve this model, we employ an optimized advantage actor-critic algorithm within reinforcement learning. Furthermore, it utilizes the CPLEX solver to validate the model’s accuracy. The experimental results obtained from CPLEX not only confirm the algorithm’s viability but also enable a comparative analysis against both the original advantage actor-critic algorithm and the actor-critic algorithm. This comparative work verifies the superiority of the proposed algorithm.

Enabling intelligent transferable energy management of series hybrid electric tracked vehicle across motion dimensions via soft actor-critic algorithm

Due to the complex driving conditions faced by hybrid electric tracked vehicles, energy management is crucial for improving fuel economy. However, developing an energy management strategy (EMS) is a time-consuming and labor-intensive task, which is challenging to generalize across different driving tasks. To solve this problem and shorten the development cycle of EMSs, this article proposes a novel transferable energy management framework for a series hybrid electric tracked vehicle (SHETV) across motion dimensions. To fully reuse the learned knowledge from longitudinal motion into both longitudinal and lateral motion, this framework merges transfer learning (TL) into the state-of-the-art deep reinforcement learning (DRL) algorithm, soft actor-critic (SAC), to formulate a novel deep transfer reinforcement learning (DTRL) method, with the transfer of both the neural networks and the pre-trained experience replay buffer. Simulation results indicate that the proposed EMS accelerates the convergence speed by 75.38%, enhances the learning ability by 19.05%, and improves the fuel economy by 5.08% compared to the baseline EMS. This article contributes to correlating different energy management tasks and reusing the existing EMS for the rapid development of a new EMS of the hybrid electric tracked vehicle.

Parameter-Free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients

Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a high variance, deep actor-critic approaches that overcome the overestimation bias lead to a substantial underestimation bias. We first address the detrimental issues in the existing approaches that aim to overcome such underestimation error. Then, through extensive statistical analysis, we introduce a novel, parameter-free Deep Q-learning variant to reduce this underestimation bias in deterministic policy gradients. By sampling the weights of a linear combination of two approximate critics from a highly shrunk estimation bias interval, our Q-value update rule is not affected by the variance of the rewards received by the agents throughout learning. We test the performance of the introduced improvement on a set of MuJoCo and Box2D continuous control tasks and demonstrate that it outperforms the existing approaches and improves the baseline actor-critic algorithm in most of the environments tested.

Energy optimization management of microgrid using improved soft actor-critic algorithm

To tackle the challenges associated with variability and uncertainty in distributed power generation, as well as the complexities of solving high-dimensional energy management mathematical models in mi-crogrid energy optimization, a microgrid energy optimization management method is proposed based on an improved soft actor-critic algorithm. In the proposed method, the improved soft actor-critic algorithm employs an entropy-based objective function to encourage target exploration without assigning signifi-cantly higher probabilities to any part of the action space, which can simplify the analysis process of distributed power generation variability and uncertainty while effectively mitigating the convergence fragility issues in solving the high-dimensional mathematical model of microgrid energy management. The effectiveness of the proposed method is validated through a case study analysis of microgrid energy op-timization management. The results revealed an increase of 51.20%, 52.38%, 13.43%, 16.50%, 58.26%, and 36.33% in the total profits of a microgrid compared with the Deep Q-network algorithm, the state-action-reward-state-action algorithm, the proximal policy optimization algorithm, the ant-colony based algorithm, a microgrid energy optimization management strategy based on the genetic algorithm and the fuzzy inference system, and the theoretical retailer stragety, respectively. Additionally, com-pared with other methods and strategies, the proposed method can learn more optimal microgrid energy management behaviors and anticipate fluctuations in electricity prices and demand.

Adaptive Evolutionary Reinforcement Learning with Policy Direction

Evolutionary Reinforcement Learning (ERL) has garnered widespread attention in recent years due to its inherent robustness and parallelism. However, the integration of Evolutionary Algorithms (EAs) and Reinforcement Learning (RL) remains relatively rudimentary and lacks dynamism, which can impact the convergence performance of ERL algorithms. In this study, a dynamic adaptive module is introduced to balance the Evolution Strategies (ES) and RL training within ERL. By incorporating elite strategies, this module leverages advantageous individuals to elevate the overall population's performance. Additionally, RL strategy updates often lack guidance from the population. To address this, we incorporate the strategies of the best individuals from the population, providing valuable policy direction. This is achieved through the formulation of a loss function that employs either L1 or L2 regularization to facilitate RL training. The proposed framework is referred to as Adaptive Evolutionary Reinforcement Learning (AERL). The effectiveness of our framework is evaluated by adopting Soft Actor-Critic (SAC) as the RL algorithm and comparing it with other algorithms in the MuJoCo environment. The results underscore the outstanding convergence performance of our proposed Adaptive Evolutionary Soft Actor-Critic (AESAC) algorithm. Furthermore, ablation experiments are conducted to emphasize the necessity of these two improvements. It is worth noting that the enhancements in AESAC are realized at the population level, enabling broader exploration and effectively reducing the risk of falling into local optima.

Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines

Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic (A-C) scheme for continuous state-action spaces is developed based on the deep deterministic policy gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing (QA), both simulated and with D-Wave QA hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research. Among others, the hybrid A-C agent is evaluated on the actual electron beam line of the Advanced Wakefield Experiment (AWAKE).

A Supervised Reinforcement Learning Algorithm for Controlling Drone Hovering

The application of drones carrying different devices for aerial hovering operations is becoming increasingly widespread, but currently there is very little research relying on reinforcement learning methods for hovering control, and it has not been implemented on physical machines. Drone’s behavior space regarding hover control is continuous and large-scale, making it difficult for basic algorithms and value-based reinforcement learning (RL) algorithms to have good results. In response to this issue, this article applies a watcher-actor-critic (WAC) algorithm to the drone’s hover control, which can quickly lock the exploration direction and achieve high robustness of the drone’s hover control while improving learning efficiency and reducing learning costs. This article first utilizes the actor-critic algorithm based on behavioral value Q (QAC) and the deep deterministic policy gradient algorithm (DDPG) for drone hover control learning. Subsequently, an actor-critic algorithm with an added watcher is proposed, in which the watcher uses a PID controller with parameters provided by a neural network as the dynamic monitor, transforming the learning process into supervised learning. Finally, this article uses a classic reinforcement learning environment library, Gym, and a current mainstream reinforcement learning framework, PARL, for simulation, and deploys the algorithm to a practical environment. A multi-sensor fusion strategy-based autonomous localization method for unmanned aerial vehicles is used for practical exercises. The simulation and experimental results show that the training episodes of WAC are reduced by 20% compared to the DDPG and 55% compared to the QAC, and the proposed algorithm has a higher learning efficiency, faster convergence speed, and smoother hovering effect compared to the QAC and DDPG.

Joint Access Points-User Association and Caching Placement Strategy for Cell-Free Massive MIMO Systems Based on Soft Actor-Critic Algorithm

Joint Access Points-User Association and Caching Placement Strategy for Cell-Free Massive MIMO Systems Based on Soft Actor-Critic Algorithm

Nonzero-sum games using actor-critic neural networks: A dynamic event-triggered adaptive dynamic programming

This paper mainly investigates the nonzero-sum games of nonlinear systems with unmatched uncertainty by using actor-critic neural networks. To handle the unmatched components, an auxiliary system with a modified value function is constructed, which transforms the robust stabilization issue into the optimal control issue. Then, a novel dynamic event-triggering condition is designed to further save bandwidth via introducing a dynamic variable. In addition, the actor-critic algorithm is employed in adaptive dynamic programming to achieve Nash equilibrium, which is tuned together with the control policy. By constructing appropriate Lyapunov functions, a criterion is established to ensure that the considered system is uniformly ultimately bounded. Finally, the effectiveness of the developed strategy is demonstrated by an example.

An Applied Framework for Smarter Buildings Exploiting a Self-Adapted Advantage Weighted Actor-Critic

Smart buildings are rapidly becoming more prevalent, aiming to create energy-efficient and comfortable living spaces. Nevertheless, the design of a smart building is a multifaceted approach that faces numerous challenges, with the primary one being the algorithm needed for energy management. In this paper, the design of a smart building, with a particular emphasis on the algorithm for controlling the indoor environment, is addressed. The implementation and evaluation of the Advantage-Weighted Actor-Critic algorithm is examined in a four-unit residential simulated building. Moreover, a novel self-adapted Advantage-Weighted Actor-Critic algorithm is proposed, tested, and evaluated in both the simulated and real building. The results underscore the effectiveness of the proposed control strategy compared to Rule-Based Controllers, Deep Deterministic Policy Gradient, and Advantage-Weighted Actor-Critic. Experimental results demonstrate a 34.91% improvement compared to the Deep Deterministic Policy Gradient and a 2.50% increase compared to the best Advantage-Weighted Actor-Critic method in the first epoch during a real-life scenario. These findings solidify the Self-Adapted Advantage-Weighted Actor-Critic algorithm’s efficacy, positioning it as a promising and advanced solution in the realm of smart building optimization.

Deep Reinforcement Learning for Autonomous Driving with an Auxiliary Actor Discriminator.

In the research of robot systems, path planning and obstacle avoidance are important research directions, especially in unknown dynamic environments where flexibility and rapid decision makings are required. In this paper, a state attention network (SAN) was developed to extract features to represent the interaction between an intelligent robot and its obstacles. An auxiliary actor discriminator (AAD) was developed to calculate the probability of a collision. Goal-directed and gap-based navigation strategies were proposed to guide robotic exploration. The proposed policy was trained through simulated scenarios and updated by the Soft Actor-Critic (SAC) algorithm. The robot executed the action depending on the AAD output. Heuristic knowledge (HK) was developed to prevent blind exploration of the robot. Compared to other methods, adopting our approach in robot systems can help robots converge towards an optimal action strategy. Furthermore, it enables them to explore paths in unknown environments with fewer moving steps (showing a decrease of 33.9%) and achieve higher average rewards (showning an increase of 29.15%).

Cooperative multi-agent game based on reinforcement learning

Multi-agent reinforcement learning holds tremendous potential for revolutionizing intelligent systems across diverse domains. However, it is also concomitant with a set of formidable challenges, which include the effective allocation of credit values to each agent, real-time collaboration among heterogeneous agents, and an appropriate reward function to guide agent behavior. To handle these issues, we propose an innovative solution named the Graph Attention Counterfactual Multiagent Actor–Critic algorithm (GACMAC). This algorithm encompasses several key components: First, it employs a multi-agent actor–critic framework along with counterfactual baselines to assess the individual actions of each agent. Second, it integrates a graph attention network to enhance real-time collaboration among agents, enabling heterogeneous agents to effectively share information during handling tasks. Third, it incorporates prior human knowledge through a potential-based reward shaping method, thereby elevating the convergence speed and stability of the algorithm. We tested our algorithm on the StarCraft Multi-Agent Challenge (SMAC) platform, which is a recognized platform for testing multi-agent algorithms, and our algorithm achieved a win rate of over 95% on the platform, comparable to the current state-of-the-art multi-agent controllers.

Automated Skin Cancer Diagnosis and Localization Using Deep Reinforcement Learning

Skin cancer is a significant and life-threatening medical condition caused by abnormal cell growth in the skin. Early detection and accurate diagnosis are critical for effective treatment and improved patient outcomes. However, manual diagnosis of skin cancer can be time-consuming and prone to inter-observer variability, necessitating the development of automated and objective diagnostic approaches. This research purports to diagnose and localize skin cancer automatically utilizing a novel Deep Reinforcement Learning technique. The proposed approach leverages a modified version of the Asynchronous Advantage Actor Critic (A3C) algorithm, comprising multiple independent agents represented as Convolutional Neural Networks (CNNs). These agents interact with the environment (skin images) and perform segmentation actions based on policies that maximize expected rewards, promoting accurate segmentation and penalizing false negatives as well as false positives. The research employs three diverse skin cancer datasets, namely PH2, ISIC 2018, and ISIC 2017 in order to assess the proposed approach’s efficiency. Quantitative evaluation metrics, including Precision, Specificity, Accuracy, Jaccard Index (IoU), Sensitivity, Dice coefficient, and F1-score are used to assess segmentation accuracy. The proposed method is compared with advanced deep-learning algorithms commonly used for skin cancer segmentation. The results demonstrate the superiority of the proposed approach in accurately localizing cancerous regions under various challenging conditions, such as hair presence, image blurriness, and the presence of other objects or obstacles. The proposed approach outperforms existing methods in terms of segmentation accuracy (98.8%) in diverse skin image sources.

Path Following for Autonomous Mobile Robots with Deep Reinforcement Learning.

Autonomous mobile robots have become integral to daily life, providing crucial services across diverse domains. This paper focuses on path following, a fundamental technology and critical element in achieving autonomous mobility. Existing methods predominantly address tracking through steering control, neglecting velocity control or relying on path-specific reference velocities, thereby constraining their generality. In this paper, we propose a novel approach that integrates the conventional pure pursuit algorithm with deep reinforcement learning for a nonholonomic mobile robot. Our methodology employs pure pursuit for steering control and utilizes the soft actor-critic algorithm to train a velocity control strategy within randomly generated path environments. Through simulation and experimental validation, our approach exhibits notable advancements in path convergence and adaptive velocity adjustments to accommodate paths with varying curvatures. Furthermore, this method holds the potential for broader applicability to vehicles adhering to nonholonomic constraints beyond the specific model examined in this paper. In summary, our study contributes to the progression of autonomous mobility by harmonizing conventional algorithms with cutting-edge deep reinforcement learning techniques, enhancing the robustness of path following.