Actor-critic Reinforcement Learning Algorithm Research Articles

Actor-Critic Reinforcement Learning Algorithms for Mean Field Games in Continuous Time, State and Action Spaces

Actor-Critic Reinforcement Learning Algorithms for Mean Field Games in Continuous Time, State and Action Spaces

An experimental application of machine learning algorithms to optimize the FEL lasing via beam trajectory tuning at Dalian Coherent Light Source

The lasing optimization of Free-Electron Laser (FEL) facilities is a time-consuming and challenging task. Instead of operating manually by experienced operators, implementation of machine learning algorithms offers a rapid and adaptable approach for FEL lasing optimization. Recently, such an experiment has been conducted at the vacuum ultraviolet FEL facility - Dalian Coherent Light Source (DCLS). Four algorithms, namely the standard and the neural network-based genetic algorithms, the deep deterministic policy gradient and the soft actor critic reinforcement learning algorithms, have been employed to enhance the FEL intensity by optimizing the electron beam trajectory. These algorithms have shown notable efficacy in enhancing the FEL lasing, especially the reinforcement learning ones which achieved convergence within only approximately 400 iterations. This study demonstrates the validity of machine learning algorithms for FEL lasing optimization, providing a forward-looking perspective on the automatic operation of DCLS.

Mobile User Interface Adaptation Based on Usability Reward Model and Multi-Agent Reinforcement Learning

Today, reinforcement learning is one of the most effective machine learning approaches in the tasks of automatically adapting computer systems to user needs. However, implementing this technology into a digital product requires addressing a key challenge: determining the reward model in the digital environment. This paper proposes a usability reward model in multi-agent reinforcement learning. Well-known mathematical formulas used for measuring usability metrics were analyzed in detail and incorporated into the usability reward model. In the usability reward model, any neural network-based multi-agent reinforcement learning algorithm can be used as the underlying learning algorithm. This paper presents a study using independent and actor-critic reinforcement learning algorithms to investigate their impact on the usability metrics of a mobile user interface. Computational experiments and usability tests were conducted in a specially designed multi-agent environment for mobile user interfaces, enabling the implementation of various usage scenarios and real-time adaptations.

Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences.

Soft actor-critic (SAC) is an off-policy actor-critic (AC) reinforcement learning (RL) algorithm, essentially based on entropy regularization. SAC trains a policy by maximizing the trade-off between expected return and entropy (randomness in the policy). It has achieved the state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. SAC works in an off-policy fashion where data are sampled uniformly from past experiences (stored in a buffer) using which the parameters of the policy and value function networks are updated. We propose certain crucial modifications for boosting the performance of SAC and making it more sample efficient. In our proposed improved SAC (ISAC), we first introduce a new prioritization scheme for selecting better samples from the experience replay (ER) buffer. Second we use a mixture of the prioritized off-policy data with the latest on-policy data for training the policy and value function networks. We compare our approach with the vanilla SAC and some recent variants of SAC and show that our approach outperforms the said algorithmic benchmarks. It is comparatively more stable and sample efficient when tested on a number of continuous control tasks in MuJoCo environments.

Reinforcement learning-based optimal control of uncertain nonlinear systems

This paper considers the optimal control of a second-order nonlinear system with unknown dynamics. A new reinforcement learning based approach is proposed with the aid of direct adaptive control. By the new approach, actor-critic reinforcement learning algorithms are proposed with neural network approximation. Simulation results are presented to show the effectiveness of the proposed algorithms.

One-shot sim-to-real transfer policy for robotic assembly via reinforcement learning with visual demonstration

AbstractReinforcement learning (RL) has been successfully applied to a wealth of robot manipulation tasks and continuous control problems. However, it is still limited to industrial applications and suffers from three major challenges: sample inefficiency, real data collection, and the gap between simulator and reality. In this paper, we focus on the practical application of RL for robot assembly in the real world. We apply enlightenment learning to improve the proximal policy optimization, an on-policy model-free actor-critic reinforcement learning algorithm, to train an agent in Cartesian space using the proprioceptive information. We introduce enlightenment learning incorporated via pretraining, which is beneficial to reduce the cost of policy training and improve the effectiveness of the policy. A human-like assembly trajectory is generated through a two-step method with segmenting objects by locations and iterative closest point for pretraining. We also design a sim-to-real controller to correct the error while transferring to reality. We set up the environment in the MuJoCo simulator and demonstrated the proposed method on the recently established The National Institute of Standards and Technology (NIST) gear assembly benchmark. The paper introduces a unique framework that enables a robot to learn assembly tasks efficiently using limited real-world samples by leveraging simulations and visual demonstrations. The comparative experiment results indicate that our approach surpasses other baseline methods in terms of training speed, success rate, and efficiency.

Computing Task Offloading in Vehicular Edge Network via Deep Reinforcement Learning

Background: In recent years, with the development of the Internet of Vehicles, a variety of novel in-vehicle application devices have surfaced, exhibiting increasingly stringent requirements for time delay. Vehicular edge networks (VEN) can fully use network edge devices, such as roadside units (RSUs), for collaborative processing, which can effectively reduce latency. background: In recent years, with the development of the field of internet of vehicles, a variety of novel in-vehicle application devices have surfaced, exhibiting increasingly stringent requirements for time delay. Vehicular edge network (VEN) can make full use of network edge devices, such as road side unit (RSU) for collaborative processing, which can effectively reduce the latency. Objective: Most extant studies, including patents, assume that RSU has sufficient computing resources to provide unlimited services. But in fact, its computing resources will be limited with the increase in processing tasks, which will restrict the delay-sensitive vehicular applications. To solve this problem, a vehicle-to-vehicle computing task offloading method based on deep reinforcement learning is proposed in this paper, which fully considers the remaining available computational resources of neighboring vehicles to minimize the total task processing latency and enhance the offloading success rate. objective: A vehicle-to-vehicle computing task offloading method based on deep reinforce-ment learning is proposed in this paper, which fully considers the remaining available computa-tional resources of neighboring vehicles with the objective of minimizing the total task processing latency and enhancing the offloading success rate. Methods: In the multi-service vehicle scenario, the analytic hierarchy process (AHP) was first used to prioritize the computing tasks of user vehicles. Next, an improved sequence-to-sequence (Seq2Seq) computing task scheduling model combined with an attention mechanism was designed, and the model was trained by an actor-critic (AC) reinforcement learning algorithm with the optimization goal of reducing the processing delay of computing tasks and improving the success rate of offloading. A task offloading strategy optimization model based on AHP-AC was obtained on this basis. Results: The average latency and execution success rate are used as performance metrics to compare the proposed method with three other task offloading methods: only-local processing, greedy strategy- based algorithm, and random algorithm. In addition, experimental validation in terms of CPU frequency and the number of SVs is carried out to demonstrate the excellent generalization ability of the proposed method. result: The average latency and execution success rate are used as performance metrics to compare the proposed method with three other task offloading methods: only-local processing, greedy strate-gy-based algorithm and random algorithm. In addition, experimental validation in terms of both CPU frequency and the number of SVs is carried out to demonstrate the good generalization abil-ity of the proposed method. Conclusion: The simulation results reveal that the proposed method outperforms other methods in reducing the processing delay of tasks and improving the success rate of task offloading, which solves the problem of limited execution of delay-sensitive tasks caused by insufficient computational resources.

Airfoils optimization based on deep reinforcement learning to improve the aerodynamic performance of rotors

Airfoil optimization is the key to improving the aerodynamic performance of a rotor. However, conventional optimization approaches cannot modify the airfoil shape intelligently in the way that an aircraft designer would. Because the optimization process is largely uninterpretable and ungeneralizable, the optimization knowledge and experience cannot be extracted and applied to similar optimization tasks. To address these issues, we propose an optimization framework for rotor airfoils based on deep reinforcement learning (DRL). Our DRL-based framework is capable of learning an interpretable and generalizable optimization strategy for alleviating the dynamic stall of a rotor airfoil. First, to enhance the efficiency of airfoil dynamic stall optimization, a deep neural network is trained to predict the dynamic stall hysteresis loops of rotor airfoils as a surrogate model. The asynchronous advantage actor–critic reinforcement learning algorithm is employed to train and learn the optimization strategy for alleviating the dynamic stall of the rotor airfoil. Next, the OA212 rotor airfoil is optimized using the well-trained optimization strategy. The results show that the dynamic stall characteristics of the airfoil are improved after optimization. The lift coefficients of the optimized airfoil are significantly enhanced, and the drag and moment coefficients peaks are reduced by 46.3% and 73.8%, respectively, compared with the baseline airfoil. Then, 20 airfoils are optimized using the well-trained optimization strategy to evaluate the generalizability of the dynamic stall optimization strategy. The results demonstrate that the optimization strategy learned by DRL for the rotor airfoil optimization is generalizable. Finally, the numerical simulation results comparing a rotor with baseline airfoils to one with optimized airfoils demonstrate that the aerodynamic performance of the optimized rotor is superior to that of the baseline rotor.

Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market

We propose a reinforcement learning (RL) approach to solve the continuous-time mean-variance portfolio selection problem in a regime-switching market, where the market regime is unobservable. To encourage exploration for learning, we formulate an exploratory stochastic control problem with an entropy-regularized mean-variance objective. We obtain semi-analytical representations of the optimal value function and optimal policy, which involve unknown solutions to two linear parabolic partial differential equations (PDEs). We utilize these representations to parametrize the value function and policy for learning with the unknown solutions to the PDEs approximated based on polynomials. We develop an actor-critic RL algorithm to learn the optimal policy through interactions with the market environment. The algorithm carries out filtering to obtain the belief probability of the market regime and performs policy evaluation and policy gradient updates alternately. Empirical results demonstrate the advantages of our RL algorithm in relatively long-term investment problems over the classical control approach and an RL algorithm developed for the continuous-time mean-variance problem without considering regime switches.

A soft actor-critic reinforcement learning algorithm for network intrusion detection

Network intrusion detection plays a very important role in network security. Although current deep learning-based intrusion detection algorithms have achieved good detection performance, there are still limitations in dealing with unbalanced datasets and identifying minority attacks and unknown attacks. In this paper, we propose an intrusion detection model AE-SAC based on adversarial environment learning and soft actor-critic reinforcement learning algorithm. First, this paper introduces an environmental agent for training data resampling to solve the imbalance problem of the original data. Second, rewards are redefined in reinforcement learning. In order to improve the recognition rate of few categories of network attacks, we set different reward values for different categories of attacks. The environment agent and classifier agent are trained adversarially around maximizing their respective reward values. Finally, a multi-classification experiment is conducted on the NSL-KDD and AWID datasets to compare with the existed excellent intrusion detection algorithms. AE-SAC achieves excellent classification performance with an accuracy of 84.15% and a f1-score of 83.97% on the NSL-KDD dataset, and an accuracy and a f1-score over 98.9% on the AWID dataset.

Blade sequencing optimization of aero-engine based on deep reinforcement learning

The unreasonable sorting of single-stage rotor blades leads to the over-tolerance of rotor unbalance, which is the main cause of excessive engine vibration. Aiming at the problems of long search time, poor repeatability, weak adaptability and difficulty in obtaining global optimum by using heuristic algorithm for blade sorting, this paper presents a deep reinforcement learning method is proposed to solve the blade ordering problem. A pointer network model including an encoder and a decoder structure is established. For the case where the blade data cannot be obtained, the unbalance of the single-stage rotor is used as the reward function, and the pointer network model is trained by the Actor Critic reinforcement learning algorithm. The experimental results show that the trained enhanced pointer network model can directly perform end-to-end reasoning on the input sequence, avoiding the iterative solution process of traditional heuristic algorithms, and has high solution efficiency. Using the enhanced pointer network blade sorting optimization model in this paper to sort a set of blade sequences, the unbalanced value of the rotor after sorting is 14.78 g.mm, which is 84.8% better than the genetic algorithm, and the search speed is increased by 95.9%. The results show that the method can quickly and accurately give the arrangement order of the leaves, and the proposed model has generalization. It can provide a reliable measurement method for rotor assembly measurement of large engine manufacturing enterprises such as China Aero-Engine Company.

Multi-actor mechanism for actor-critic reinforcement learning

Value estimation is a critical problem in Value-Based reinforcement learning. Most related studies focus on using multi-critic to reduce estimation bias and seldom consider the multi-actor impact on value estimation. This paper proposes a multi-actor mechanism (MAM) for Actor-Critic reinforcement learning that can provide multiple behavior choices in the same state, resulting in diverse Q-values that provide richer information and enhance exploration capability. MAM contains two technologies. One is obsolescence technology, which quickly generates high-quality experience to help the agent find the optimal policy. The other is Q-value weighting technology, which leverages multiple Q-values to achieve a more accurate value estimation. The proposed mechanism, MAM, is general and can be applied to any Actor-Critic reinforcement learning algorithm. Specifically, we embed MAM into DDPG and TD3 and demonstrate that MAM can mitigate estimation bias, enhance exploration, and yield state-of-the-art results in various MuJoCo tasks, including the challenging Humanoid-v2 and Walker2d-v2 benchmarks.

Dynamic pricing based electric vehicle charging station location strategy using reinforcement learning

To accommodate the increased demand for electric vehicle(EV) charging in China, this paper examines electric vehicle charging station(EVCS) location strategies from the perspective of private investors, taking full account of the competitive environment of existing EVCSs in the region. First of all, a three-level location model considering dynamic pricing is developed, which includes user decisions, EVCS pricing, and EVCS location decisions. Then, the soft actor-critic(SAC) reinforcement learning algorithm is used to train the optimal pricing strategy for EVCS to guarantee the maximum cumulative revenue. The proposed methodology is verified through case studies based on an industrial park in China. The results show that the proposed methodology can make more economical and scientific location decisions than the traditional method. The dynamic pricing method based on reinforcement learning can provide a reference for the location and operation of more EVCSs.

Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an epsilon-optimal solution with at most an order of epsilon to -2.5 sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.

Online PID Tuning Strategy for Hydraulic Servo Control Systems via SAC-Based Deep Reinforcement Learning

Proportional–integral–derivative (PID) control is the most common control technique used in hydraulic servo control systems. However, the nonlinearity and uncertainty of the hydraulic system make it challenging for PID control to achieve high-precision control. This paper proposes a novel control strategy that combines the soft actor-critic (SAC) reinforcement learning algorithm with the PID method to address this issue. The proposed control strategy consists of an upper-level controller based on the SAC algorithm and a lower-level controller based on the PID control method. The upper-level controller continuously tunes the control parameters of the lower-level controller based on the tracking error and system status. The lower-level controller performs real-time control for the hydraulic servo system with a control frequency 10 times higher than the upper controllers. Simulation experiments demonstrate that the proposed SAC-PID control strategy can effectively address disturbances and achieve high precision control for hydraulic servo control systems in uncertain working conditions compared with PID and fuzzy PID control methods. Therefore, the proposed control strategy offers a promising approach to improving the tracking performance of hydraulic servo systems.

Simultaneous locomotion and manipulation control of quadruped robots using reinforcement learning-based adaptive fractional-order sliding-mode control

This paper investigates a model-free reinforcement learning-based approach that enables the quadruped robot to manipulate objects while maintaining its balance and dynamic stability during walking. At first, the dynamics of quadruped robots in two sub-spaces, position control space and force control space, are developed. Then, a new long-term performance index is introduced, and a radial basis function neural network as a critic network is presented to estimate the unobtainable long-term performance index. Based on the exported reinforcement signal, the actor neural network is introduced to generate the feedforward compensation term to cope with the nonlinear dynamics and the system uncertainties. The robustness of the actor-critic reinforcement learning algorithm is enhanced by using a fractional-order sliding-mode controller in the closed-loop system. The online adaptive laws for both the critic and actor-network weights are obtained using the Lyapunov stability theory. As a result, the uniformly ultimately boundedness of the position and the force tracking errors are proven. Finally, numerical simulations are conducted to illustrate the feasibility and effectiveness of the proposed adaptive actor-critic learning-based control scheme.

A networked multi-agent reinforcement learning approach for cooperative FemtoCaching assisted wireless heterogeneous networks

To meet the explosive growth of mobile traffic requirement in the 5th generation (5 G) mobile system, FemtoCaching at the network edge has been regarded as a promising technique for the 5 G mobile system. In this paper, we focus on studying the cooperative FemtoCaching problem in wireless heterogeneous networks (HetNets), which is aimed to minimize the overall fetching delays of all users. Owing to the NP-hardness of the problem, we formulate the cooperative FemtoCaching problem as a Networked Multi-Agent Reinforcement Learning (NMARL) problem and accordingly propose a Soft Attentional Networked Multi-Agent Actor-Critic (SAN-AC) Reinforcement Learning algorithm. Simulation results demonstrate that the proposed algorithm can significantly increase the overall performance compared with existing work.

Reinforcement Learning Based Underwater Wireless Optical Communication Alignment for Autonomous Underwater Vehicles

With the developments in underwater wireless optical communication (UWOC) technology, UWOC can be used in conjunction with autonomous underwater vehicles (AUVs) for high-speed data sharing among the vehicle formation during underwater exploration. A beam alignment problem arises during communication due to the transmission range, external disturbances and noise, and uncertainties in the AUV dynamic model. In this article, we propose an acoustic navigation method to guide the alignment process without requiring beam directors, light intensity sensors, and/or scanning algorithms as used in previous research. The AUVs need stably maintain a specific relative position and orientation for establishing an optical link. We model the alignment problem as a partially observable Markov decision process (POMDP) that takes manipulation, navigation, and energy consumption of underwater vehicles into account. However, finding an efficient policy for the POMDP under high partial observability and environmental variability is challenging. Therefore, for successful policy optimization, we utilize the soft actor–critic reinforcement learning algorithm together with AUV-specific belief updates and reward shaping based curriculum learning. Our approach outperformed baseline approaches in a simulation environment and successfully performed the beam alignment process from one AUV to another on the real AUV Tri-TON 2.

Hybrid feedback and reinforcement learning-based control of machine cycle time for a multi-stage production system

With the increasing need of flexible manufacturing systems, machine flexibility has become one of the major impact factors. An important aspect of machine flexibility is the ability to change an individual machine’s capacity (or cycle time) to improve the overall system efficiency. In this paper, a novel control method is proposed for multi-stage production systems to dynamically change the individual machines’ cycle time to improve overall system efficiency. The proposed control method integrates distributed feedback control scheme and a Reinforcement Learning (RL) control scheme based on an extended actor-critic algorithm. The feedback control will determine whether a machine is turned on or off using real-time system status, while the RL control scheme will decide how to increase or decrease a machine’s cycle time when a machine is on. An improved actor-critic RL algorithm is developed to add an auxiliary model-based path to the standard model-free RL to enhance the learning performance. To demonstrate the effectiveness of the proposed method, numerical case studies have been performed that clearly show improvements in the overall profits and energy savings compared to other methods.

Adaptive Partial Offloading and Resource Harmonization in Wireless Edge Computing-Assisted IoE Networks

Wireless backhaul is considered a portable and cost-effective solution for deploying small cell-assisted communications in mobile Internet of Everything (IoE) networks. In this system, the wireless backhaul between a central gNodeB (gNB) and small cell base stations (SBSs) shares the frequency spectrum with the wireless fronthaul between the SBSs and user devices. In these circumstances, efficient edge computing deployment requires joint optimization of wireless resource harmonization and partial offloading scheduling to accommodate heterogeneous IoE services. Therefore, in this study, we formulate the system model as a Markov decision process (MDP) to minimize the weighted sum of computation overheads in terms of the latency and energy costs of all user equipment (UE) where network dynamics are considered in the system state. Consequently, an actor–critic reinforcement learning algorithm, i.e., a deep deterministic policy gradient (DDPG)-based algorithm with replay memory technique, is proposed to realize an adaptive offloading decision scheme and optimal wireless resource harmonization between the backhaul and fronthaul. Extensive simulation results reveal that the proposed algorithm reliably converges and provides approximately 70%, 55%, 36%, and 11% lower total computation overhead than UE execution, random, MEC execution, and DQN-based schemes, respectively.