Actor Critic Research Articles

The Shared Experience Actor–Critic (SEAC) Approach for Allocating Radio Resources and Mitigating Resource Collisions in 5G-NR-V2X Mode 2 Under Aperiodic Traffic Conditions

5G New Radio (NR)-V2X, standardized by 3GPP Release 16, includes a distributed resource allocation Mode, known as Mode 2, that allows vehicles to autonomously select transmission resources using either sensing-based semi-persistent scheduling (SB-SPS) or dynamic scheduling (DS). In unmanaged 5G-NR-V2X scenarios, SB-SPS loses effectiveness with aperiodic and variable data. DS, while better for aperiodic traffic, faces challenges due to random selection, particularly in high traffic density scenarios, leading to increased collisions. To address these limitations, this study models the Cellular V2X network as a decentralized multi-agent networked Markov decision process (MDP), where each vehicle agent uses the Shared Experience Actor–Critic (SEAC) technique to optimize performance. The superiority of SEAC over SB-SPS and DS is demonstrated through simulations, showing that the SEAC with an N-step approach achieves an average improvement of approximately 18–20% in enhancing reliability, reducing collisions, and improving resource utilization under high vehicular density scenarios with aperiodic traffic patterns.

Priority-based intelligent resolution method of multi-aircraft flight conflicts

Abstract The rising demand for air traffic will inevitably result in a surge in both the number and complexity of flight conflicts, necessitating intelligent strategies for conflict resolution. This study addresses the critical challenges of scalability and real-time performance in multi-aircraft flight conflict resolution by proposing a comprehensive method that integrates a priority ranking mechanism with a conflict resolution model based on the Markov decision process (MDP). Within this framework, the proximity between aircraft in a multi-aircraft conflict set is dynamically assessed to establish a conflict resolution ranking mechanism. The problem of multi-aircraft conflict resolution is formalised through the MDP, encompassing the design of state space, discrete action space and reward function, with the transition function implemented via simulation prediction using model-free methods. To address the positional uncertainty of aircraft in real-time scenarios, the conflict detection mechanism introduces the aircraft’s positional error. A deep reinforcement learning (DRL) environment is constructed incorporating actual airspace structures and traffic densities, leveraging the Actor Critic using Kronecker-factored Trust Region (ACKTR) algorithm to determine resolution actions. The experimental results indicate that with 20–30 aircraft in the airspace, the success rate can reach 94% for the training set and 85% for the test set. Furthermore, this study analyses the impact of varying aircraft numbers on the success rate within a specific airspace scenario. The outcomes of this research provide valuable insights for the automation of flight conflict resolution.

A path planning method based on deep reinforcement learning for AUV in complex marine environment

The potential of autonomous underwater vehicle (AUV) on future applications is significant due to advancements in autonomy and intelligence. Path planning is a critical technology for AUV to perform operational missions in complex marine environments. To this end, this paper proposes a path planning method for AUV based on deep reinforcement learning. Initially, considering actual requirements, a complex marine environment model containing underwater terrain, sonobuoy detection, and ocean currents is established. Subsequently, the corresponding state space, action space, and reward function are formulated. Furthermore, to address the inherent limitations of existing deep reinforcement learning algorithms in terms of training efficiency, a mixed experience replay (MER) strategy is proposed. This strategy aims to enhance the efficiency of sample learning by integrating prior knowledge and exploration experience. Lastly, a novel HMER-SAC algorithm for AUV path planning is proposed by integrating the Soft Actor–Critic (SAC) algorithm with the hierarchical reinforcement learning strategy and the MER strategy. The results of the simulation and experiment demonstrate that the method is capable of efficiently planning executable paths in complex marine environments and exhibits superior training efficiency, stability, and performance.

Deep Reinforcement Learning in Cryptocurrency Trading: A Profitable Approach

This study proposes an Automatic Cryptocurrency Trading System using Deep Reinforcement Learning (DRL). Six popular cryptocurrencies were used: Bitcoin, Ethereum, BinanceCoin, DogeCoin, Cardano, and WAVES. Development of the trading system started with building three timeseries models – Temporal Convolutional Neural Network (TCNN), Long Short-Term Memory Network (LSTM), and Gated Recurrent Unit Network (GRU) – to predict future prices. Then, cryptocurrency sentiment data was scraped using the Alternative.me API. Data on historical prices, predicted future prices, cryptocurrency sentiment index, technical indicators, and trading account information was fed as input states to three DRL Agents — Deep Q Network (DQN), Advantage Actor Critic (A2C), and Recurrent Proximal Policy Optimization (RPPO) — which were trained using a custom-developed trading environment. Each agent was given $1000 initial capital for all six cryptocurrencies to trade using three possible actions — Buy, Sell and Hold — and were back-tested on one year of unseen data. Our DQN model had the highest overall return on investment (ROI) of $740, an average 12.3% ROI across all six cryptocurrencies, with an ROI of 63.98% achieved for BinanceCoin. However, A2C and RPPO both had negative ROI.

Multi-Objective Combinatorial Optimization Algorithm Based on Asynchronous Advantage Actor–Critic and Graph Transformer Networks

Multi-objective combinatorial optimization problems (MOCOPs) are designed to identify solution sets that optimally balance multiple competing objectives. Addressing the challenges inherent in applying deep reinforcement learning (DRL) to solve MOCOPs, such as model non-convergence, lengthy training periods, and insufficient diversity of solutions, this study introduces a novel multi-objective combinatorial optimization algorithm based on DRL. The proposed algorithm employs a uniform weight decomposition method to simplify complex multi-objective scenarios into single-objective problems and uses asynchronous advantage actor–critic (A3C) instead of conventional REINFORCE methods for model training. This approach effectively reduces variance and prevents the entrapment in local optima. Furthermore, the algorithm incorporates an architecture based on graph transformer networks (GTNs), which extends to edge feature representations, thus accurately capturing the topological features of graph structures and the latent inter-node relationships. By integrating a weight vector layer at the encoding stage, the algorithm can flexibly manage issues involving arbitrary weights. Experimental evaluations on the bi-objective traveling salesman problem demonstrate that this algorithm significantly outperforms recent similar efforts in terms of training efficiency and solution diversity.

Assessing generalizability of Deep Reinforcement Learning algorithms for Automated Vulnerability Assessment and Penetration Testing

Modern cybersecurity best practices and standards require continuous Vulnerability Assessment (VA) and Penetration Test (PT). These activities are human- and time-expensive. The research community is trying to propose autonomous or semi-autonomous solutions based on Deep Reinforcement Learning (DRL) agents, but current proposals require further investigations. We observe that related literature reports performance tests of the proposed agents against a limited subset of the hosts used to train the models, thus raising questions on their applicability in realistic scenarios. The main contribution of this paper is to fill this gap by investigating the generalization capabilities of existing DRL agents to extend their VAPT operations to hosts that were not used in the training phase. To this purpose, we define a novel VAPT environment through which we devise multiple evaluation scenarios. While evidencing the limited capabilities of shallow RL approaches, we consider three state-of-the-art deep RL agents, namely Deep Q-Network (DQN), Proximal Policy Optimization (PPO), and Advantage Actor–Critic (A2C), and use them as bases for VAPT operations. The results show that the algorithm using A2C DRL agent outperforms the others because it is more adaptable to unknown hosts and converges faster. Our methodology can guide future researchers and practitioners in designing a new generation of semi-autonomous VAPT tools that are suitable for real-world contexts.

Reinforcement learning for heliostat aiming: Improving the performance of Solar Tower plants

Solar Tower (ST) systems use heliostats to concentrate solar radiation onto a tower-mounted receiver. Optimizing the aiming strategy for these heliostats over the receiver remains a critical challenge due to the dynamic nature of solar radiation and the need to maximize energy capture while ensuring operational safety. This paper introduces a novel, model-free deep Reinforcement Learning (RL) approach to optimize heliostat aiming strategies, utilizing the Soft Actor–Critic (SAC) algorithm. This advanced RL method enhances the traditional Actor–Critic framework with two neural networks. The proposal dynamically adjusts the aiming points across the receiver surface in real time, trying to improve the overall performance of the ST plant. The strategy was simulated and evaluated over a full operational year and compared with traditional methods. The results show an increase of more than 8.8% in yearly absorbed power, a significant improvement that directly enhances performance and contributes to better economic outcomes for the technology. This technique also eliminates the need for constant human intervention and is applicable to both existing and future plants.

Knowledge-Assisted Actor Critic Proximal Policy Optimization-Based Service Function Chain Reconfiguration Algorithm for 6G IoT Scenario.

Future 6G networks will inherit and develop Network Function Virtualization (NFV) architecture. With the NFV-enabled network architecture, it becomes possible to establish different virtual networks within the same infrastructure, create different Virtual Network Functions (VNFs) in different virtual networks, and form Service Function Chains (SFCs) that meet different service requirements through the orderly combination of VNFs. These SFCs can be deployed to physical entities as needed to provide network functions that support different services. To meet the highly dynamic service requirements in the future 6G Internet of Things (IoT) scenario, the highly flexible and efficient SFC reconfiguration algorithm is the key research direction. Deep-learning-based algorithms have shown their advantages in solving this type of dynamic optimization problem. Considering that the efficiency of the traditional Actor Critic (AC) algorithm is limited, the policy does not directly participate in the value function update. In this paper, we use the Proximal Policy Optimization (PPO) clip function to restrict the difference between the new policy and the old policy, to ensure the stability of the updating process. We combine PPO with AC, and further bring the historical decision information as the network knowledge to offer better initial policies, to accelerate the training speed. We also propose the Knowledge = Assisted Actor Critic Proximal Policy Optimization (KA-ACPPO)-based SFC reconfiguration algorithm to ensure the Quality of Service (QoS) of end-to-end services. Simulation results show that the proposed KA-ACPPO algorithm can effectively reduce computing cost and power consumption.

Adversarial attacks on reinforcement learning agents for command and control

Given the recent impact of deep reinforcement learning in training agents to win complex games such as StarCraft and DoTA (Defense Of The Ancients)—there has been a surge in research for exploiting learning-based techniques for professional wargaming, battlefield simulation, and modeling. Real-time strategy games and simulators have become a valuable resource for operational planning and military research. However, recent work has shown that such learning-based approaches are highly susceptible to adversarial perturbations. In this paper, we investigate the robustness of an agent trained for a command and control task in an environment that is controlled by an active adversary. The C2 agent is trained on custom StarCraft II maps using the state-of-the-art RL algorithms—Asynchronous Advantage Actor Critic (A3C) and proximal policy optimization (PPO). We empirically show that an agent trained using these algorithms is highly susceptible to noise injected by the adversary and investigate the effects these perturbations have on the performance of the trained agent. Our work highlights the urgent need to develop more robust training algorithms especially for critical arenas like the battlefield.

A novel Soft Actor–Critic framework with disjunctive graph embedding and autoencoder mechanism for Job Shop Scheduling Problems

The Job-Shop Scheduling Problem (JSSP) is a well-established and classic NP-hard combinatorial optimization issue. The quality of its scheduling scheme directly affects the operational efficiency of manufacturing systems. Priority Dispatching Rules (PDRs) are often utilized to address JSSP in real-world contexts, but the process of creating effective PDRs can be daunting and time-consuming. It also necessitates comprehensive domain knowledge, typically resulting in mediocre performance. In this paper, we introduce a novel reinforcement learning (RL) model called Disjunctive Graph Embedding with Autoencoder Mechanism for Job Shop Scheduling Problems (DGEAM-JSSP), designed to automate PDRs learning. Our proposed model confronts the issue using a Graph Neural Network (GNN) to learn node features that encapsulate the spatial structure of the JSSP graph representation. The ensuing policy network is size-agnostic, enabling effective generalization on larger-scale instances. Additionally, we employ a transformer encoder, incorporating parallel encoding and a self-attention mechanism, to successfully recognize long-term dependencies among operations in large-scale scheduling problems. We also implemented an end-to-end training approach using the Soft Actor–Critic (SAC) algorithm to instruct the two modules. Computational experiment results reveal that, with a single training, our agent successfully learns a superior dispatching policy, surpassing PDRs and state-of-the-art RL frameworks specifically tailored for each JSSP instance size in solution quality, as well as OR-Tools in execution speed. Moreover, results from random and benchmark instances illustrate that the uniquely-modeled learned policies have impressive generalization performance on real-world instances and significantly larger-scale scenarios involving up to 2000 operations.

Actor-Critic With Synthesis Loss for Solving Approximation Biases.

Approximation biases of value functions are considered a key problem in reinforcement learning (RL). In particular, existing RL algorithms are hindered by overestimation and underestimation biases, i.e., value mismatching between RL's actual returns and action-value approximations limits the performance of RL algorithms. In this article, we first develop a new synthesis loss function for RL's action-value estimation integrating a regularization term and a modified "clipped double Q-learning" structure for solving overestimation and underestimation biases. To minimize the differences between action-value estimations and actual returns in RL, we develop a new discrepancy function to determine the type and magnitude of approximation biases. Then, two coefficients embedded in the synthesis loss are automatically tuned by minimizing the discrepancy function during training to minimize approximation biases. We further design a new actor-critic (AC) algorithm, named AC with synthesis loss (ACSL), by integrating the synthesis loss function and an error-controlled mechanism. Experimental results on continuous control tasks illustrate that the proposed ACSL algorithm outperforms other cutting-edge RL methods in many tasks and that the proposed synthesis loss function is easily implemented into other algorithms and significantly reduces approximation biases while improving performance. The proposed method can successfully handle many complex continuous control tasks and can greatly outperform other state-of-the-art algorithms on most tasks.

Real-Time Optimization of Urban Rail Transit Train Scheduling via Advantage Actor–Critic Deep Reinforcement Learning

Real-Time Optimization of Urban Rail Transit Train Scheduling via Advantage Actor–Critic Deep Reinforcement Learning

ADT2R: Adaptive Decision Transformer for Dynamic Treatment Regimes in Sepsis.

Dynamic treatment regimes (DTRs), which comprise a series of decisions taken to select adequate treatments, have attracted considerable attention in the clinical domain, especially from sepsis researchers. Existing sepsis DTR learning studies are mainly based on offline reinforcement learning (RL) approaches working on electronic healthcare records data. However, a trained policy may choose a treatment different from a human clinician's prescription. Furthermore, most of them do not consider: 1) heterogeneity in sepsis; 2) short-term transitions; and 3) the relationship between a patient's health state and the prescription. We propose a novel framework, an adaptive decision transformer for DTR (ADT 2 R), which recommends an optimal treatment action for each time step depending on the heterogeneity of the sepsis and a patient's evolving health states. Specifically, we devise a trajectory-optimization-based module to be trained with supervision for treatments and adaptively aggregate the multihead self-attentions by deliberating on various inherent time-varying patterns among sepsis patients. Furthermore, we estimate the patient's health state by adopting an actor-critic (AC) algorithm and inform the treatment recommendation by learning about its short-term changes. We validated the effectiveness of the proposed framework on the Medical Information Mart for Intensive Care III (MIMIC-III) dataset, an extensive intensive care database, by demonstrating performance comparable to the state-of-the-art methods.

Digital twins-boosted intelligent maintenance of ageing bridge hangers exposed to coupled corrosion–fatigue deterioration

The corrosion–fatigue coupled deterioration of ageing steel bridge hangers presents significant structural challenges, demanding rigorous condition assessments and timely maintenance interventions. This paper introduces a framework of digital twins-boosted intelligent maintenance (DTIM) tailored for ageing hangers, integrating the prediction model, monitoring data and inspection results. The DTIM features a suite of algorithms adaptation and innovations, including dynamic Partially Observable Markov Decision Processes (POMDP), Asynchronous Advantage Actor Critic (A3C), and Bayesian Dynamic Linear Models (BDLM). The DTIM emphasises regular early-life repairs, strategic inspections, and timely replacements towards life-end, tailored to the condition of specific bridge hangers in the case study presented. By coordinating actions across hangers, the DTIM enables opportunistic maintenance to further optimise resource allocation. The output highlights digital twins in exploring the add-on value of monitoring and inspection for the proactive and sustainable maintenance of ageing infrastructure, promising enhanced structural integrity and serviceability in a cost-effective and well-informed manner.

UAV-Based Emergency Communications: An Iterative Two-Stage Multiagent Soft Actor–Critic Approach for Optimal Association and Dynamic Deployment

UAV-Based Emergency Communications: An Iterative Two-Stage Multiagent Soft Actor–Critic Approach for Optimal Association and Dynamic Deployment

Actor Critic Based Reinforcement Learning for Joint Resource Allocation and Throughput Maximization in 5G RAN Slicing

Actor Critic Based Reinforcement Learning for Joint Resource Allocation and Throughput Maximization in 5G RAN Slicing

SEGAC: Sample Efficient Generalized Actor Critic for the Stochastic On-Time Arrival Problem

SEGAC: Sample Efficient Generalized Actor Critic for the Stochastic On-Time Arrival Problem

Observer‐based adaptive neural networks optimal control for spacecraft proximity maneuver with state constraints

AbstractThis article proposes an adaptive neural network (NN) optimal control approach for autonomous relative motion control of non‐cooperative spacecraft in proximity. The proposed method aims to minimize fuel consumption under various challenges including model uncertainty, state constraints, external disturbances, and input saturation. To account for uncertain parameters of non‐cooperative target and external disturbances, we start by designing a NN disturbance observer. Subsequently, a novel optimal control index function is presented. An adaptive NN based on the actor‐critic (A‐C) framework and backstepping theory is then utilized to approximate the solution of Hamilton–Jacobi–Bellman (HJB) equation and obtain an optimal control law. The Lyapunov framework is leveraged to establish the stability of the closed‐loop control system. Finally, numerical simulations are conducted to assess the feasibility and effectiveness of the proposed control scheme in comparison with an existing approach.

Reinforcement learning for battery energy management: A new balancing approach for Li-ion battery packs

This study investigates the challenge of cell balancing in battery management systems (BMS) for lithium-ion batteries. Effective cell balancing is crucial for maximizing the usable capacity and lifespan of battery packs, which is essential for the widespread adoption of electric vehicles and the reduction of greenhouse gas emissions. A novel deep reinforcement learning (deep RL) approach is proposed for passive balancing with switched shunt resistors. Notable deep RL algorithms capable of handling discrete actions, such as Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Q-Network (DQN), Augmented Random Search (ARS), and Asynchronous Advantage Actor Critic (A3C), are investigated. TRPO demonstrates superior performance compared to other deep RL algorithms and rule-based methods in both charging and discharging scenarios without requiring fine-tuning, optimizing the balance between cell balancing and switch changes. It achieves up to 16.8% improvement in battery pack capacity, 69.4% reduction in state-of-charge variance among cells, and 40.4% decrease in the number of switching operations in simulation results for five li-ion cells connected in series.The study introduces an innovative application of deep RL for passive balancing, a comprehensive battery cell modeling technique, and a tailored multi-objective reward function that balances cell balancing and switching costs. This work represents a significant advancement in applying deep RL to battery management systems, providing a framework for further research and practical implementation in energy storage systems.

Improving Exploration in Actor-Critic With Weakly Pessimistic Value Estimation and Optimistic Policy Optimization.

Deep off-policy actor-critic algorithms have been successfully applied to challenging tasks in continuous control. However, these methods typically suffer from the poor sample efficiency problem, limiting their widespread adoption in real-world domains. To mitigate this issue, we propose a novel actor-critic algorithm with weakly pessimistic value estimation and optimistic policy optimization (WPVOP) for continuous control. WPVOP integrates two key ingredients: 1) a weakly pessimistic value estimation, which compensates the pessimism of lower confidence bound in conventional value function (i.e., clipped double Q -learning) to trigger exploration in low-value state-action regions and 2) an optimistic policy optimization algorithm by sampling actions that could benefit the policy learning most toward optimal Q -values for efficient exploration. We theoretically analyze that the proposed weakly pessimistic value estimation method is lower and upper bounded, and empirically show that it could avoid extremely over-optimistic value estimates. We show that these two ideas are largely complementary, and can be fruitfully integrated to improve performance and promote sample efficiency of exploration. We evaluate WPVOP on the suite of continuous control tasks from MuJoCo, achieving state-of-the-art sample efficiency and performance.