Traditional Reinforcement Learning Algorithms Research Articles

Evaluating the stealth of reinforcement learning-based cyber attacks against unknown scenarios using knowledge transfer techniques

Organizations are vulnerable to cyber attacks as they rely on computer networks and the internet for communication and data storage. While Reinforcement Learning (RL) is a widely used strategy to simulate and learn from these attacks, RL-guided offensives against unknown scenarios often lead to early exposure due to low stealth resulting from mistakes during the training phase. To address this issue, this work evaluates if the use of Knowledge Transfer Techniques (KTT), such as Transfer Learning and Imitation Learning, reduces the probability of early exposure by smoothing mistakes during training. This study developed a laboratory platform and a method to compare RL-based cyber attacks using KTT for unknown scenarios. The experiments simulated 2 unknown scenarios using 4 traditional RL algorithms and 4 KTT. In the results, although some algorithms using KTT obtained superior results, they were not so significant for stealth during the initial epochs of training. Nevertheless, experiments also revealed that throughout the entire learning cycle, Trust Region Policy Optimization (TRPO) is a promising algorithm for conducting cyber offensives based on Reinforcement Learning.

Advancing spacecraft rendezvous and docking through safety reinforcement learning and ubiquitous learning principles

As spacecraft rendezvous and docking missions become increasingly complex, the need for advanced solutions has surged. In recent years, the application of reinforcement learning techniques to tackle spacecraft rendezvous guidance challenges has emerged as a prominent international trend. Vital to ensuring the secure rendezvous and docking of spacecraft is the task of obstacle avoidance. However, traditional reinforcement learning algorithms lack the ability to enforce safety constraints within the exploration space, which presents a formidable obstacle in the design of spacecraft rendezvous guidance strategies. In response to this challenge, a spacecraft rendezvous guidance methodology founded on safety reinforcement learning is proposed. Firstly, a Markov model is crafted for autonomous spacecraft rendezvous in scenarios involving collision avoidance. A reward system, contingent on obstacle warnings and collision avoidance constraints, is introduced to establish a safety reinforcement learning framework for devising spacecraft rendezvous guidance strategies. Secondly, within the framework of safety reinforcement learning, two deep reinforcement learning (DRL) algorithms, Proximal Policy Optimisation (PPO) and Deep Deterministic Policy Gradient (DDPG), are leveraged to generate these guidance strategies. Experimental findings validate the effectiveness of this approach in successfully executing obstacle avoidance and achieving rendezvous with remarkable precision. Furthermore, through an analysis of the performance and generalization capabilities of these two algorithms, the efficacy of the proposed methodology is further underscored. This fusion of advanced space guidance technology with the principles of Ubiquitous Learning marks a significant step forward in the quest for safer and more efficient spacecraft rendezvous and docking operations.

Learning strategies for underwater robot autonomous manipulation control

Autonomous manipulation operations represent the high intelligent coordination from robotic vision and control, it is also a symbol of the advances of robotic intelligence. The limitations of visual sensing and the increasingly complex experimental conditions make autonomous manipulation operations more difficult, particularly for deep reinforcement learning methods, which can enhance robotic control intelligence but require a lot of training process. Due to the high-dimensional continuous state space and continuous action space characteristics of underwater operations, this paper adopts a policy-based reinforcement learning method as the foundational approach. To address the issues of instability and low convergence efficiency in traditional policy-based reinforcement learning algorithms during the learning process, this paper proposes a novel policy learning method. This method adopts the Proximal Policy Optimization algorithm (PPOClip) and optimizes it through an actor-critic network. The aim is to improve the stability and effectiveness of convergence in the learning process. In the underwater training environment, a new reward shaping scheme has been designed to address the issue of reward sparsity during the training process. The manually crafted dense reward function is utilized as attractive and repulsive potential functions for goal manipulation and obstacle avoidance. On the highly complex underwater manipulation and training environment, transferred learning algorithm has been established to reduce the training times and compensate the differences between the simulation and experiment. Simulations and tank experiments have verified the performance of the proposed strategy learning method.

Optimization of Image Transmission in Cooperative Semantic Communication Networks

In this paper, a semantic communication framework for image data transmission is developed. In the investigated framework, a set of servers cooperatively transmit image data to a set of users utilizing semantic communication techniques, which enable servers to transmit only the semantic information that accurately captures the meaning of images. To evaluate the performance of studied semantic communication system, a multimodal metric called image-to-graph semantic similarity (ISS) is proposed to measure the correlation between the extracted semantic information and the original image. To meet the ISS requirement of each user, each server must jointly determine the semantic information to be transmitted and the resource blocks (RBs) used for semantic information transmission. Due to the co-channel interference among users associated with different servers, each server must cooperate with other servers to find a globally optimal semantic oriented RB allocation. We formulate this problem as an optimization problem whose goal is to minimize the sum of the average transmission latency of each server while reaching the ISS requirement. To solve this problem, we propose a value decomposition based entropy-maximized multi-agent reinforcement learning (RL) algorithm. The proposed algorithm enables each server to coordinate with other servers in training stage and execute RB allocation in a distributed manner to approach to a globally optimal performance with less training iterations. Compared to traditional multi-agent RL algorithms, the proposed RL framework improves the exploration of valuable action of servers and the probability of finding a globally optimal RB allocation policy based on local observation of wireless and semantic communication environments. Simulation results show that the proposed algorithm can reduce the transmission delay by up to 16.1% and improve the convergence speed by up to 100% compared to the traditional multi-agent RL algorithms.

Electric vehicle charging navigation strategy in coupled smart grid and transportation network: A hierarchical reinforcement learning approach

Most of the existing electric vehicle (EV) charging navigation methods do not simultaneously take into account the electric vehicle charging destination optimization and path planning. Moreover, they are unable to provide online real-time decision-making under a variety of uncertain factors. To address these problems, this paper first establishes a bilevel stochastic optimization model for EV charging navigation considering various uncertainties, and then proposes an EV charging navigation method based on the hierarchical enhanced deep Q network (HEDQN) to solve the above stochastic optimization model in real-time. The proposed HEDQN contains two enhanced deep Q networks, which are utilized to optimize the charging destination and charging route path of EVs, respectively. Finally, the proposed method is simulated and validated in two urban transportation networks. The simulation results demonstrate that compared with the Dijkstra shortest path algorithm, single-layer deep reinforcement learning algorithm, and traditional hierarchical deep reinforcement learning algorithm, the proposed HEDQN algorithm can effectively reduce the total charging cost of electric vehicles and realize online real-time charging navigation of electric vehicles, that shows excellent generalization ability and scalability.

Successful application of predictive information in deep reinforcement learning control: A case study based on an office building HVAC system

Reinforcement Learning (RL), a promising algorithm for the operational control of Heating, Ventilation, and Air Conditioning (HVAC) systems, has garnered considerable attention and applications. However, traditional RL algorithms typically do not incorporate predictive information for future scenarios, and only a limited number of studies have examined the enhancement and impact of predictive information on RL algorithms. To address the issue of coupling RL and predictive information in HVAC system operation optimization, we employed an open-source framework to examine the impact of various predictive information strategies on RL outcomes. We propose a joint gated recurrent unit (GRU)-RL algorithm to handle situations where a time-series exists in state space. The results from four classic test cases demonstrate that the proposed GRU-RL method can reduce operating costs by approximately 14.5% and increase comfort performance by 88.4% in indoor comfort control and cost-management tasks. Moreover, the GRU-RL method outperformed the conventional DRL method and was merely augmented with prediction information. In indoor temperature regulation, the GRU-RL algorithm improves control efficacy by 14.2% compared to models without predictive information and offers an approximately 5% improvement over traditional network models. Finally, all models were made open source for easy replication and further research.

The Application of Residual Connection-Based State Normalization Method in GAIL

In the domain of reinforcement learning (RL), deriving efficacious state representations and maintaining algorithmic stability are crucial for optimal agent performance. However, the inherent dynamism of state representations often complicates the normalization procedure. To overcome these challenges, we present an innovative RL framework that integrates state normalization techniques with residual connections and incorporates attention mechanisms into generative adversarial imitation learning (GAIL). This combination not only enhances the expressive capability of state representations, thereby improving the agent’s accuracy in state recognition, but also significantly mitigates the common issues of gradient dissipation and explosion. Compared to traditional RL algorithms, GAIL combined with the residual connection-based state normalization method empowers the agent to markedly reduce the exploration duration such that feedback concerning rewards in the current state can be provided in real time. Empirical evaluations demonstrate the superior efficacy of this methodology across various RL environments.

A reinforcement learning-based routing algorithm for large street networks

Evacuation planning and emergency routing systems are crucial in saving lives during disasters. Traditional emergency routing systems, despite their best efforts, often struggle to accurately capture the dynamic nature of flood conditions, road closures, and other real-time changes inherent in urban disaster logistics. This paper introduces the ReinforceRouting model, a novel approach to optimizing evacuation routes using reinforcement learning (RL). The model incorporates a unique RL environment that considers multiple criteria, such as traffic conditions, hazardous situations, and the availability of safe routes. The RL agent in this model learns optimal actions through interaction with the environment, receiving feedback in the form of rewards or penalties. The ReinforceRouting model excels in executing prompt and accurate route planning on large road networks, outperforming traditional RL algorithms and shortest-path-based algorithms. A higher safety score and episode reward of the model are demonstrated when compared to these classical methods. This innovative approach to disaster evacuation planning offers a promising avenue for enhancing the efficiency, safety, and reliability of emergency responses in dynamic urban environments.

Prioritized experience replay based deep distributional reinforcement learning for battery operation in microgrids

Reinforcement Learning (RL) provides a pathway for efficiently utilizing the battery storage in a microgrid. However, traditional value-based RL algorithms used in battery management focus on formulating the policies based on the reward expectation rather than its probability distribution. Hence the scheduling strategy is solely based on the expectation of the rewards rather than the distribution. This paper focuses on scheduling strategy based on probability distribution of the rewards which optimally reflects the uncertainties in the incoming dataset. Furthermore, the prioritized experience replay samples of the training experience are used to enhance the quality of the learning by reducing bias. The results are obtained with different variants of distributional RL algorithms like C51, Quantile Regression Deep Q-Network (QR-DQN), Fully Quantizable Function (FQF), Implicit Quantile Networks (IQN) and rainbow. Moreover, the results are compared with the traditional deep Q-learning algorithm with prioritized experienced replay. The convergence results on the training dataset are further analyzed by varying the action spaces, using randomized experience replay and without including the tariff-based action while enforcing the penalties for violating battery SoC limits. The best trained Q-network is tested with different load and PV profiles to obtain the battery operation and costs. The performance of the distributional RL algorithms is analyzed under different schemes of Time of Use (ToU) tariff. QR-DQN with prioritized experience replay has been found to be the best performing algorithm in terms of convergence on the training dataset, with least fluctuation in validation dataset and battery operations during different tariff regimes during the day.

Doubly constrained offline reinforcement learning for learning path recommendation

Learning path recommendation refers to the application of interactive recommendation systems in the field of education, aimed at optimizing learning outcomes while minimizing the workload of learners, teachers, and curriculum designers. Reinforcement Learning (RL) has proven effective in capturing and modeling the complex interactions among course activities, learner behaviors, and educational outcomes. Therefore, combining the two approaches presents endless possibilities for personalized education through the use of interactive recommendation systems in the education domain. However, traditional RL algorithms require extensive interaction with the environment during the training phase. Using unverified recommendation logic in interactions with actual students can give rise to unmanageable problems and hinder effective performance in an educational setting. This is because extrapolation introduces substantial evaluation errors that result in recommendations deviating significantly from the actual educational requirements. To address this limitation, we propose a novel method of offline reinforcement learning called Doubly Constrained deep Q-learning Network (DCQN). This method utilizes two generative models to fit existing student historical interaction data, which in turn, constrains the original policy network to generate new actions based on past interactions, avoiding the occurrence of overestimated actions and reducing extrapolation errors. Empirical results on demonstrate that this approach performs better than existing techniques across D4RL, i.e., datasets for deep data-driven reinforcement learning and real educational datasets.

Mean Field Multi-Agent Reinforcement Learning Method for Area Traffic Signal Control

Reinforcement learning is an effective method for adaptive traffic signal control in urban transportation networks. As the number of training rounds increases, the optimal control strategy is learned, and the learning capabilities of deep neural networks are further enhanced, thereby avoiding the limitations of traditional signal control methods. However, when faced with the sequential decision tasks of regional signal control, it encounters issues such as the curse of dimensionality and environmental non-stationarity. To address the limitations of traditional reinforcement learning algorithms applied to multiple intersections, the mean field theory is applied. This models the traffic signal control problem at multiple intersections within a region as interactions between individual intersections and the average effects of neighboring intersections. By decomposing the Q-function through bilateral estimation between the agent and its neighbors, this method reduces the complexity of interactions between agents while preserving global interactions between the agents. A traffic signal control model based on Mean Field Multi-Agent Reinforcement Learning (MFMARL) was constructed, containing two algorithms: Mean Field Q-Network Area Traffic Signal Control (MFQ-ATSC) and Mean Field Actor-Critic Network Area Traffic Signal Control (MFAC-ATSC). The model was validated using the SUMO simulation platform. The experimental results indicate that across different metrics, such as average speed, the mean field reinforcement learning method outperforms classical signal control methods and several existing approaches.

Safety Constraint-Guided Reinforcement Learning with Linear Temporal Logic

In the context of reinforcement learning (RL), ensuring both safety and performance is crucial, especially in real-world scenarios where mistakes can lead to severe consequences. This study aims to address this challenge by integrating temporal logic constraints into RL algorithms, thereby providing a formal mechanism for safety verification. We employ a combination of theoretical and empirical methods, including the use of temporal logic for formal verification and extensive simulations to validate our approach. Our results demonstrate that the proposed method not only maintains high levels of safety but also achieves comparable performance to traditional RL algorithms. Importantly, our approach fills a critical gap in the existing literature by offering a solution that is both mathematically rigorous and empirically validated. The study concludes that the integration of temporal logic into RL offers a promising avenue for developing algorithms that are both safe and efficient. This work lays the foundation for future research aimed at generalizing this approach to various complex systems and applications.

Optimization of Robotic Arm Grasping through Fractional-Order Deep Deterministic Policy Gradient Algorithm

With the rapid development of robotics technology, robotic arm grasping has gained significant attention in the fields of automation and artificial intelligence. In this study, we propose a fractional-order deep deterministic policy gradient (DDPG) algorithm for optimizing robotic arm grasping tasks. Traditional machine learning algorithms face challenges in handling continuous action spaces, while the DDPG algorithm effectively addresses this issue. In this research, we first review the background and challenges of robotic arm grasping and provide an overview of the application of traditional reinforcement learning algorithms in grasping tasks. Subsequently, we introduce the principles and fundamental ideas of the DDPG algorithm in detail, discussing its potential for optimizing robotic arm grasping. To further enhance the performance of robotic arm grasping, we propose an improved approach based on fractional-order control. Fractional-order control exhibits unique advantages in environmental dynamics modeling and grasp posture optimization, enhancing the robustness and adaptability of robotic arm grasping. Through a series of experiments, we validate the effectiveness and superiority of the fractional-order DDPG algorithm in robotic arm grasping tasks. Our algorithm achieves significant improvements in grasping success rate and stability compared to traditional methods. The experimental results demonstrate that the fractional-order DDPG algorithm is better equipped to handle control challenges in continuous action spaces and optimize the performance of robotic arm grasping tasks.

Data-Driven Load Frequency Control Based on Multi-Agent Reinforcement Learning With Attention Mechanism

With the massive penetration of renewable energy, traditional reinforcement learning algorithms suffer from slow convergence and area control error (ACE) in interconnected power systems. This paper proposes data-driven load frequency control (LFC) based on multi-agent reinforcement learning with attention mechanism in interconnected power systems. It can be divided into two phases; in the centralized training, the agents are trained by an experience replay mechanism; in the decentralized execution, the trained agent automatically regulates the generation power to control the load frequency by real-time access to the grid data in the area. The agent can selectively focus on specific information in the environment by introducing a criticism network with an attention mechanism. The attention mechanism can reduce the training time for reinforcement learning while improving control performance under disturbance. A novel reward function based on a cooperation mechanism is used to score the performance of agent, which can guide the reinforcement learning algorithm to reduce the ACE of each area simultaneously. The proposed method is validated by the IEEE three-area interconnected power system, and it is concluded that the method can reduce the ACE caused by load and renewable power disturbances, and greatly reduce the training time of the algorithm.

A Hybrid Online Off-Policy Reinforcement Learning Agent Framework Supported by Transformers.

Reinforcement learning (RL) is a powerful technique that allows agents to learn optimal decision-making policies through interactions with an environment. However, traditional RL algorithms suffer from several limitations such as the need for large amounts of data and long-term credit assignment, i.e. the problem of determining which actions actually produce a certain reward. Recently, Transformers have shown their capacity to address these constraints in this area of learning in an offline setting. This paper proposes a framework that uses Transformers to enhance the training of online off-policy RL agents and address the challenges described above through self-attention. The proposal introduces a hybrid agent with a mixed policy that combines an online off-policy agent with an offline Transformer agent using the Decision Transformer architecture. By sequentially exchanging the experience replay buffer between the agents, the agent's learning training efficiency is improved in the first iterations and so is the training of Transformer-based RL agents in situations with limited data availability or unknown environments.

Toward personalized decision making for autonomous vehicles: A constrained multi-objective reinforcement learning technique

Reinforcement learning promises to provide a state-of-the-art solution to the decision making problem of autonomous driving. Nonetheless, numerous real-world decision making problems involve balancing multiple conflicting or competing objectives. In addition, passengers may typically prefer to explore diversified driving modes through their specific preferences (i.e., relative importance of different objectives). Taking into account these demands, traditional reinforcement learning algorithms with applications in personalized self-driving vehicles remain challenging. Consequently, here we present a novel constrained multi-objective reinforcement learning technique for personalized decision making in autonomous driving, with the goal of learning a single model for Pareto optimal policies across the space of all possible user preferences. Specifically, a nonlinear constraint incorporating a user-specified preference and a vectorized action–value function is introduced to ensure both diversity in learned decision behaviors and efficient alignment between the user-specified preference and the corresponding optimal policy. Additionally, a constrained multi-objective actor–critic approach is advanced to approximate the Pareto optimal policies for any user-specified preferences while adhering to the nonlinear constraint. Finally, the proposed personalized decision making scheme for autonomous driving is assessed in a highway on-ramp merging scenario with dynamic traffic flows. The results demonstrate the effectiveness of our method by comparing it with classical and state-of-the-art baselines.

Path Planning for Autonomous Vehicles in Unknown Dynamic Environment Based on Deep Reinforcement Learning

Autonomous vehicles can reduce labor power during cargo transportation, and then improve transportation efficiency, for example, the automated guided vehicle (AGV) in the warehouse can improve the operation efficiency. To overcome the limitations of traditional path planning algorithms in unknown environments, such as reliance on high-precision maps, lack of generalization ability, and obstacle avoidance capability, this study focuses on investigating the Deep Q-Network and its derivative algorithm to enhance network and algorithm structures. A new algorithm named APF-D3QNPER is proposed, which combines the action output method of artificial potential field (APF) with the Dueling Double Deep Q Network algorithm, and experience sample rewards are considered in the experience playback portion of the traditional Deep Reinforcement Learning (DRL) algorithm, which enhances the convergence ability of the traditional DRL algorithm. A long short-term memory (LSTM) network is added to the state feature extraction network part to improve its adaptability in unknown environments and enhance its spatiotemporal sensitivity to the environment. The APF-D3QNPER algorithm is compared with mainstream deep reinforcement learning algorithms and traditional path planning algorithms using a robot operating system and the Gazebo simulation platform by conducting experiments. The results demonstrate that the APF-D3QNPER algorithm exhibits excellent generalization abilities in the simulation environment, and the convergence speed, the loss value, the path planning time, and the path planning length of the APF-D3QNPER algorithm are all less than for other algorithms in diverse scenarios.

Hierarchical Maneuver Decision Method Based on PG-Option for UAV Pursuit-Evasion Game

Aiming at the autonomous decision-making problem in an Unmanned aerial vehicle (UAV) pursuit-evasion game, this paper proposes a hierarchical maneuver decision method based on the PG-option. Firstly, considering various situations of the relationship of both sides comprehensively, this paper designs four maneuver decision options: advantage game, quick escape, situation change and quick pursuit, and the four options are trained by Soft Actor-Critic (SAC) to obtain the corresponding meta-policy. In addition, to avoid high dimensions in the state space in the hierarchical model, this paper combines the policy gradient (PG) algorithm with the traditional hierarchical reinforcement learning algorithm based on the option. The PG algorithm is used to train the policy selector as the top-level strategy. Finally, to solve the problem of frequent switching of meta-policies, this paper sets the delay selection of the policy selector and introduces the expert experience to design the termination function of the meta-policies, which improves the flexibility of switching policies. Simulation experiments show that the PG-option algorithm has a good effect on UAV pursuit-evasion game and adapts to various environments by switching corresponding meta-policies according to current situation.

Asynchronous Federated Deep Reinforcement Learning-Based URLLC-Aware Computation Offloading in Space-Assisted Vehicular Networks

Space-assisted vehicular networks (SAVN) provide seamless coverage and on-demand data processing services for user vehicles (UVs). However, ultra-reliable and low-latency communication (URLLC) demands imposed by emerging vehicular applications are hard to be satisfied in SAVN by existing computation offloading techniques. Traditional deep reinforcement learning algorithms are unsuitable for highly dynamic SAVN due to the underutilization of environment observations. An AsynchronouS federaTed deep Q-learning (DQN)-basEd and URLLC-aware cOmputatIon offloaDing algorithm (ASTEROID) is presented in this paper to achieve throughput maximization considering the long-term URLLC constraints. Specifically, we first establish an extreme value theory-based URLLC constraint model. Second, the task offloading and computation resource allocation are decomposed by employing Lyapunov optimization. Finally, an asynchronous federated DQN-based (AF-DQN) algorithm is presented to address the UV-side task offloading problem. The server-side computation resource allocation is settled by an queue backlog-aware algorithm. Simulation results verify that ASTEROID achieves superior throughput and URLLC performances.

Comprehensive Ocean Information-Enabled AUV Motion Planning Based on Reinforcement Learning

Motion planning based on the reinforcement learning algorithms of the autonomous underwater vehicle (AUV) has shown great potential. Motion planning algorithms are primarily utilized for path planning and trajectory-tracking. However, prior studies have been confronted with some limitations. The time-varying ocean current affects algorithmic sampling and AUV motion and then leads to an overestimation error during path planning. In addition, the ocean current makes it easy to fall into local optima during trajectory planning. To address these problems, this paper presents a reinforcement learning-based motion planning algorithm with comprehensive ocean information (RLBMPA-COI). First, we introduce real ocean data to construct a time-varying ocean current motion model. Then, comprehensive ocean information and AUV motion position are introduced, and the objective function is optimized in the state-action value network to reduce overestimation errors. Finally, state transfer and reward functions are designed based on real ocean current data to achieve multi-objective path planning and adaptive event triggering in trajectorytracking to improve robustness and adaptability. The numerical simulation results show that the proposed algorithm has a better path planning ability and a more robust trajectory-tracking effect than those of traditional reinforcement learning algorithms.