Reinforcement Learning Research Articles

Variation and evolution analysis of SARS-CoV-2 using self-game sequence optimization

IntroductionThe evolution of SARS-CoV-2 has precipitated the emergence of new mutant strains, some exhibiting enhanced transmissibility and immune evasion capabilities, thus escalating the infection risk and diminishing vaccine efficacy. Given the continuous impact of SARS-CoV-2 mutations on global public health, the economy, and society, a profound comprehension of potential variations is crucial to effectively mitigate the impact of viral evolution. Yet, this task still faces considerable challenges.MethodsThis study introduces DARSEP, a method based on Deep learning Associates with Reinforcement learning for SARS-CoV-2 Evolution Prediction, combined with self-game sequence optimization and RetNet-based model.ResultsDARSEP accurately predicts evolutionary sequences and investigates the virus’s evolutionary trajectory. It filters spike protein sequences with optimal fitness values from an extensive mutation space, selectively identifies those with a higher likelihood of evading immune detection, and devises a superior evolutionary analysis model for SARS-CoV-2 spike protein sequences. Comprehensive downstream task evaluations corroborate the model’s efficacy in predicting potential mutation sites, elucidating SARS-CoV-2’s evolutionary direction, and analyzing the development trends of Omicron variant strains through semantic changes.ConclusionOverall, DARSEP enriches our understanding of the dynamic evolution of SARS-CoV-2 and provides robust support for addressing present and future epidemic challenges.

Intelligent multi-robot collaborative transport system

The research on multi-robot systems has been divided into various fields, such as communication, navigation, task allocation, and collaborative transport. While significant progress has been made in each area, there has been limited research integrating these fields to build a fully autonomous multi-robot collaborative transport system. Therefore, we identify the key issues and propose a multi-robot collaborative transport system founded on ROS1 and conduct validation in a simulated environment, laying a solid foundation for the system to run on real robots. The primary contributions of this study include three key areas: (1) modeling and validating robot collaborative transport, (2) developing a visual task allocation system leveraging FastDDS service, and (3) resolving path collision issues in multi-robot navigation through both traditional methods and reinforcement learning techniques. Extensive experimental evaluations demonstrate that the proposed intelligent multi-robot collaborative transport system can autonomously navigate to target points for collaborative transport task. Performance assessments, based on the error between the target point and the object’s arrival point as well as the transport trajectory error, reveal that the system effectively completes the assigned tasks.

Multi objective constellation optimization and dynamic link utilization for sustainable information delivery using PD-NOMA deep reinforcement learning

Multi objective constellation optimization and dynamic link utilization for sustainable information delivery using PD-NOMA deep reinforcement learning

Deep Learning Methods in Soft Robotics: Architectures and Applications

The area of soft robotics has been subject to intense research efforts in the past two decades and constitutes a paradigm for advanced machine design in future robotic applications. However, standard methods for industrial robotics may be difficult to apply when analyzing soft robots. Deep learning, which has undergone rapid and transformative advancements in recent years, offers a set of powerful tools for analyzing and designing complex soft machines capable of operating in unstructured environments and interacting with humans and objects in a delicate manner. This review summarizes the most important state‐of‐the‐art deep learning architectures classified under supervised, unsupervised, semisupervised, and reinforcement learning scenarios and discusses their main features and benefits for different soft robotic applications, including soft robot manipulators, soft grippers, soft sensors, and e‐skins, as well as bioinspired soft robots. Specific properties of recent deep learning architectures and the usefulness of their features in addressing various types of issues found in soft robotics are analyzed. The existing challenges and future prospects are identified and discussed in view of the enhanced integration of both areas, which improves the performance of next‐generation soft machines operating in real‐world conditions.

Novelty Classification Model Use in Reinforcement Learning for Cervical Cancer

Purpose: Cervical cancer significantly impacts global health, where early detection is piv- otal for improving patient outcomes. This study aims to enhance the accuracy of cervical cancer diagnosis by addressing class imbalance through a novel hybrid deep learning model. Methods: The proposed model, RL-CancerNet, integrates EfficientNetV2 and Vision Transformers (ViTs) within a Reinforcement Learning (RL) framework. EfficientNetV2 extracts local features from cervical cytology images to capture fine-grained details, while ViTs analyze these features to recognize global dependencies across image patches. To address class imbalance, an RL agent dynamically adjusts the focus towards minority classes, thus reducing the common bias towards majority classes in medical image classification. Additionally, a Supporter Module incorporating Conv3D and BiLSTM layers with an attention mechanism enhances contextual learning. Results: RL-CancerNet was evaluated on the benchmark cervical cytology datasets Herlev and SipaKMeD, achieving an exceptional accuracy of 99.7%. This performance surpasses several state-of-the-art models, demonstrating the model’s effectiveness in identifying subtle diagnostic features in complex backgrounds. Conclusions: The integration of CNNs, ViTs, and RL into RL-CancerNet significantly improves the diagnostic accuracy of cervical cancer screenings. This model not only advances the field of automated medical screening but also provides a scalable framework adaptable to other medical imaging tasks, potentially enhancing diagnostic processes across various medical domains.

Spatio‐temporal dynamic navigation for electric vehicle charging using deep reinforcement learning

AbstractThis paper considers the real‐time spatio‐temporal electric vehicle charging navigation problem in a dynamic environment by utilizing a shortest path‐based reinforcement learning approach. In a data sharing system including transportation network, an electric vehicle (EV) and EV charging stations (EVCSs), it is aimed to determine the most convenient EVCS and the optimal path for reducing the travel, charging and waiting costs. To estimate the waiting times at EVCSs, Gaussian process regression algorithm is integrated using a real‐time dataset comprising of state‐of‐charge and arrival‐departure times of EVs. The optimization problem is modelled as a Markov decision process with unknown transition probability to overcome the uncertainties arising from time‐varying variables. A recently proposed on‐policy actor–critic method, phasic policy gradient (PPG) which extends the proximal policy optimization algorithm with an auxiliary optimization phase to improve training by distilling features from the critic to the actor network, is used to make EVCS decisions on the network where EV travels through the optimal path from origin node to EVCS by considering dynamic traffic conditions, unit value of EV owner and time‐of‐use charging price. Three case studies are carried out for 24 nodes Sioux‐Falls benchmark network. It is shown that phasic policy gradient achieves an average of 9% better reward compared to proximal policy optimization and the total time decreases by 7–10% when EV owner cost is considered.

AI System for Autonomous Vehicles

The rapid advancement of artificial intelligence (AI) has revolutionized the development of autonomous vehicles, offering transformative potential for the future of transportation. This research investigates the implementation of AI-driven algorithms in autonomous vehicles, focusing on their ability to enhance decision-making, navigation, and safety. By employing state-of-the-art machine learning models, including deep learning and reinforcement learning, the study explores how these technologies can optimize real-time processing of sensor data, environmental perception, and adaptive control mechanisms. The findings demonstrate that AI algorithms can significantly improve the accuracy of object detection, trajectory prediction, and path planning, thereby reducing the likelihood of collisions and enhancing overall road safety. A key contribution of this work is the integration of a multi-modal sensor fusion approach, combining data from LiDAR, cameras, radar, and GPS to create a comprehensive and reliable understanding of the vehicle’s surroundings. Additionally, the research highlights the role of AI in enabling autonomous vehicles to learn from vast amounts of driving data, facilitating continuous improvement and adaptability in diverse driving conditions. The implications of this study are profound, suggesting that AI-powered autonomous vehicles could lead to safer, more efficient, and environmentally sustainable transportation systems. However, the research also identifies challenges related to computational complexity, real-time decision-making, and ethical considerations in AI-driven autonomy. Future work will focus on addressing these challenges and exploring the broader societal impacts of widespread autonomous vehicle adoption.

The Future of Smart Home Security: Generative AI and LLMs for Intelligent Event Detection and Personalized Notifications

Abstract—Smart home security cameras are becoming more common, but their usefulness can be diminished by notification fatigue from too many alerts about minor incidents. This paper examines the gaps of existing event detection and notification systems in security cameras and then recommends using Generative AI and Large Language Models (LLMs) to add intelligence which would improve user experience. Generative AI can be leveraged to classify events more accurately and assist with anomaly detection. LLMs can further be used to create notifications that are tailored to the context and personalized to users behavior, helping to reduce notification fatigue and provide meaningful user alerts. The paper also looks into wider applications of these technologies to add intelligence and improve other related experiences like automated video summarization, proactive security measures, and improved privacy controls. The integration of Generative AI and LLMs with smart home security camera systems advances the smart cameras capabilities and offers enhanced security, personalized user experiences. Keywords—Smart home security, Generative AI, Large Language Models (LLMs), Event detection, Anomaly detection, Notification fatigue, Context-aware notifications, Personalized security, Reinforcement Learning from Human Feedback (RLHF), Internet of Things (IoT).

Adaptive Random Early Detection Using Deep Reinforcement Learning for Enhanced Congestion Control in Dynamic TCP/IP Networks

AQM is used to meet targets for congestion control and low delay, high throughput in TCP/IP networks. But methods developed for AQM like Random Early Detection (RED) work strictly on the control parameters pre-set on them and thus may not respond very well to changing network conditions. Thus, this paper presents a composite AQM model using DRL based on DQN, and the ability to learn independently regarding the AQM weight parameter of the queue. The fact that DRL is adaptive means the effectiveness of the proposed system is also assured. As it is observed in implementing traditional model-based approach, the stability and performance degrade when tested in different network conditions; however, DRL’s ability to learn independently enables ML to solve network congestion as it happens. Consequently, more stability is achieved, the delay is lesser, more bandwidth is used and the overall packet drop rate is low; the proposed DRL-RED model is ideal for dynamic network environments. The proposed DRL-RED model is compared with the regular RED algorithm for both low density and high-density networks. This proposed model has achieved sustained throughput of up to 49.9 Mbps with 0.949 % reduction on delay and very low PLR of 8.38043%. From the comparative analysis, it is clear that the employment of DRL with RED improves the stock of network throughput, the reduction on packet drop frequency, and the general delay in network situations. There are two main disadvantages of this approach: first, it cannot reach the heavy-traffic, the problem partly solved by model-based approach; second, the values of the congestion-control parameters cannot be reset, an issue eradicated by DRL as a result of its inherent adaptiveness. Consequently, new stability and increased performance can be achieved under various network conditions.

Resource Allocation in UAV-D2D Networks: A Scalable Heterogeneous Multi-Agent Deep Reinforcement Learning Approach

In unmanned aerial vehicle (UAV)-assisted device-to-device (D2D) caching networks, the uncertainty from unpredictable content demands and variable user positions poses a significant challenge for traditional optimization methods, often making them impractical. Multi-agent deep reinforcement learning (MADRL) offers significant advantages in optimizing multi-agent system decisions and serves as an effective and practical alternative. However, its application in large-scale dynamic environments is severely limited by the curse of dimensionality and communication overhead. To resolve this problem, we develop a scalable heterogeneous multi-agent mean-field actor-critic (SH-MAMFAC) framework. The framework treats ground users (GUs) and UAVs as distinct agents and designs cooperative rewards to convert the resource allocation problem into a fully cooperative game, enhancing global network performance. We also implement a mixed-action mapping strategy to handle discrete and continuous action spaces. A mean-field MADRL framework is introduced to minimize individual agent training loads while enhancing total cache hit probability (CHP). The simulation results show that our algorithm improves CHP and reduces transmission delay. A comparative analysis with existing mainstream deep reinforcement learning (DRL) algorithms shows that SH-MAMFAC significantly reduces training time and maintains high CHP as GU count grows. Additionally, by comparing with SH-MAMFAC variants that do not include trajectory optimization or power control, the proposed joint design scheme significantly reduces transmission delay.

Application of Reinforcement Learning in Controlling Quadrotor UAV Flight Actions

Most literature has extensively discussed reinforcement learning (RL) for controlling rotorcraft drones during flight for traversal tasks. However, most studies lack adequate details regarding the design of reward and punishment mechanisms, and there is a limited exploration of the feasibility of applying reinforcement learning in actual flight control following simulation experiments. Consequently, this study focuses on the exploration of reward and punishment design and state input for RL. The simulation environment is constructed using AirSim and Unreal Engine, with onboard camera footage serving as the state input for reinforcement learning. The research investigates three RL algorithms suitable for discrete action training. The Deep Q Network (DQN), Advantage Actor–Critic (A2C), and Proximal Policy Optimization (PPO) were combined with three different reward and punishment design mechanisms for training and testing. The results indicate that employing the PPO algorithm along with a continuous return method as the reward mechanism allows for effective convergence during the training process, achieving a target traversal rate of 71% in the testing environment. Furthermore, this study proposes integrating the YOLOv7-tiny object detection (OD) system to assess the applicability of reinforcement learning in real-world settings. Unifying the state inputs of simulated and OD environments and replacing the original simulated image inputs with a maximum dual-target approach, the experimental simulation achieved a target traversal rate of 52% ultimately. In summary, this research formulates a set of logical frameworks for an RL reward and punishment design deployed with real-time Yolo’s OD implementation synergized as a useful aid for related RL studies.

A novel reinforcement learning-based method for structure optimization

With the rapid development of deep learning technology, Reinforcement Learning (RL) has garnered considerable acclaim within the realm of structural optimization owing to its excellent exploration mechanism. However, the widespread application of RL in this field is limited owing to the excessive number of iterations required to converge and the expensive computational cost it brings. To address these challenges, this article presents a novel RL framework for structural optimization, combining Monte Carlo tree search with the proximal policy optimization method, called LMPOM. The key contributions of LMPOM encompass: (1) an enhanced Monte Carlo tree search strategy for partitioning the hybrid design space; (2) a strategy for adaptively updating surrogate models to reduce simulation costs; and (3) the introduction of a novel termination condition for the RL algorithms. Through tests on three benchmark problems, compared with previous RL algorithms, LMPOM consistently shows fewer iterations and better optimization results.

Employing battery energy storage systems for flexible ramping products in a fully renewable energy power grid: A market mechanism and strategy analysis through multi-Agent Markov games

In high-proportion renewable energy power systems, flexible ramping products (FRPs) are critical for mitigating the volatility of renewable energy outputs and enhancing the adaptability of power systems. This paper aims to explore a fully renewable energy power system, with a battery energy storage system (BESS) as the sole provider of FRPs. An innovative market mechanism and new approaches to market equilibrium analysis are introduced. Initially, a common day-ahead (DA) market is established, covering both electricity and ancillary services with FRPs. The setup allows FRP providers to make bidding decisions and get rewarded based on actual ramping mileages. A bi-level multi-agent Markov game (MG) model is formulated. The upper level integrates the proposed strategies of wind power plants (WPP), photovoltaic power plants (PVP), and BESS, while the lower level encompasses the joint market clearing mechanism. An iterative solution algorithm, using reinforcement learning, is also introduced to address the model, effectively circumventing the privacy concerns associated with traditional mathematical programming methods. The proposed solution models and strategies are validated and demonstrated to be effective through two illustrative examples, highlighting their application and relevance in fully renewable energy power systems.

Reinforcement learning for dynamic pricing and capacity allocation in monetized customer wait-skipping services

ABSTRACT We consider how to facilitate a dynamically-priced premium service option that enables customer parties to shorten their wait in a queue. Offering such a service requires that some of a business’s capacity be reserved continuously and kept ready for premium customers. In tandem with capacity reservation, pricing must be coordinated. Hence, a joint dynamic pricing and capacity allocation problem lies at the heart of this service. We propose a conceptual solution architecture and employ Proximal Policy Optimization (PPO) for dynamic pricing and capacity allocation to maximize total revenue. Simulation experiments over multiple scenarios compared PPO against a human-engineered policy and a baseline policy having no premium option. The human-engineered policy led to significantly greater revenues than the baseline policy in each scenario, illustrating the potential increase in revenues afforded by this concept. The PPO agent substantially improved upon the human-engineered policy advantage, with improvements ranging from 28% to 161%.

Access management based on deep reinforcement learning for effective cloud storage security

Access management based on deep reinforcement learning for effective cloud storage security

Low-Carbon Dispatch Method for Active Distribution Network Based on Carbon Emission Flow Theory

In the context of integrating renewable energy sources such as wind and solar energy sources into distribution networks, this paper proposes a proactive low-carbon dispatch model for active distribution networks based on carbon flow calculation theory. This model aims to achieve accurate carbon measurement across all operational aspects of distribution networks, reduce their carbon emissions through controlling unit operations, and ensure stable and safe operation. First, we propose a method for measuring carbon emission intensity on the source and network sides of active distribution networks with network losses, allowing for the calculation of total carbon emissions throughout the operation of networks and their equipment. Next, based on the carbon flow distribution of distribution networks, we construct a low-carbon dispatch model and formulate its optimization problem within a Markov Decision Process framework. We improve the Soft Actor–Critic (SAC) algorithm by adopting a Gaussian-distribution-based reward function to train and deploy agents for optimal low-carbon dispatch. Finally, the effectiveness of the proposed model and the superiority of the improved algorithm are demonstrated using a modified IEEE 33-bus distribution network test case.

Enhancing stability and explainability in reinforcement learning with machine learning

Abstract. In the field of reinforcement learning, training agents using machine learning algorithms to learn and perform tasks in complex environments has become a prevalent approach. However, reinforcement learning faces challenges such as training instability and decision opacity, which limit its feasibility in real-world applications. To solve the problems of stability and transparency in reinforcement learning, this project will use advanced algorithms like Proximal Policy Optimization (PPO), Q-DAGGER, and Gradient Boosting Decision Trees to set up reinforcement learning agents in the OpenAI Gymnasium environment. Specifically, the study selected the Atari game Breakout as the testbed, enhancing training efficiency and game performance by refining reward structures and decision-making processes, and integrating interpretable models to provide explanations for agent decisions. This study has successfully developed robust reinforcement learning agents that excel in complex environments. By employing advanced algorithms like PPO, Q-DAGGER, and Gradient Boosting Decision Trees, the study has addressed issues of training instability, and improved game performance through optimized reward structures and decision processes. Additionally, by integrating interpretable models, the study has provided insights into the learned strategies of the agents, thereby enhancing decision transparency. These findings provide crucial support for the broader application of reinforcement learning in real-world scenarios and offer valuable insights for tackling other complex tasks.

Evade Unknown Pursuer via Pursuit Strategy Identification and Model Reference Policy Adaptation (MRPA) Algorithm

The game of pursuit–evasion has always been a popular research subject in the field of Unmanned Aerial Vehicles (UAVs). Current evasion decision making based on reinforcement learning is generally trained only for specific pursuers, and it has limited performance for evading unknown pursuers and exhibits poor generalizability. To enhance the ability of an evasion policy learned by reinforcement learning (RL) to evade unknown pursuers, this paper proposes a pursuit UAV attitude estimation and pursuit strategy identification method and a Model Reference Policy Adaptation (MRPA) algorithm. Firstly, this paper constructs a Markov decision model for the pursuit–evasion game of UAVs that includes the pursuer’s attitude and trains an evasion policy for a specific pursuit strategy using the Soft Actor–Critic (SAC) algorithm. Secondly, this paper establishes a novel relative motion model of UAVs in pursuit–evasion games under the assumption that proportional guidance is used as the pursuit strategy, based on which the pursuit UAV attitude estimation and pursuit strategy identification algorithm is proposed to provide adequate information for decision making and policy adaptation. Furthermore, a Model Reference Policy Adaptation (MRPA) algorithm is presented to improve the generalizability of the evasion policy trained by RL in certain environments. Finally, various numerical simulations imply the precision of pursuit UAV attitude estimation and the accuracy of pursuit strategy identification. Also, the ablation experiment verifies that the MRPA algorithm can effectively enhance the performance of the evasion policy to deal with unknown pursuers.

Research on robot path planning and obstacle avoidance algorithm in dynamic environment based on deep reinforcement learning

Abstract. In dynamic environments, robot path planning and obstacle avoidance are critical tasks, especially in applications such as autonomous driving, industrial automation, and mobile robotics. These tasks are inherently challenging due to the unpredictability of the environment and the need for real-time decision-making. This paper seeks to address these challenges by developing and analyzing both traditional and optimized models for robot navigation. The initial model utilizes a basic Q-learning algorithm, which provides a straightforward approach to learning from the environment but often struggles with the complexity of dynamic scenarios. To this end, an optimized model is developed that combines the Double Deep Q-Learning algorithm (Double DQN) in conjunction with heuristic strategies. The research employs the MATLAB Reinforcement Learning Toolbox to implement and train these models, and utilizes a simulated environment with dynamic obstacles as a testing site. The simulation generates the necessary data to allow for comprehensive testing and evaluation of the models performance. The results show that the optimized model greatly exceeds the initial model in terms of path planning efficiency and obstacle avoidance capabilities, and that the combination of advanced reinforcement learning techniques and heuristic strategies is extremely important for enhancing the performance and reliability of robotic systems in complex, dynamic environments, offering valuable insights for future applications in various fields of robotics.

Curiosity-Driven Multi-Level Intrinsic Reward DQN for Enhanced Exploration in Reinforcement Learning

Abstract. Reinforcement learning (RL) has shown great potential in solving complex decision-making tasks. However, efficient exploration remains a significant challenge, particularly in environments with sparse or deceptive rewards. This paper introduces DQN-Mult-Cur, a novel reinforcement learning algorithm that enhances exploration by integrating curiosity-driven and multi-level intrinsic rewards within the Deep Q-Network (DQN) framework. The proposed method addresses the limitations of conventional exploration strategies by incentivizing agents to explore novel and meaningful states, thereby improving learning efficiency and performance. Extensive experiments across three standard environmentsCartPole-v1, MountainCar-v0 and Acrobot-v1demonstrate that DQN-Mult-Cur outperforms traditional DQN variants, achieving faster convergence, higher rewards, and greater stability. An ablation study further highlights the importance of each intrinsic reward component, confirming the robustness of the proposed approach. The results suggest that DQN-Mult-Cur offers a comprehensive solution to the exploration-exploitation trade-off in reinforcement learning, making it applicable to a wide range of challenging environments.