Reinforcement Learning Method Research Articles

An algorithm that excavates suboptimal states and improves Q-learning

Abstract Reinforcement learning is inspired by the trial-and-error method in animal learning, where the reward values obtained from the interaction of the agent with the environment are used as feedback signals to train the agent. Reinforcement learning has attracted extensive attention in recent years. It is mainly used to solve sequential decision-making problems and has been applied to various aspects of life, such as autonomous driving, game gaming, and robotics. Exploration and exploitation are the main characteristics that distinguish reinforcement learning methods from other learning methods. Reinforcement learning methods need reward optimization algorithms to better balance exploration and exploitation. Aiming at the problems of unbalanced exploration and a large number of repeated explorations in the Q-learning algorithm in the MDP environment, an algorithm that excavates suboptimal states and improves Q-learning was proposed. It adopts the exploration idea of ‘exploring the potential of the second-best’, and explores the state with suboptimal state value, and calculates the exploration probability value according to the distance between the current state and the goal state. The larger the distance, the higher the exploration demand of the agent. In addition, only the immediate reward and the maximum action value of the next state are needed to calculate the Q value. Through the simulation experiments in two different MDP environments, The frozenLake8x8 environment and the CliffWalking environment, the results verify that the proposed algorithm obtains the highest average cumulative reward and the least total time consumption.

Novel reinforcement learning technique based parameter estimation for proton exchange membrane fuel cell model

Proton Exchange Membrane Fuel Cells (PEMFCs) offer a clean and sustainable alternative to traditional engines. PEMFCs play a vital role in progressing hydrogen-based energy solutions. Accurate modeling of PEMFC performance is essential for enhancing their efficiency. This paper introduces a novel reinforcement learning (RL) approach for estimating PEMFC parameters, addressing the challenges of the complex and nonlinear dynamics of the PEMFCs. The proposed RL method minimizes the sum of squared errors between measured and simulated voltages and provides an adaptive and self-improving RL-based Estimation that learns continuously from system feedback. The RL-based approach demonstrates superior accuracy and performance compared with traditional metaheuristic techniques. It has been validated through theoretical and experimental comparisons and tested on commercial PEMFCs, including the Temasek 1 kW, the 6 kW Nedstack PS6, and the Horizon H-12 12 W. The dataset used in this study comes from experimental data. This research contributes to the precise modeling of PEMFCs, improving their efficiency, and developing wider adoption of PEMFCs in sustainable energy solutions.

Decision-Making Policy for Autonomous Vehicles on Highways Using Deep Reinforcement Learning (DRL) Method

Automated driving (AD) is a new technology that aims to mitigate traffic accidents and enhance driving efficiency. This study presents a deep reinforcement learning (DRL) method for autonomous vehicles that can safely and efficiently handle highway overtaking scenarios. The first step is to create a highway traffic environment where the agent can be guided safely through surrounding vehicles. A hierarchical control framework is then provided to manage high-level driving decisions and low-level control commands, such as speed and acceleration. Next, a special DRL-based method called deep deterministic policy gradient (DDPG) is used to derive decision strategies for use on the highway. The performance of the DDPG algorithm is compared with that of the DQN and PPO algorithms, and the results are evaluated. The simulation results show that the DDPG algorithm can effectively and safely handle highway traffic tasks.

A Comprehensive Review of Mobile Robot Navigation Using Deep Reinforcement Learning Algorithms in Crowded Environments

Navigation is a crucial challenge for mobile robots. Currently, deep reinforcement learning has attracted considerable attention and has witnessed substantial development owing to its robust performance and learning capabilities in real-world scenarios. Scientists leverage the advantages of deep neural networks, such as long short-term memory, recurrent neural networks, and convolutional neural networks, to integrate them into mobile robot navigation based on deep reinforcement learning. This integration aims to enhance the robot's motion control performance in both static and dynamic environments. This paper illustrates a comprehensive survey of deep reinforcement learning methods applied to mobile robot navigation systems in crowded environments, exploring various navigation frameworks based on deep reinforcement learning and their benefits over traditional simultaneous localization and mapping-based frameworks. Subsequently, we comprehensively compare and analyze the relationships and differences among three types of navigation: autonomous-based navigation, navigation based on simultaneous localization and mapping, and planning-based navigation. Moreover, the crowded environment includes static, dynamic, and a combination of obstacles in different typical application scenarios. Finally, we offer insights into the evolution of navigation based on deep reinforcement learning, addressing the problems and providing potential solutions associated with this emerging field.

Multi-Vehicle Cooperative Decision-Making in Merging Area Based on Deep Multi-Agent Reinforcement Learning

In recent years, reinforcement learning (RL) methods have shown powerful learning capabilities in single-vehicle autonomous driving. However, few studies have focused on multi-vehicle cooperative driving based on RL, particularly in the dynamically changing traffic environments of highway ramp merge zones. In this paper, a multi-agent deep reinforcement learning (MARL) framework for multi-vehicle cooperative decision-making is proposed based on actor–critic, which categorizes vehicles into two groups according to their origins in the merging area. At the same time, the complexity of the network is reduced and the training process of the model is accelerated by utilizing mechanisms such as partial parameter sharing and experience playback. Additionally, a combination of global and individual rewards is adopted to promote cooperation in connected autonomous vehicles (CAVs) and balance individual and group interests. The training performance of the model is compared under three traffic densities, and our method is also compared with state-of-the-art benchmark methods. The simulation results show that the proposed MARL framework can have stronger policy learning capability and stability under various traffic flow conditions. Moreover, it can also effectively improve the speed of vehicles in the merging zone and reduce traffic conflicts.

Joint Learning of Volume Scheduling and Order Placement Policies for Optimal Order Execution

Order execution is an extremely important problem in the financial domain, and recently, more and more researchers have tried to employ reinforcement learning (RL) techniques to solve this challenging problem. There are a lot of difficulties for conventional RL methods to tackle the order execution problem, such as the large action space including price and quantity, and the long-horizon property. As naturally order execution is composed of a low-frequency volume scheduling stage and a high-frequency order placement stage, most existing RL-based order execution methods treat these stages as two distinct tasks and offer a partial solution by addressing either one individually. However, the current literature fails to model the non-negligible mutual influence between these two tasks, leading to impractical order execution solutions. To address these limitations, we propose a novel automatic order execution approach based on the hierarchical RL framework (OEHRL), which jointly learns the policies for volume scheduling and order placement. OEHRL first extracts the state embeddings at both the macro and micro levels with a sequential variational auto-encoder model. Based on the effective embeddings, OEHRL generates a hindsight expert dataset, which is used to train a hierarchical order execution policy. In the hierarchical structure, the high-level policy is in charge of the target volume and the low-level learns to determine the prices for a series of the allocated sub-orders from the high level. These two levels collaborate seamlessly and contribute to the optimal order execution policy. Extensive experiment results on 200 stocks across the US and China A-share markets validate the effectiveness of the proposed approach.

Lunar Leap Robot: 3M Architecture–Enhanced Deep Reinforcement Learning Method for Quadruped Robot Jumping in Low-Gravity Environment

Lunar Leap Robot: 3M Architecture–Enhanced Deep Reinforcement Learning Method for Quadruped Robot Jumping in Low-Gravity Environment

Reinforcement learning method based on sample regularization and adaptive learning rate for AGV path planning

Reinforcement learning method based on sample regularization and adaptive learning rate for AGV path planning

Meta-learning and proximal policy optimization driven two-stage emergency allocation strategy for multi-energy system against typhoon disasters

To achieve resilience improvement of the multi-energy system against typhoon disasters, this study designs a novel two-stage optimization framework that considers the emergency allocation of distributed resources under typhoon disasters to fully exploit the potential of distributed resources for resilience enhancement. Firstly, we formulate the emergency allocation of distributed resources as a Markov decision process. Then, a meta-learning-driven proximal policy optimization method is utilized to solve it. Different from that the existing reinforcement learning methods always ignore the unpredictable change caused by typhoon and keep multi-energy system dynamics invariant, limiting its control performance. The proposed method embeds meta-learning to fine-tune the pre-trained allocation policy to new tasks with high adaptability and few interactions. Finally, comparison results with other benchmark methods are carried out and shows that the proposed method can learn the appropriate resource allocation policy for multi-energy system and achieve better resilience enhancement, yielding fast application efficiency and good generalization ability for emergency fault conditions.

A Two-Stage Multi-Agent Deep Reinforcement Learning Method for Urban Distribution Network Reconfiguration Considering Switch Contribution

A Two-Stage Multi-Agent Deep Reinforcement Learning Method for Urban Distribution Network Reconfiguration Considering Switch Contribution

Research on Bionic Robot Motion Control Based on Reinforcement Learning

This research explores the application of reinforcement learning (RL) to enhance the motion control of bio-inspired robots across various environments. Focusing on underwater, terrestrial, and aerial robotic models, the study integrates model-free Q-learning, Deep Q Network (DQN), State-Action-Reward-State-Action (SARSA), and Double Deep Q Network (DDQN) algorithms. These methods are employed to achieve adaptive and efficient movement strategies without relying on pre-defined environmental models. The methodologies range from real-time data capture using live camera feeds for terrestrial robots to simulating aquatic and flight dynamics in controlled environments. Experimental results confirm the efficacy of these RL methods, demonstrating significant improvements in the robots' ability to adapt to dynamic and unknown environments, optimize movement efficiency, and navigate complex scenarios. The findings suggest a promising direction for future robotic applications, emphasizing the need for further research on optimizing these algorithms and expanding their real-world applicability. This study highlights the potential of RL to revolutionize robotic motion control, making robots more versatile and capable in varied and unpredictable settings.

Novel Application of Quantum Computing for Routing and Spectrum Assignment in Flexi-Grid Optical Networks

Flexi-grid technology has revolutionized optical networking by enabling Elastic Optical Networks (EONs) that offer greater flexibility and dynamism compared to traditional fixed-grid systems. As data traffic continues to grow exponentially, the need for efficient and scalable solutions to the routing and spectrum assignment (RSA) problem in EONs becomes increasingly critical. The RSA problem, being NP-Hard, requires solutions that can simultaneously address both spatial routing and spectrum allocation. This paper proposes a novel quantum-based approach to solving the RSA problem. By formulating the problem as a Quadratic Unconstrained Binary Optimization (QUBO) model, we employ the Quantum Approximate Optimization Algorithm (QAOA) to effectively solve it. Our approach is specifically designed to minimize end-to-end delay while satisfying the continuity and contiguity constraints of frequency slots. Simulations conducted using the Qiskit framework and IBM-QASM simulator validate the effectiveness of our method. We applied the QAOA-based RSA approach to small network topology, where the number of nodes and frequency slots was constrained by the limited qubit count on current quantum simulator. In this small network, the algorithm successfully converged to an optimal solution in less than 30 iterations, with a total runtime of approximately 10.7 s with an accuracy of 78.8%. Additionally, we conducted a comparative analysis between QAOA, integer linear programming, and deep reinforcement learning methods to evaluate the performance of the quantum-based approach relative to classical techniques. This work lays the foundation for future exploration of quantum computing in solving large-scale RSA problems in EONs, with the prospect of achieving quantum advantage as quantum technology continues to advance.

Learning from different perspectives for regret reduction in reinforcement learning: A free energy approach

Reinforcement learning (RL) is the core method for interactive learning in living and artificial creatures. Nevertheless, in contrast to humans and animals, artificial RL agents are very slow in learning and suffer from the curse of dimensionality. This is partially due to using RL in isolation; i.e. lack of social learning and social diversity. We introduce a free energy-based social RL for learning novel tasks. Society is formed by the learning agent and some diverse virtual ones. That diversity is in their perception while all agents use the same interaction samples for learning and share the same action set. Individual difference in perception is mostly the cause of perceptual aliasing however, it can result in virtual agents’ faster learning in early trials. Our free energy method provides a knowledge integration method for the main agent to benefit from that diversity to reduce its regret. It rests upon Thompson sampling policy and behavioral policy of main and virtual agents. Therefore, it is applicable to a variety of tasks, discrete or continuous state space, model-free, and model-based tasks as well as to different reinforcement learning methods. Through a set of experiments, we show that this general framework highly improves learning speed and is clearly superior to previous existing methods. We also provide convergence proof.

The Advantage of Board Game with Deep Reinforcement Learning and Causal Inference

With the continuous progress of artificial intelligence technology, the deep reinforcement learning method combining behaviourism and connectionism has shown fantastic performance far beyond the human level in chess games. This paper systematically introduces two mainstream chess games, including the standard algorithms of artificial intelligence technology in chess and Go, including Monte Carlo and deep reinforcement learning, and expounds on their algorithm principles and the core reasons for their strong learning ability. In addition, this paper discusses the limitations of existing methods in interpretability and puts forward the possibility of applying causal reasoning to deep reinforcement learning to solve interpretable problems. This study will provide a valuable reference for practitioners in related fields.

Reinforcement Learning Algorithm for Optimising Durian Irrigation Systems: Maximising Growth and Water Efficiency

This study presents a Reinforcement Learning-based algorithm designed to optimise irrigation for Durio Zibethinus (i.e., durian) trees, aiming to maximise tree growth and reduce water usage. Traditional irrigation methods, as well as current machine learning models, often focus only on soil moisture and weather data, neglecting critical factors like actual tree growth. This study proposed a reinforcement learning irrigation (RL-Irr) algorithm incorporating tree growth stages, soil moisture, and weather conditions to determine precise irrigation needs. The algorithm was developed by calibrating the AQUACROP model using data from actual durian plantations where rain-fed irrigation (rain-fed) was practised. Daily irrigation volumes were calculated based on real-time soil moisture, weather forecasts, and weekly tree growth measurements. The reinforcement learning method was used to optimise irrigation schedules, with rewards based on soil moisture, tree growth, rainfall, and weather conditions. The algorithm was tested using AQUACROP simulations and compared against soil moisture balance irrigation (SMB-Irr) and rain-fed. The results showed that the RL-Irr reduced water use by up to 75 percent while maintaining tree growth. These findings suggest the algorithm could significantly improve water efficiency in durian farming, though real-world applications should consider potential model limitations.

Dynamic recommendation for anticoagulant treatment in patients with atrial fibrillation using deep reinforcement learning method

Abstract Background Recent evidence suggests that the guideline-directed anticoagulant therapy for atrial fibrillation (AF) remains controversial. Widely-used CHA2DS2-VASc score is solely based on limited traditional cardiovascular risk factors, omitting AF characteristics and other markers of thromboembolic risk. A more efficient, safer and more personalized anticoagulant approach is warranted. Purpose To develop a data-driven deep reinforcement learning (DRL) model for guiding dynamic anticoagulant treatment in AF patients to improve cardiovascular outcomes. Methods Participants of this study were enrolled from the multicentred China Atrial Fibrillation (China-AF) Registry between August 2011 and December 2022, who were with regular follow-up every 6 months. We excluded patients on warfarin at baseline due to its declining usage trend in non-valvular AF patients in China. The DRL model was trained in 70% randomly selected patients for optimal dynamic decision-making, and then subsequently tested in the remaining 30%. Data of sociodemographic characteristics, AF characteristics, medical history, lifestyle factors, laboratory examination, and medications were input for model training. Concordance rate between DRL model’s recommendations and physicians’ actual decisions of non-vitamin-K-antagonist oral anticoagulant (NOAC) prescription among all visits before censoring was calculated for each patient. Primary outcome was the composite of cardiovascular death, ischemic stroke, transient ischemic attack or systemic embolism (SSE), and major bleeding. Shapley additive explanation analysis ranked the most important factors affecting decision-making of the DRL model. Results A total of 20068 patients (mean age: 63.0±12.0 years; 36.2% female) were randomly divided into a training cohort of 14050 patients and a testing cohort of 6018 patients. The model’s NOAC recommendations were mostly affected by age, prior NOAC prescription, body mass index, hypertension history and prior statin prescription (Figure 1). Patients with concordance rates of 50.1%-75% and 75.1%-100% had significant risk reductions for the primary outcome (adjusted HR =0.63; 95% CI, 0.46-0.85; P = 0.003 and adjusted HR =0.59; 95% CI, 0.46-0.75; P <0.001, respectively), compared to those with a concordance rate of 0-25%. Similar results were observed for all-cause death, cardiovascular death and SSE outcomes, that patients with the highest concordance rate had a significant lower risk, compared to those with the lowest concordance rate. There was a nonsignificant but similar trend with regard to major bleeding events (Figure 2). Conclusions This modelling study suggests that a data-driven DRL model might provide more efficient, safer and more personalized anticoagulant suggestions, potentially assisting physicians in clinical practice.

Multi-UAV Escape Target Search: A Multi-Agent Reinforcement Learning Method

The multi-UAV target search problem is crucial in the field of autonomous Unmanned Aerial Vehicle (UAV) decision-making. The algorithm design of Multi-Agent Reinforcement Learning (MARL) methods has become integral to research on multi-UAV target search owing to its adaptability to the rapid online decision-making required by UAVs in complex, uncertain environments. In non-cooperative target search scenarios, targets may have the ability to escape. Target probability maps are used in many studies to characterize the likelihood of a target’s existence, guiding the UAV to efficiently explore the task area and locate the target more quickly. However, the escape behavior of the target causes the target probability map to deviate from the actual target’s position, thereby reducing its effectiveness in measuring the target’s probability of existence and diminishing the efficiency of the UAV search. This paper investigates the multi-UAV target search problem in scenarios involving static obstacles and dynamic escape targets, modeling the problem within the framework of decentralized partially observable Markov decision process. Based on this model, a spatio-temporal efficient exploration network and a global convolutional local ascent mechanism are proposed. Subsequently, we introduce a multi-UAV Escape Target Search algorithm based on MAPPO (ETS–MAPPO) for addressing the escape target search difficulty problem. Simulation results demonstrate that the ETS–MAPPO algorithm outperforms five classic MARL algorithms in terms of the number of target searches, area coverage rate, and other metrics.

Reinforcement Learning Model-Based and Model-Free Paradigms for Optimal Control Problems in Power Systems: Comprehensive Review and Future Directions

This paper reviews recent works related to applications of reinforcement learning in power system optimal control problems. Based on an extensive analysis of works in the recent literature, we attempt to better understand the gap between reinforcement learning methods that rely on complete or incomplete information about the model dynamics and data-driven reinforcement learning approaches. More specifically we ask how such models change based on the application or the algorithm, what the currently open theoretical and numerical challenges are in each of the leading applications, and which reinforcement-based control strategies will rise in the following years. The reviewed research works are divided into “model-based” methods and “model-free” methods in order to highlight the current developments and trends within each of these two groups. The optimal control problems reviewed are energy markets, grid stability and control, energy management in buildings, electrical vehicles, and energy storage.

Multi-Agent Deep Reinforcement Learning-Based Distributed Voltage Control of Flexible Distribution Networks with Soft Open Points

The increasing number of distributed generators (DGs) leads to the frequent occurrence of voltage violations in distribution networks. The soft open point (SOP) can adjust the transmission power between feeders, leading to the evolution of traditional distribution networks into flexible distribution networks (FDN). The problem of voltage violations can be effectively tackled with the flexible control of SOPs. However, the centralized control method for SOP may make it difficult to achieve real-time control due to the limitations of communication. In this paper, a distributed voltage control method is proposed for FDN with SOPs based on the multi-agent deep reinforcement learning (MADRL) method. Firstly, a distributed voltage control framework is proposed, in which the updating algorithm of the intelligent agent of MADRL is expounded considering experience sharing. Then, a Markov decision process for multi-area SOP coordinated voltage control is proposed, where the control areas are divided based on electrical distance. Finally, an IEEE 33-node test system and a practical system in Taiwan are used to verify the effectiveness of the proposed method. It shows that the proposed multi-area SOP coordinated control method can achieve real-time control while ensuring a better control effect.

Unification of probabilistic graph model and deep reinforcement learning (UPGMDRL) for multi-intersection traffic signal control

Traffic signals play a pivotal role in modern life by preventing collisions, regulating traffic flow, and ensuring a predictable and efficient transportation system. Adaptive traffic light signal control (ATSC) is a promising paradigm for mitigating traffic congestion in Intelligent Transportation Systems (ITS). Among various AI-based approaches, Deep Reinforcement Learning (DRL) has gained widespread application, demonstrating superior performance. This paper aims to develop a latent space reinforcement learning method for intelligent traffic control, with a focus on making explainable decisions. According to the latent model and hidden Markov mixed model, this method integrated both to develop an ATSC framework for traffic networks with multiple intersections. Given the challenges posed by high-dimensional data and a limited understanding of the task, traditional decision-making methods often struggle with understanding the environment. This paper aims to provide semantic information and an enhanced understanding of the environment by offering interpretable states. The latent model is employed to extract task-relevant information from underlying representations within a framework that unifies representation learning and DRL. The experimental results demonstrate how our approach effectively and efficiently balances traffic flow, leading to improved traffic management.