Reinforcement Learning Model Research Articles

Exploring the limits of hierarchical world models in reinforcement learning.

Hierarchical model-based reinforcement learning (HMBRL) aims to combine the sample efficiency of model-based reinforcement learning with the abstraction capability of hierarchical reinforcement learning. While HMBRL has great potential, the structural and conceptual complexities of current approaches make it challenging to extract general principles, hindering understanding and adaptation to new use cases, and thereby impeding the overall progress of the field. In this work we describe a novel HMBRL framework and evaluate it thoroughly. We construct hierarchical world models that simulate the environment at various levels of temporal abstraction. These models are used to train a stack of agents that communicate top-down by proposing goals to their subordinate agents. A significant focus of this study is the exploration of a static and environment agnostic temporal abstraction, which allows concurrent training of models and agents throughout the hierarchy. Unlike most goal-conditioned H(MB)RL approaches, it also leads to comparatively low dimensional abstract actions. Although our HMBRL approach did not outperform traditional methods in terms of final episode returns, it successfully facilitated decision-making across two levels of abstraction. A central challenge in enhancing our method's performance, as uncovered through comprehensive experimentation, is model exploitation on the abstract level of our world model stack. We provide an in depth examination of this issue, discussing its implications and suggesting directions for future research to overcome this challenge. By sharing these findings, we aim to contribute to the broader discourse on refining HMBRL methodologies.

Human-like Decision Making for autonomous lane change driving: a hybrid inverse reinforcement learning with game-theoretical vehicle interaction model

The development of automated driving vehicles aims to provide safer, comfortable, and more efficient mobility options. However, the decision-making control of autonomous vehicles still embraces limitations on human performance mimicry. These limitations become particularly evident in complex and unfamiliar driving scenarios, where weak decisionmaking abilities and poor adaptation of vehicle behaviour are prominent issues. This paper proposes a game-theoretic decisionmaking algorithm for human-like driving in the vehicle lane change scenario. Firstly, an inverse reinforcement learning (IRL) model is used to quantitatively analyse the lane change trajectories of the natural driving dataset, establishing the human-like human cost function. Subsequently, joint safety, comfort to build the comprehensive decision cost function. Use the combined decision cost function to conduct a non-cooperative game of vehicle lane changing decision to solve the optimal decision of host vehicle lane changing. The host vehicle lane-changing decision problem is formulated as a Stackelberg game optimization problem. To verify the feasibility and effectiveness of the algorithm proposed in this study, a lane change test scenario has been established. Firstly, we analyse the human-like decision-making model derived by the maximum entropy inverse reinforcement learning algorithm to verify the effectiveness and robustness of the IRL algorithm. Secondly, the human-like game decisionmaking algorithm in this paper is validated by conducting an interactive lane-changing experiment with obstacle vehicles of different driving styles. The experimental results prove that the human-like driving decision-making model proposed in this study can make lane-changing behaviours in line with human driving patterns in lane-changing scenarios of expressway.

Reinforcement learning in motor skill acquisition: using the reward positivity to understand the mechanisms underlying short- and long-term behavior adaptation

IntroductionAccording to reinforcement learning, humans adjust their behavior based on the difference between actual and anticipated outcomes (i.e., prediction error) with the main goal of maximizing rewards through their actions. Despite offering a strong theoretical framework to understand how we acquire motor skills, very few studies have investigated reinforcement learning predictions and its underlying mechanisms in motor skill acquisition.MethodsIn the present study, we explored a 134-person dataset consisting of learners’ feedback-evoked brain activity (reward positivity; RewP) and motor accuracy during the practice phase and delayed retention test to investigate whether these variables interacted according to reinforcement learning predictions.ResultsResults showed a non-linear relationship between RewP and trial accuracy, which was moderated by the learners’ performance level. Specifically, high-performing learners were more sensitive to violations in reward expectations compared to low-performing learners, likely because they developed a stronger representation of the skill and were able to rely on more stable outcome predictions. Furthermore, contrary to our prediction, the average RewP during acquisition did not predict performance on the delayed retention test.DiscussionTogether, these findings support the use of reinforcement learning models to understand short-term behavior adaptation and highlight the complexity of the motor skill consolidation process, which would benefit from a multi-mechanistic approach to further our understanding of this phenomenon.

Hybrid Artificial Intelligence Strategies for Drone Navigation

Objective: This paper describes the development of hybrid artificial intelligence strategies for drone navigation. Methods: The navigation module combines a deep learning model with a rule-based engine depending on the agent state. The deep learning model has been trained using reinforcement learning. The rule-based engine uses expert knowledge to deal with specific situations. The navigation module incorporates several strategies to explain the drone decision based on its observation space, and different mechanisms for including human decisions in the navigation process. Finally, this paper proposes an evaluation methodology based on defining several scenarios and analyzing the performance of the different strategies according to metrics adapted to each scenario. Results: Two main navigation problems have been studied. For the first scenario (reaching known targets), it has been possible to obtain a 90% task completion rate, reducing significantly the number of collisions thanks to the rule-based engine. For the second scenario, it has been possible to reduce 20% of the time required to locate all the targets using the reinforcement learning model. Conclusions: Reinforcement learning is a very good strategy to learn policies for drone navigation, but in critical situations, it is necessary to complement it with a rule-based module to increase task success rate.

Dynamic recommendation for anticoagulant treatment in patients with atrial fibrillation using deep reinforcement learning method

Abstract Background Recent evidence suggests that the guideline-directed anticoagulant therapy for atrial fibrillation (AF) remains controversial. Widely-used CHA2DS2-VASc score is solely based on limited traditional cardiovascular risk factors, omitting AF characteristics and other markers of thromboembolic risk. A more efficient, safer and more personalized anticoagulant approach is warranted. Purpose To develop a data-driven deep reinforcement learning (DRL) model for guiding dynamic anticoagulant treatment in AF patients to improve cardiovascular outcomes. Methods Participants of this study were enrolled from the multicentred China Atrial Fibrillation (China-AF) Registry between August 2011 and December 2022, who were with regular follow-up every 6 months. We excluded patients on warfarin at baseline due to its declining usage trend in non-valvular AF patients in China. The DRL model was trained in 70% randomly selected patients for optimal dynamic decision-making, and then subsequently tested in the remaining 30%. Data of sociodemographic characteristics, AF characteristics, medical history, lifestyle factors, laboratory examination, and medications were input for model training. Concordance rate between DRL model’s recommendations and physicians’ actual decisions of non-vitamin-K-antagonist oral anticoagulant (NOAC) prescription among all visits before censoring was calculated for each patient. Primary outcome was the composite of cardiovascular death, ischemic stroke, transient ischemic attack or systemic embolism (SSE), and major bleeding. Shapley additive explanation analysis ranked the most important factors affecting decision-making of the DRL model. Results A total of 20068 patients (mean age: 63.0±12.0 years; 36.2% female) were randomly divided into a training cohort of 14050 patients and a testing cohort of 6018 patients. The model’s NOAC recommendations were mostly affected by age, prior NOAC prescription, body mass index, hypertension history and prior statin prescription (Figure 1). Patients with concordance rates of 50.1%-75% and 75.1%-100% had significant risk reductions for the primary outcome (adjusted HR =0.63; 95% CI, 0.46-0.85; P = 0.003 and adjusted HR =0.59; 95% CI, 0.46-0.75; P <0.001, respectively), compared to those with a concordance rate of 0-25%. Similar results were observed for all-cause death, cardiovascular death and SSE outcomes, that patients with the highest concordance rate had a significant lower risk, compared to those with the lowest concordance rate. There was a nonsignificant but similar trend with regard to major bleeding events (Figure 2). Conclusions This modelling study suggests that a data-driven DRL model might provide more efficient, safer and more personalized anticoagulant suggestions, potentially assisting physicians in clinical practice.

Integration of investor behavioral perspective and climate change in reinforcement learning for portfolio optimization

Addressing environmental impact is increasingly imperative for individual investors and large financial institutions, making it a key objective of socially responsible investing. However, there is a noticeable gap in research on integrating sustainability and low-carbon considerations into machine learning-based portfolio optimization. To meet this challenge, this study introduces a Portfolio Emissions Sentiment Attention Aware Reinforcement Learning (PESAARL) model based on the Proximal Policy Optimization (PPO) algorithm to optimize a portfolio of Dow Jones Industrial Average (DJIA) stocks. PESAARL uniquely integrates environmental impact considerations, specifically carbon footprint using the firm level scope 1 and scope 2 emissions data, alongside firm-level investor sentiment and attention, into the investment decision-making process. Through multiple experiments, PESAARL demonstrates significant advantages, in terms of financial and environmental performance, over the benchmarks.

Youths' sensitivity to social media feedback: A computational account.

While it is often argued that continuous exposure to social feedback is specifically challenging for the hypersensitive developing brain, empirical evidence is lacking. Across three studies, we reveal the developmental differences and computational mechanisms that underlie the social media engagement and feedback processing of adolescents and adults. First, using a reinforcement learning model on a large Instagram trace dataset (N = 16,613, 1.6+ million posts), we show that adolescents are more sensitive to social feedback than adults. Second, in an experimental study (N = 194), we show that adolescents' mood is affected more strongly by a reduction in likes than adults. Last, in a neuroimaging study (N = 96), we show that social media feedback sensitivity is related to individual differences in subcortical-limbic brain volumes of emerging adults. Together, these findings highlight the need for digital competence programs to help youth manage the constant feedback they encounter on social media platforms.

Autonomous driving decision-making control algorithm based on hierarchical reinforcement learning

In the decision-making model of autonomous driving based on learning, the reinforcement learning model has the problems of slow convergence and single application scenario, while the imitation learning model has the problem of poor generalization. In order to solve this problem, a two-layer reinforcement learning framework is proposed to replace the decision layer and control layer in the autonomous driving task. The decision layer divides the driving behavior into lane keeping, left lane change and right lane change. After the decision layer selects the corresponding behavior, it completes the behavior by changing the input of the control layer. Then, a new method for training the control layer, RL_COE (Reinforcement learning combined with online experts), is proposed by combining reinforcement learning and online experts. Finally, a highway simulation environment is built in Carla to verify the proposed algorithm and compare it with the reinforcement learning baseline algorithm. The results show that this method greatly improves the convergence speed and stability of the algorithm and can better complete the driving task.

A Generalized Deep Reinforcement Learning Model for Distribution Network Reconfiguration with Power Flow-Based Action-Space Sampling

Distribution network reconfiguration (DNR) is used by utilities to enhance power system performance in various ways, such as reducing line losses. Conventional DNR algorithms rely on accurate values of network parameters and lack scalability and optimality. To tackle these issues, a new data-driven algorithm based on reinforcement learning is developed for DNR in this paper. The proposed algorithm comprises two main parts. The first part, named action-space sampling, aims at analyzing the network structure, finding all feasible reconfiguration actions, and reducing the size of the action space to only the most optimal actions. In the second part, deep Q-learning (DQN) and dueling DQN methods are used to train an agent to take the best switching actions according to the switch states and loads of the system. The results show that both DQN and dueling DQN are effective in reducing system losses through grid reconfiguration. The proposed methods have faster execution time compared to the conventional methods and are more scalable.

Reinforcement Learning-Based Optimal Hull Form Design with Variations in Fore and Aft Parts

Abstract With recent advancements in artificial intelligence technology, various studies are being conducted in the shipbuilding industry. Traditionally, hull form variation methods have relied on the intuition and expertise of designers, leading to inconsistent results and unintended changes in the ship's main dimensions depending on the designer's competence. Moreover, the iterative process of design variation and analysis to derive the optimal hull form is both costly and time-consuming. To address these issues, this study proposes an optimal hull design technique utilizing reinforcement learning, a type of unsupervised learning in machine learning. Reinforcement learning allows the model to learn from past policies by recording and accumulating the rewards associated with various actions taken by an agent in a specific environment. In this study, after calculating the main parameters of the ship, the agent defines a state representing hull information and performs local transformations of the bow and stern. The reward of reinforcement learning is defined as the change in total resistance due to the hull deformation, constrained by limiting the tolerance of the ship's prismatic coefficient (CP) and longitudinal center of buoyancy (LCB). In this study, the problem is solved by comparing the proximal policy optimization (PPO) algorithm and the deep deterministic policy gradient (DDPG) algorithm to find the best deep reinforcement learning (DRL) model for the hull optimization problem. The results were compared with the genetic algorithm and speed-constrained multi-objective particle swarm optimization (SMPSO), and the optimal hull resistance values were less different, but the time of the reinforcement learning model was five times shorter.

Deep reinforcement learning augmented decision model for intelligent driving cars

In view of the fact that the state machine decision model cannot effectively handle the rich contextual information and the influence of uncertain factors in ice and snow environments, a deep reinforcement learning agent based on the deep Q network algorithm (DQN) was constructed. The motion planner was used to augment the agent, and the rule-based decision planning module and the deep reinforcement learning model were integrated to establish the DQN-planner model, thereby improving the convergence speed and driving ability of the reinforcement learning agent. Finally, based on the CARLA simulation platform, a comparative experiment was conducted on the driving ability of the DQN model and the DQN-planner model on low-adhesion ice and snow roads, and the training process and verification results were analyzed respectively.

Reward-driven cerebellar climbing fiber activity influences both neural and behavioral learning.

The cerebellum plays a key role in motor coordination and learning. In contrast with classical supervised learning models, recent work has revealed that CFs can signal reward-predictive information in some behaviors. This raises the question of whether CFs may also operate according to principles similar to those described by reinforcement learning models. To test how CFs operate during reward-guided behavior, and evaluate the role of reward-related CF activity in learning, we have measured CF responses in Purkinje cells of the lateral cerebellum during a Pavlovian task using 2-photon calcium imaging. Specifically, we have performed multi-stimulus experiments to determine whether CF activity meets the requirements of a reward prediction error (rPE) signal for transfer from an unexpected reward to a reward-predictive cue. We find that once CF activity is transferred to a conditioned stimulus, and there is no longer a response to reward, CFs cannot generate learned responses to a second conditioned stimulus that carries the same reward prediction. In addition, by expressing the inhibitory opsin GtACR2 in neurons of the inferior olive, and optically inhibiting these neurons across behavioral training at the time of unexpected reward, we find that the transfer of CF signals to the conditioned stimulus is impaired. Moreover, this optogenetic inhibition also impairs learning, resulting in a deficit in anticipatory lick timing. Together, these results indicate that CF signals can exhibit several characteristics in common with rPEs during reinforcement learning, and that the cerebellum can harness these learning signals to generate accurately timed motor behavior.

Reinforcement learning-based assimilation of the WOFOST crop model

Crop model assimilation is a crucial technique for improving the accuracy and precision of crop models by integrating observational data and crop models. Although conventional data assimilation methods such as Kalman filtering and variational methods have been widely applied, these methods often face limitations in data quality, model bias, and high computational complexity. This study explored the potential of reinforcement learning (RL) for crop model assimilation, which has the advantage of not requiring large datasets. Based on the WOFOST crop model, two RL environments were constructed: a Daily-Data Driven approach and a Time-Series Driven approach. The Proximal Policy Optimization (PPO) algorithm was used to train these environments for 100,000 iterations. The assimilation results were compared with the commonly used SUBPLEX optimization algorithm using four-year field measurement data and a public dataset with added random errors. Our results demonstrate that the Time-Series Driven RL model achieved assimilation accuracy comparable to the SUBPLEX optimization algorithm, with an average MAE of 0.65 compared to 0.76 for SUBPLEX, and a slight decrease in RMSE, while significantly reducing the computational burden by 365 times. In a multi-year stability test, the Time-Series Driven RL model and SUBPLEX had similar assimilation performance. This study demonstrates the potential of RL for crop model assimilation, providing a novel approach to overcome the limitations of conventional assimilation algorithms. The findings suggest that RL-based crop model assimilation can improve model accuracy and efficiency, with potential for practical applications in precision agriculture.

Learning to Play Guitar with Robotic Hands

AbstractPlaying the guitar is a dexterous human skill that poses significant challenges in computer graphics and robotics due to the precision required in finger positioning and coordination between hands. Current methods often rely on motion capture data to replicate specific guitar playing segments, which restricts the range of performances and demands intricate post‐processing. In this paper, we introduce a novel reinforcement learning model that can play the guitar using robotic hands, without the need for motion capture datasets, from input tablatures. To achieve this, we divide the simulation task for playing guitar into three stages. (a): for an input tablature, we first generate corresponding fingerings that align with human habits. (b): based on the generated fingerings as the guidance, we train a neural network for controlling the fingers of the left hand using deep reinforcement learning, and (c): we generate plucking movements for the right hand based on inverse kinematics according to the tablature. We evaluate our method by employing precision, recall, and F1 scores as quantitative metrics to thoroughly assess its performance in playing musical notes. In addition, we conduct qualitative analysis through user studies to evaluate the visual and auditory effects of guitar performance. The results demonstrate that our model excels in playing most moderately difficult and easier musical pieces, accurately playing nearly all notes.

Individualized decision making in on-scene resuscitation time for out-of-hospital cardiac arrest using reinforcement learning

On-scene resuscitation time is associated with out-of-hospital cardiac arrest (OHCA) outcomes. We developed and validated reinforcement learning models for individualized on-scene resuscitation times, leveraging nationwide Korean data. Adult OHCA patients with a medical cause of arrest were included (N = 73,905). The optimal policy was derived from conservative Q-learning to maximize survival. The on-scene return of spontaneous circulation hazard rates estimated from the Random Survival Forest were used as intermediate rewards to handle sparse rewards, while patients’ historical survival was reflected in the terminal rewards. The optimal policy increased the survival to hospital discharge rate from 9.6% to 12.5% (95% CI: 12.2–12.8) and the good neurological recovery rate from 5.4% to 7.5% (95% CI: 7.3–7.7). The recommended maximum on-scene resuscitation times for patients demonstrated a bimodal distribution, varying with patient, emergency medical services, and OHCA characteristics. Our survival analysis-based approach generates explainable rewards, reducing subjectivity in reinforcement learning.

Optimizing Maintenance of Energy Supply Systems in City Logistics with Heuristics and Reinforcement Learning

In urban logistics, effective maintenance is crucial for maintaining the reliability and efficiency of energy supply systems, impacting both asset performance and operational stability. This paper addresses the scheduling and routing plans for maintenance of power generation assets over a multi-period horizon. We model this problem as a multi-period team orienteering problem. To address this multi-period challenge, we propose a dual approach: a novel reinforcement learning (RL) framework and a biased-randomized heuristic algorithm. The RL-based method dynamically learns from real-time operational data and evolving asset conditions, adapting to changes in asset health and failure probabilities to optimize decision making. In addition, we develop and apply a biased-randomized heuristic algorithm designed to provide effective solutions within practical computational limits. Our approach is validated through a series of computational experiments comparing the RL model and the heuristic algorithm. The results demonstrate that, when properly trained, the RL-based model is able to offer equivalent or even superior performance compared to the heuristic algorithm.

PID Controller Based on Improved DDPG for Trajectory Tracking Control of USV

When navigating dynamic ocean environments characterized by significant wave and wind disturbances, USVs encounter time-varying external interferences and underactuated limitations. This results in reduced navigational stability and increased difficulty in trajectory tracking. Controllers based on deterministic models or non-adaptive control parameters often fail to achieve the desired performance. To enhance the adaptability of USV motion controllers, this paper proposes a trajectory tracking control algorithm that calculates PID control parameters using an improved Deep Deterministic Policy Gradient (DDPG) algorithm. Firstly, the maneuvering motion model and parameters for USVs are introduced, along with the guidance law for path tracking and the PID control algorithm. Secondly, a detailed explanation of the proposed method is provided, including the state, action, and reward settings for training the Reinforcement Learning (RL) model. Thirdly, the simulations of various algorithms, including the proposed controller, are presented and analyzed for comparison, demonstrating the superiority of the proposed algorithm. Finally, a maneuvering experiment under wave conditions was conducted in a marine tank using the proposed algorithm, proving its feasibility and effectiveness. This research contributes to the intelligent navigation of USVs in real ocean environments and facilitates the execution of subsequent specific tasks.

Optimizing a Dynamic Vehicle Routing Problem with Deep Reinforcement Learning: Analyzing State-Space Components

Background: The dynamic vehicle routing problem (DVRP) is a complex optimization problem that is crucial for applications such as last-mile delivery. Our goal is to develop an application that can make real-time decisions to maximize total performance while adapting to the dynamic nature of incoming orders. We formulate the DVRP as a vehicle routing problem where new customer requests arrive dynamically, requiring immediate acceptance or rejection decisions. Methods: This study leverages reinforcement learning (RL), a machine learning paradigm that operates via feedback-driven decisions, to tackle the DVRP. We present a detailed RL formulation and systematically investigate the impacts of various state-space components on algorithm performance. Our approach involves incrementally modifying the state space, including analyzing the impacts of individual components, applying data transformation methods, and incorporating derived features. Results: Our findings demonstrate that a carefully designed state space in the formulation of the DVRP significantly improves RL performance. Notably, incorporating derived features and selectively applying feature transformation enhanced the model’s decision-making capabilities. The combination of all enhancements led to a statistically significant improvement in the results compared with the basic state formulation. Conclusions: This research provides insights into RL modeling for DVRPs, highlighting the importance of state-space design. The proposed approach offers a flexible framework that is applicable to various variants of the DVRP, with potential for validation using real-world data.

Trends in quantum reinforcement learning: State‐of‐the‐arts and the road ahead

AbstractThis paper presents the basic quantum reinforcement learning theory and its applications to various engineering problems. With the advances in quantum computing and deep learning technologies, various research works have focused on quantum deep learning and quantum machine learning. In this paper, quantum neural network (QNN)‐based reinforcement learning (RL) models are discussed and introduced. Moreover, the pros of the QNN‐based RL algorithms and models, such as fast training, high scalability, and efficient learning parameter utilization, are presented along with various research results. In addition, one of the well‐known multi‐agent extensions of QNN‐based RL models, the quantum centralized‐critic and multiple‐actor network, is also discussed and its applications to multi‐agent cooperation and coordination are introduced. Finally, the applications and future research directions are introduced and discussed in terms of federated learning, split learning, autonomous control, and quantum deep learning software testing.

Intelligent Music Emotion Analysis and Creation Based on Reinforcement Learning Model

Intelligent Music Emotion Analysis and Creation Based on Reinforcement Learning Model