Reinforcement Learning Model Research Articles

Reinforcement learning model for optimizing dexmedetomidine dosing to prevent delirium in critically ill patients

AbstractDelirium can result in undesirable outcomes including increased length of stays and mortality in patients admitted to the intensive care unit (ICU). Dexmedetomidine has emerged for delirium prevention in these patients; however, optimal dosing is challenging. A reinforcement learning-based Artificial Intelligence model for Delirium prevention (AID) is proposed to optimize dexmedetomidine dosing. The model was developed and internally validated using 2416 patients (2531 ICU admissions) and externally validated on 270 patients (274 ICU admissions). The estimated performance return of the AID policy was higher than that of the clinicians’ policy in both derivation (0.390 95% confidence interval [CI] 0.361 to 0.420 vs. −0.051 95% CI −0.077 to −0.025) and external validation (0.186 95% CI 0.139 to 0.236 vs. −0.436 95% CI −0.474 to −0.402) cohorts. Our finding indicates that AID might support clinicians’ decision-making regarding dexmedetomidine dosing to prevent delirium in ICU patients, but further off-policy evaluation is required.

Developing a jam-absorption strategy for mixed traffic flow at signalized intersections using deep reinforcement learning

ABSTRACT Jam-absorption driving (JAD) can effectively prevent the generation and propagation of traffic oscillation. To alleviate the traffic congestion in the signalized intersection with mixed traffic flow, including human driving vehicles (HDVs) and connected and automated vehicles (CAVs), this study provides a jam-absorption driving strategy based on the traffic delay prediction of the mixed platoon under traffic congestion. An online traffic congestion prediction method with the objective of JAD is proposed and focuses on the leaving state of the trajectory to achieve fast capture of congestion features. Then, with real-time status and prediction information, we develop a Jam-absorption driving strategy based on a deep reinforcement learning (DRL) model to improve adaptability to the mixed traffic environment. The results show that this strategy can suppress more than 70% of traffic oscillations with excellent execution efficiency, improving traffic safety and efficiency.

Optimizing intelligent penetration path planning using reinforcement learning: A focus on valid action masking and sample enhancement

Abstract. As penetration testing is a crucial technique for the rapid evolution of cybersecurity to identify system vulnerabilities, traditional penetration testing methods face the difficulty of large amounts of manual labor and professional knowledge, which makes them difficult to scale. This paper proposes a highly automated and efficient penetration testing approach based on the MASK-SALT-DQN algorithm. MASK-SALT-DQN is a reinforcement learning model that effectively optimizes penetration path planning by valid action masking and sample enhancement. Through valid action masking, the MASK-SALT-DQN model filters out redundant and invalid actions in the solution space, reducing the complexity of the action space and improving the convergence speed and efficiency. In addition, the sample enhancement method increases the frequency of critical exploitation actions and positive rewards in the sparse-reward environment, speeding up the learning process of the agent. Experiments have been conducted on the NASim attack simulation network to demonstrate the advantages of the proposed model over the baseline method. The experimental results show that our model has a significant performance advantage over baseline methods in large networks, and it can be applied to the field of scalable and efficient automated penetration testing.

Reinforcement learning in sentiment analysis: a review and future directions

Sentiment analysis in natural language processing (NLP) is used to understand the polarity of human emotions (e.g., positive and negative) and preferences (e.g., price and quality). Reinforcement learning (RL) enables a decision maker (or agent) to observe the operating environment (or the current state) and select the optimal action to receive feedback signals (or reward) from the operating environment. Deep reinforcement learning (DRL) extends RL with deep neural networks (i.e., main and target networks) to capture the state information of inputs and address the curse of dimensionality issue of RL. In sentiment analysis, RL and DRL reduce the need for a large labeled dataset and linguistic resources, increasing scalability and preserving the context and order of logical partitions. Through enhancement, the RL and DRL algorithms identify negations, enhance the quality of the generated responses, predict the logical partitions, remove the irrelevant aspects, and ultimately capture the correct sentiment polarity. This paper presents a review of RL and DRL models and algorithms with their objectives, applications, datasets, performance, and open issues in sentiment analysis.

Exploring the limits of hierarchical world models in reinforcement learning.

Hierarchical model-based reinforcement learning (HMBRL) aims to combine the sample efficiency of model-based reinforcement learning with the abstraction capability of hierarchical reinforcement learning. While HMBRL has great potential, the structural and conceptual complexities of current approaches make it challenging to extract general principles, hindering understanding and adaptation to new use cases, and thereby impeding the overall progress of the field. In this work we describe a novel HMBRL framework and evaluate it thoroughly. We construct hierarchical world models that simulate the environment at various levels of temporal abstraction. These models are used to train a stack of agents that communicate top-down by proposing goals to their subordinate agents. A significant focus of this study is the exploration of a static and environment agnostic temporal abstraction, which allows concurrent training of models and agents throughout the hierarchy. Unlike most goal-conditioned H(MB)RL approaches, it also leads to comparatively low dimensional abstract actions. Although our HMBRL approach did not outperform traditional methods in terms of final episode returns, it successfully facilitated decision-making across two levels of abstraction. A central challenge in enhancing our method's performance, as uncovered through comprehensive experimentation, is model exploitation on the abstract level of our world model stack. We provide an in depth examination of this issue, discussing its implications and suggesting directions for future research to overcome this challenge. By sharing these findings, we aim to contribute to the broader discourse on refining HMBRL methodologies.

A Survey of AI-Powered Proactive Threat-Hunting Techniques: Challenges and Future Directions

The study narrates the dimensions of AI-powered threat hunting techniques that keep changing and draws on their importance in cybersecurity measures while dealing with advanced cyber threats. While the cyber adversary is continuously adapting and innovating, the integration of advanced AI algorithms, including reinforcement learning and hybrid models, has become one of the key strategies for real-time threat detection and response. The research investigates the transformative potential of these algorithms to facilitate adaptive systems that can quickly recognize and act on new attack vectors. It also discusses the integration of AI with emergent technologies like blockchain, quantum computing, and IoT in the dimension of their potential to leverage threat intelligence and response mechanisms. This would naturally lead to increased efficiency in threat hunting by leveraging the strengths brought into this endeavor by both the AI systems and human analysts. The study also advocates for the development of strong policy and governance frameworks to drive home some ethical concerns in deploying AI in cybersecurity. Such a framework is very important for addressing challenges on data privacy, accountability, and transparency. The findings bring into sharp relief that, in this continuing evolution of the cybersecurity landscape, what will work best is a proactive stance-a proactive stance that incorporates innovative technologies with human ingenuity and ethics. This white paper broadly documents the considerable promise of AI in threat hunting and strategic advances needed to harden organizational defenses against a dynamic threat environment.

Centerline-guided reinforcement learning model for pancreatic duct identifications.

Pancreatic ductal adenocarcinoma is forecast to become the second most significant cause of cancer mortality as the number of patients with cancer in the main duct of the pancreas grows, and measurement of the pancreatic duct diameter from medical images has been identified as relevant for its early diagnosis. We propose an automated pancreatic duct centerline tracing method from computed tomography (CT) images that is based on deep reinforcement learning, which employs an artificial agent to interact with the environment and calculates rewards by combining the distances from the target and the centerline. A deep neural network is implemented to forecast step-wise values for each potential action. With the help of this mechanism, the agent can probe along the pancreatic duct centerline using the best possible navigational path. To enhance the tracing accuracy, we employ landmark-based registration, which enables the generation of a probability map of the pancreatic duct. Subsequently, we utilize a gradient-based method on the registered data to extract a probability map specifically indicating the centerline of the pancreatic duct. Three datasets with a total of 115 CT images were used to evaluate the proposed method. Using image hold-out from the first two datasets, the method performance was 2.0, 4.0, and 2.1mm measured in terms of the mean detection error, Hausdorff distance (HD), and root mean squared error (RMSE), respectively. Using the first two datasets for training and the third one for testing, the method accuracy was 2.2, 4.9, and 2.6mm measured in terms of the mean detection error, HD, and RMSE, respectively. We present an algorithm for automated pancreatic duct centerline tracing using deep reinforcement learning. We observe that validation on an external dataset confirms the potential for practical utilization of the presented method.

Risk-averse supply chain management via robust reinforcement learning

Classical reinforcement learning (RL) may suffer performance degradation when the environment deviates from training conditions, limiting its application in risk-averse supply chain management. This work explores using robust RL in supply chain operations to hedge against environment inconsistencies and changes. Two robust RL algorithms, Qˆ-learning and β-pessimistic Q-learning, are examined against conventional Q-learning and a baseline order-up-to inventory policy. Furthermore, this work extends RL applications from forward to closed-loop supply chains. Two case studies are conducted using a supply chain simulator developed with agent-based modeling. The results show that Q-learning can outperform the baseline policy under normal conditions, but notably degrades under environment deviations. By comparison, the robust RL models tend to make more conservative inventory decisions to avoid large shortage penalties. Specifically, fine-tuned β-pessimistic Q-learning can achieve good performance under normal conditions and maintain robustness against moderate environment inconsistencies, making it suitable for risk-averse decision-making.

Human-like Decision Making for autonomous lane change driving: a hybrid inverse reinforcement learning with game-theoretical vehicle interaction model

The development of automated driving vehicles aims to provide safer, comfortable, and more efficient mobility options. However, the decision-making control of autonomous vehicles still embraces limitations on human performance mimicry. These limitations become particularly evident in complex and unfamiliar driving scenarios, where weak decisionmaking abilities and poor adaptation of vehicle behaviour are prominent issues. This paper proposes a game-theoretic decisionmaking algorithm for human-like driving in the vehicle lane change scenario. Firstly, an inverse reinforcement learning (IRL) model is used to quantitatively analyse the lane change trajectories of the natural driving dataset, establishing the human-like human cost function. Subsequently, joint safety, comfort to build the comprehensive decision cost function. Use the combined decision cost function to conduct a non-cooperative game of vehicle lane changing decision to solve the optimal decision of host vehicle lane changing. The host vehicle lane-changing decision problem is formulated as a Stackelberg game optimization problem. To verify the feasibility and effectiveness of the algorithm proposed in this study, a lane change test scenario has been established. Firstly, we analyse the human-like decision-making model derived by the maximum entropy inverse reinforcement learning algorithm to verify the effectiveness and robustness of the IRL algorithm. Secondly, the human-like game decisionmaking algorithm in this paper is validated by conducting an interactive lane-changing experiment with obstacle vehicles of different driving styles. The experimental results prove that the human-like driving decision-making model proposed in this study can make lane-changing behaviours in line with human driving patterns in lane-changing scenarios of expressway.

Reinforcement learning in motor skill acquisition: using the reward positivity to understand the mechanisms underlying short- and long-term behavior adaptation.

According to reinforcement learning, humans adjust their behavior based on the difference between actual and anticipated outcomes (i.e., prediction error) with the main goal of maximizing rewards through their actions. Despite offering a strong theoretical framework to understand how we acquire motor skills, very few studies have investigated reinforcement learning predictions and its underlying mechanisms in motor skill acquisition. In the present study, we explored a 134-person dataset consisting of learners' feedback-evoked brain activity (reward positivity; RewP) and motor accuracy during the practice phase and delayed retention test to investigate whether these variables interacted according to reinforcement learning predictions. Results showed a non-linear relationship between RewP and trial accuracy, which was moderated by the learners' performance level. Specifically, high-performing learners were more sensitive to violations in reward expectations compared to low-performing learners, likely because they developed a stronger representation of the skill and were able to rely on more stable outcome predictions. Furthermore, contrary to our prediction, the average RewP during acquisition did not predict performance on the delayed retention test. Together, these findings support the use of reinforcement learning models to understand short-term behavior adaptation and highlight the complexity of the motor skill consolidation process, which would benefit from a multi-mechanistic approach to further our understanding of this phenomenon.

Hybrid Artificial Intelligence Strategies for Drone Navigation

Objective: This paper describes the development of hybrid artificial intelligence strategies for drone navigation. Methods: The navigation module combines a deep learning model with a rule-based engine depending on the agent state. The deep learning model has been trained using reinforcement learning. The rule-based engine uses expert knowledge to deal with specific situations. The navigation module incorporates several strategies to explain the drone decision based on its observation space, and different mechanisms for including human decisions in the navigation process. Finally, this paper proposes an evaluation methodology based on defining several scenarios and analyzing the performance of the different strategies according to metrics adapted to each scenario. Results: Two main navigation problems have been studied. For the first scenario (reaching known targets), it has been possible to obtain a 90% task completion rate, reducing significantly the number of collisions thanks to the rule-based engine. For the second scenario, it has been possible to reduce 20% of the time required to locate all the targets using the reinforcement learning model. Conclusions: Reinforcement learning is a very good strategy to learn policies for drone navigation, but in critical situations, it is necessary to complement it with a rule-based module to increase task success rate.

Dynamic recommendation for anticoagulant treatment in patients with atrial fibrillation using deep reinforcement learning method

Abstract Background Recent evidence suggests that the guideline-directed anticoagulant therapy for atrial fibrillation (AF) remains controversial. Widely-used CHA2DS2-VASc score is solely based on limited traditional cardiovascular risk factors, omitting AF characteristics and other markers of thromboembolic risk. A more efficient, safer and more personalized anticoagulant approach is warranted. Purpose To develop a data-driven deep reinforcement learning (DRL) model for guiding dynamic anticoagulant treatment in AF patients to improve cardiovascular outcomes. Methods Participants of this study were enrolled from the multicentred China Atrial Fibrillation (China-AF) Registry between August 2011 and December 2022, who were with regular follow-up every 6 months. We excluded patients on warfarin at baseline due to its declining usage trend in non-valvular AF patients in China. The DRL model was trained in 70% randomly selected patients for optimal dynamic decision-making, and then subsequently tested in the remaining 30%. Data of sociodemographic characteristics, AF characteristics, medical history, lifestyle factors, laboratory examination, and medications were input for model training. Concordance rate between DRL model’s recommendations and physicians’ actual decisions of non-vitamin-K-antagonist oral anticoagulant (NOAC) prescription among all visits before censoring was calculated for each patient. Primary outcome was the composite of cardiovascular death, ischemic stroke, transient ischemic attack or systemic embolism (SSE), and major bleeding. Shapley additive explanation analysis ranked the most important factors affecting decision-making of the DRL model. Results A total of 20068 patients (mean age: 63.0±12.0 years; 36.2% female) were randomly divided into a training cohort of 14050 patients and a testing cohort of 6018 patients. The model’s NOAC recommendations were mostly affected by age, prior NOAC prescription, body mass index, hypertension history and prior statin prescription (Figure 1). Patients with concordance rates of 50.1%-75% and 75.1%-100% had significant risk reductions for the primary outcome (adjusted HR =0.63; 95% CI, 0.46-0.85; P = 0.003 and adjusted HR =0.59; 95% CI, 0.46-0.75; P <0.001, respectively), compared to those with a concordance rate of 0-25%. Similar results were observed for all-cause death, cardiovascular death and SSE outcomes, that patients with the highest concordance rate had a significant lower risk, compared to those with the lowest concordance rate. There was a nonsignificant but similar trend with regard to major bleeding events (Figure 2). Conclusions This modelling study suggests that a data-driven DRL model might provide more efficient, safer and more personalized anticoagulant suggestions, potentially assisting physicians in clinical practice.

Integration of investor behavioral perspective and climate change in reinforcement learning for portfolio optimization

Addressing environmental impact is increasingly imperative for individual investors and large financial institutions, making it a key objective of socially responsible investing. However, there is a noticeable gap in research on integrating sustainability and low-carbon considerations into machine learning-based portfolio optimization. To meet this challenge, this study introduces a Portfolio Emissions Sentiment Attention Aware Reinforcement Learning (PESAARL) model based on the Proximal Policy Optimization (PPO) algorithm to optimize a portfolio of Dow Jones Industrial Average (DJIA) stocks. PESAARL uniquely integrates environmental impact considerations, specifically carbon footprint using the firm level scope 1 and scope 2 emissions data, alongside firm-level investor sentiment and attention, into the investment decision-making process. Through multiple experiments, PESAARL demonstrates significant advantages, in terms of financial and environmental performance, over the benchmarks.

Youths' sensitivity to social media feedback: A computational account.

While it is often argued that continuous exposure to social feedback is specifically challenging for the hypersensitive developing brain, empirical evidence is lacking. Across three studies, we reveal the developmental differences and computational mechanisms that underlie the social media engagement and feedback processing of adolescents and adults. First, using a reinforcement learning model on a large Instagram trace dataset (N = 16,613, 1.6+ million posts), we show that adolescents are more sensitive to social feedback than adults. Second, in an experimental study (N = 194), we show that adolescents' mood is affected more strongly by a reduction in likes than adults. Last, in a neuroimaging study (N = 96), we show that social media feedback sensitivity is related to individual differences in subcortical-limbic brain volumes of emerging adults. Together, these findings highlight the need for digital competence programs to help youth manage the constant feedback they encounter on social media platforms.

Autonomous driving decision-making control algorithm based on hierarchical reinforcement learning

In the decision-making model of autonomous driving based on learning, the reinforcement learning model has the problems of slow convergence and single application scenario, while the imitation learning model has the problem of poor generalization. In order to solve this problem, a two-layer reinforcement learning framework is proposed to replace the decision layer and control layer in the autonomous driving task. The decision layer divides the driving behavior into lane keeping, left lane change and right lane change. After the decision layer selects the corresponding behavior, it completes the behavior by changing the input of the control layer. Then, a new method for training the control layer, RL_COE (Reinforcement learning combined with online experts), is proposed by combining reinforcement learning and online experts. Finally, a highway simulation environment is built in Carla to verify the proposed algorithm and compare it with the reinforcement learning baseline algorithm. The results show that this method greatly improves the convergence speed and stability of the algorithm and can better complete the driving task.

A Generalized Deep Reinforcement Learning Model for Distribution Network Reconfiguration with Power Flow-Based Action-Space Sampling

Distribution network reconfiguration (DNR) is used by utilities to enhance power system performance in various ways, such as reducing line losses. Conventional DNR algorithms rely on accurate values of network parameters and lack scalability and optimality. To tackle these issues, a new data-driven algorithm based on reinforcement learning is developed for DNR in this paper. The proposed algorithm comprises two main parts. The first part, named action-space sampling, aims at analyzing the network structure, finding all feasible reconfiguration actions, and reducing the size of the action space to only the most optimal actions. In the second part, deep Q-learning (DQN) and dueling DQN methods are used to train an agent to take the best switching actions according to the switch states and loads of the system. The results show that both DQN and dueling DQN are effective in reducing system losses through grid reconfiguration. The proposed methods have faster execution time compared to the conventional methods and are more scalable.

Reinforcement Learning-Based Optimal Hull Form Design with Variations in Fore and Aft Parts

Abstract With recent advancements in artificial intelligence technology, various studies are being conducted in the shipbuilding industry. Traditionally, hull form variation methods have relied on the intuition and expertise of designers, leading to inconsistent results and unintended changes in the ship's main dimensions depending on the designer's competence. Moreover, the iterative process of design variation and analysis to derive the optimal hull form is both costly and time-consuming. To address these issues, this study proposes an optimal hull design technique utilizing reinforcement learning, a type of unsupervised learning in machine learning. Reinforcement learning allows the model to learn from past policies by recording and accumulating the rewards associated with various actions taken by an agent in a specific environment. In this study, after calculating the main parameters of the ship, the agent defines a state representing hull information and performs local transformations of the bow and stern. The reward of reinforcement learning is defined as the change in total resistance due to the hull deformation, constrained by limiting the tolerance of the ship's prismatic coefficient (CP) and longitudinal center of buoyancy (LCB). In this study, the problem is solved by comparing the proximal policy optimization (PPO) algorithm and the deep deterministic policy gradient (DDPG) algorithm to find the best deep reinforcement learning (DRL) model for the hull optimization problem. The results were compared with the genetic algorithm and speed-constrained multi-objective particle swarm optimization (SMPSO), and the optimal hull resistance values were less different, but the time of the reinforcement learning model was five times shorter.

Deep reinforcement learning augmented decision model for intelligent driving cars

In view of the fact that the state machine decision model cannot effectively handle the rich contextual information and the influence of uncertain factors in ice and snow environments, a deep reinforcement learning agent based on the deep Q network algorithm (DQN) was constructed. The motion planner was used to augment the agent, and the rule-based decision planning module and the deep reinforcement learning model were integrated to establish the DQN-planner model, thereby improving the convergence speed and driving ability of the reinforcement learning agent. Finally, based on the CARLA simulation platform, a comparative experiment was conducted on the driving ability of the DQN model and the DQN-planner model on low-adhesion ice and snow roads, and the training process and verification results were analyzed respectively.

Reward-driven cerebellar climbing fiber activity influences both neural and behavioral learning.

The cerebellum plays a key role in motor coordination and learning. In contrast with classical supervised learning models, recent work has revealed that CFs can signal reward-predictive information in some behaviors. This raises the question of whether CFs may also operate according to principles similar to those described by reinforcement learning models. To test how CFs operate during reward-guided behavior, and evaluate the role of reward-related CF activity in learning, we have measured CF responses in Purkinje cells of the lateral cerebellum during a Pavlovian task using 2-photon calcium imaging. Specifically, we have performed multi-stimulus experiments to determine whether CF activity meets the requirements of a reward prediction error (rPE) signal for transfer from an unexpected reward to a reward-predictive cue. We find that once CF activity is transferred to a conditioned stimulus, and there is no longer a response to reward, CFs cannot generate learned responses to a second conditioned stimulus that carries the same reward prediction. In addition, by expressing the inhibitory opsin GtACR2 in neurons of the inferior olive, and optically inhibiting these neurons across behavioral training at the time of unexpected reward, we find that the transfer of CF signals to the conditioned stimulus is impaired. Moreover, this optogenetic inhibition also impairs learning, resulting in a deficit in anticipatory lick timing. Together, these results indicate that CF signals can exhibit several characteristics in common with rPEs during reinforcement learning, and that the cerebellum can harness these learning signals to generate accurately timed motor behavior.

Reinforcement learning-based assimilation of the WOFOST crop model

Crop model assimilation is a crucial technique for improving the accuracy and precision of crop models by integrating observational data and crop models. Although conventional data assimilation methods such as Kalman filtering and variational methods have been widely applied, these methods often face limitations in data quality, model bias, and high computational complexity. This study explored the potential of reinforcement learning (RL) for crop model assimilation, which has the advantage of not requiring large datasets. Based on the WOFOST crop model, two RL environments were constructed: a Daily-Data Driven approach and a Time-Series Driven approach. The Proximal Policy Optimization (PPO) algorithm was used to train these environments for 100,000 iterations. The assimilation results were compared with the commonly used SUBPLEX optimization algorithm using four-year field measurement data and a public dataset with added random errors. Our results demonstrate that the Time-Series Driven RL model achieved assimilation accuracy comparable to the SUBPLEX optimization algorithm, with an average MAE of 0.65 compared to 0.76 for SUBPLEX, and a slight decrease in RMSE, while significantly reducing the computational burden by 365 times. In a multi-year stability test, the Time-Series Driven RL model and SUBPLEX had similar assimilation performance. This study demonstrates the potential of RL for crop model assimilation, providing a novel approach to overcome the limitations of conventional assimilation algorithms. The findings suggest that RL-based crop model assimilation can improve model accuracy and efficiency, with potential for practical applications in precision agriculture.