Reinforcement Learning Principles Research Articles

Design of a Model Using Machine Learning and Deep Dyna Q Learning Integration for Improved Disease Prediction in Remote Healthcare

In the domain of proactive healthcare management, the imperative for remote health monitoring has escalated, the remote health care in this scenario specially means, the patient is seating at the remote location that is not in the hospital setting, and doctor or healthcare worker is monitoring the health parameters gathered using biomedical sensors and passed through the network. Conventional methodologies, while partially effective, encounter challenges in predictive precision, responsiveness to evolving health dynamics, and managing the vast array of patient data. These limitations underscore the demand for a sophisticated, holistic solution catering to diverse use cases. This work introduces a pioneering framework amalgamating traditional machine learning (ML) models with the advanced capabilities of Deep Dyna Q Learning process to overcome existing constraints. This framework strategically utilizes ensemble of traditional algorithms which amalgamates the strengths of these diverse models. Central to this model is the integration of Deep Dyna Q Learning, empowering the system with real-time adaptability and dynamic decision-making process through reinforcement learning principles, thereby deriving insights from historical and simulated datasets to foster more nuanced, patient-centric decisions. The impact of this comprehensive approach is profound, evidenced by preliminary results showcasing significant enhancements in the efficiency of remote health monitoring systems. Notably, the model achieves increase in precision, accuracy and recall for disease prediction. These improvements signify a paradigm shift towards proactive and efficient healthcare interventions, especially in remote settings. The fusion of traditional ML techniques with Deep Dyna Q Learning emerges as a potent solution, heralding a revolution in remote health monitoring and establishing a new benchmark for proactive healthcare delivery scenarios.

Optimization and Application of English Word Memory Algorithm Based on Reinforcement Learning

The optimization and application of an English word memory algorithm based on reinforcement learning involve refining a system that utilizes reinforcement learning techniques to enhance vocabulary retention. By leveraging reinforcement learning, the algorithm can adapt its strategies for presenting and reviewing English words based on users' responses and performance. This adaptive approach enables personalized and effective learning experiences, wherein the algorithm adjusts the frequency and timing of word repetitions to maximize memorization efficiency. Applications of this optimized algorithm span language learning platforms, educational apps, and personalized tutoring systems, offering learners tailored support to strengthen their English vocabulary acquisition and retention skills. This paper introduces the Spider Swarm Optimized English Word Memory Algorithm (SSOEWMA) algorithm, a novel approach aimed at optimizing language processing and learning tasks. Leveraging principles of reinforcement learning and swarm intelligence, SSOEWMA demonstrates its effectiveness in enhancing various aspects of language processing, including word difficulty assessment, sentence estimation, memory retention, and student performance evaluation. Through a series of experiments and evaluations, we showcase SSOEWMA ability to optimize dataset attributes, accurately estimate sentence characteristics, and improve memory retention performance. Simulation results achieved an average word recall accuracy of 85% across multiple evaluations. Through its utilization of reinforcement learning principles and swarm intelligence, SSOEWMA significantly improved memory retention performance, reducing recall latency by an average of 20% compared to baseline measures. Additionally, the algorithm demonstrated strong learning efficiency, converging to optimal solutions with a speed increase of 30% compared to traditional methods.

A Reinforcement Learning-Based Traffic Engineering Algorithm for Enterprise Network Backbone Links

Large enterprise networks typically rely on expensive, high-speed backbone links to connect multiple campuses across diverse regions. As the volume of traffic traversing these backbone links increases, traffic engineering techniques are employed to filter or redirect traffic flows. Nevertheless, simple rerouting strategies can introduce business disruptions such as packet reordering, which significantly impact the user experience. To address this issue, we introduce an enhanced traffic scheduling algorithm named Critical Flow Rerouting with Weight- Reinforcement Learning(CFRW-RL), which builds upon the critical flow rerouting-reinforcement learning (CFR-RL) algorithm. CFRW-RL incorporates the principles of reinforcement learning, accounting for both the weights and classifications of data flows. This approach enables the algorithm to prioritize flows with lower weights for rerouting. The simulation results demonstrate that CFRW-RL significantly minimizes the rerouting of high-priority business flows and reduces business interference compared with the CFR-RL algorithm and that it maintains a similar computational complexity.

Advancing spacecraft rendezvous and docking through safety reinforcement learning and ubiquitous learning principles

As spacecraft rendezvous and docking missions become increasingly complex, the need for advanced solutions has surged. In recent years, the application of reinforcement learning techniques to tackle spacecraft rendezvous guidance challenges has emerged as a prominent international trend. Vital to ensuring the secure rendezvous and docking of spacecraft is the task of obstacle avoidance. However, traditional reinforcement learning algorithms lack the ability to enforce safety constraints within the exploration space, which presents a formidable obstacle in the design of spacecraft rendezvous guidance strategies. In response to this challenge, a spacecraft rendezvous guidance methodology founded on safety reinforcement learning is proposed. Firstly, a Markov model is crafted for autonomous spacecraft rendezvous in scenarios involving collision avoidance. A reward system, contingent on obstacle warnings and collision avoidance constraints, is introduced to establish a safety reinforcement learning framework for devising spacecraft rendezvous guidance strategies. Secondly, within the framework of safety reinforcement learning, two deep reinforcement learning (DRL) algorithms, Proximal Policy Optimisation (PPO) and Deep Deterministic Policy Gradient (DDPG), are leveraged to generate these guidance strategies. Experimental findings validate the effectiveness of this approach in successfully executing obstacle avoidance and achieving rendezvous with remarkable precision. Furthermore, through an analysis of the performance and generalization capabilities of these two algorithms, the efficacy of the proposed methodology is further underscored. This fusion of advanced space guidance technology with the principles of Ubiquitous Learning marks a significant step forward in the quest for safer and more efficient spacecraft rendezvous and docking operations.

Distilling dynamical knowledge from stochastic reaction networks

Stochastic reaction networks are widely used in the modeling of stochastic systems across diverse domains such as biology, chemistry, physics, and ecology. However, the comprehension of the dynamic behaviors inherent in stochastic reaction networks is a formidable undertaking, primarily due to the exponential growth in the number of possible states or trajectories as the state space dimension increases. In this study, we introduce a knowledge distillation method based on reinforcement learning principles, aimed at compressing the dynamical knowledge encoded in stochastic reaction networks into a singular neural network construct. The trained neural network possesses the capability to accurately predict the state conditional joint probability distribution that corresponds to the given query contexts, when prompted with rate parameters, initial conditions, and time values. This obviates the need to track the dynamical process, enabling the direct estimation of normalized state and trajectory probabilities, without necessitating the integration over the complete state space. By applying our method to representative examples, we have observed a high degree of accuracy in both multimodal and high-dimensional systems. Additionally, the trained neural network can serve as a foundational model for developing efficient algorithms for parameter inference and trajectory ensemble generation. These results collectively underscore the efficacy of our approach as a universal means of distilling knowledge from stochastic reaction networks. Importantly, our methodology also spotlights the potential utility in harnessing a singular, pretrained, large-scale model to encapsulate the solution space underpinning a wide spectrum of stochastic dynamical systems.

Multi-granularity coverage criteria for deep reinforcement learning systems

Deep reinforcement learning (DRL) systems are progressively being deployed in safety- and security-critical domains, such as self-driving cars and unmanned aerial vehicles, raising concerns about their trustworthiness. DRL, a integration of deep learning (DL) and reinforcement learning (RL) principles, represents a machine learning technique. Existing DL coverage criteria primarily focus on neuron coverage, overlooking the unique features of RL, thus falling short in assessing the test adequacy of DRL systems.This paper introduces the first set of coverage criteria designed to systematically measure the elements of DRL systems exercised by test inputs. DRL elements, including states, actions, and policies, are leveraged to define coverage criteria that consider multi-levels and multi-granularities. Furthermore, these coverage criteria undergo optimization through the application of combinatorial coverage principles. State coverage is employed as feedback to guide test case selection for DRL systems. Empirical studies have been conducted to assess the performance of the proposed coverage criteria on five well-known DRL environments. The experiments demonstrate the effectiveness of these coverage criteria in detecting unexpected behaviors, highlighting that the proposed test case selection guided by state coverage serves as an effective strategy.

Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care.

Medical treatment decisions inherently involve a series of sequential choices, each informed by the outcomes of preceding decisions. This process closely aligns with the principles of reinforcement learning (RL), which also focuses on sequential decisions aimed at maximizing cumulative rewards. Consequently, RL holds significant promise for developing data-driven treatment plans. However, a major challenge in applying RL within medical contexts lies in the sparse nature of the rewards, which are primarily based on mortality outcomes. This sparsity can reduce the stability of offline estimates, posing a significant hurdle in fully utilizing RL for medical decision-making. In this work, we introduce a deep Q-learning approach able to obtain more reliable critical care policies. This method integrates relevant but noisy intermediate biomarker signals into the reward specification without compromising the optimization of the main outcome of interest (e.g., patient survival). We achieve this by first pruning the action space based on all available rewards, and then training a final model based on the (sparse) main reward, while only choosing actions available within the pruned action space. By disentangling sparse rewards and frequently measured reward proxies through action pruning, potential distortions of the main objective are minimized, all while enabling the extraction of valuable information from intermediate signals that can guide the learning process. We evaluate our method in both off-policy and offline settings using simulated environments and real health records of patients in intensive care units. Our empirical results indicate that our method outperforms common offline RL methods such as conservative Q-learning and batch-constrained deep Q-learning. Our work is a step towards developing reliable policies by effectively harnessing the wealth of available information in data-intensive critical care environments.

RL-HAT: A New Framework for Understanding Human-Agent Teaming

This paper presents a novel framework for human-agent teaming grounded in the principles of Reinforcement Learning (RL). Recognizing the need for a unified language across various disciplines, we utilize RL concepts to provide a standard for the understanding and evaluation of diverse teaming strategies. Our framework extends beyond traditional RL constructs, integrating aspects such as belief states, prior knowledge, social considerations, situational awareness, and mental models. A particular focus is placed on the role of ethics and trust in effective teaming. Additionally, we discuss how sensor data, perception models, and actuator modules can be incorporated, emphasizing the adaptability of our framework to a broad range of tasks and environments. We believe this work forms a substantial contribution to the field of human-agent teaming, establishing a solid foundation for future research and application.

Soft actor-critic DRL algorithm for interval optimal dispatch of integrated energy systems with uncertainty in demand response and renewable energy

The collaborative optimization dispatching of multiple energy flows plays a crucial role in achieving the economic and efficient low-carbon operation of integrated energy systems (IESs). However, the dispatching problem for IESs is characterized by high dimensionality, non-linearity, and complex coupling. Furthermore, the integration of renewable energy sources and flexible loads has transformed the IES into a complex dynamic system with significant uncertainty. Traditional intelligent optimization algorithms exhibit poor adaptability and lengthy solution computation time when tackling such problems. In contrast, deep reinforcement learning (DRL), as an interactive trial-and-error learning method, has shown improved decision-making capabilities. In view of this, a data-driven soft actor-critic (SAC) deep reinforcement learning-based approach is proposed in this paper for interval optimal dispatch of IESs considering multiple uncertainties. First, the basic principle of SAC reinforcement learning is introduced in detail, and the basic framework of reinforcement learning for interval optimal scheduling of IESs is constructed. Then, the environment model of agent interaction is constructed, and the action and state space of DRL, as well as the reward mechanism and neural network structure, are designed. Finally, a typical IES case is experimentally analyzed and compared with three popular DRL algorithms and five state-of-the-art intelligent optimization algorithms. The experimental results demonstrate the advantages and effectiveness of the proposed method in solving the optimal dispatching of IESs.

Review of Application of Model-free Reinforcement Learning in Intelligent Decision

With the continuous development of intelligent technologies, the traditional field of decision making is gradually evolving towards Intelligent Decision (ID), but with the increasing complexity of the environment, the explosive growth of data volume and the uncertainty of the decision making process, the difficulty of decision analysis is in-creasing. As a branch of Machine Learning, Reinforcement Learning (RL) uses Agent to train and generate rewards from the environment, ultimately resulting in intelligent models. Model-free Reinforcement Learning (MFRL) is a type of reinforcement learning in which an Agent does not need a predefined model of the environment, but inter-acts directly with the environment and learns autonomously to generate optimal strategies for model generation in complex environments. Model-free Reinforcement Learning techniques applied in the field of Intelligent Decision making can improve the efficiency and accuracy of decision making in complex environments. In this paper, we provide an overview of Model-free Reinforcement Learning in intelligent decision making and introduce the basic principles of reinforcement learning and its two branches (Model-based Reinforcement Learning and Model-free Reinforcement Learning). Various algorithms of Model-free Reinforcement Learning are analyzed and disassembled from two different functions (value-based function and policy-based function), and the characteristics, applicability range, and research results of each algorithm are derived. The typical applications of Model-free Reinforcement Learning in the field of intelligent decision making are classified and analyzed. Finally, a summary and outlook on the application of Model-free Reinforcement Learning in Intelligent Decision making are presented.

Optimizing fuel economy and durability of hybrid fuel cell electric vehicles using deep reinforcement learning-based energy management systems

An effective energy management strategy (EMS) is crucial for the reliable operation of fuel cell hybrid electric vehicles (FCHEVs). This study proposes a power distribution optimization strategy for FCHEVs that leverages deep reinforcement learning (DRL) and Pontryagin's minimum principle (PMP). The DRL algorithm effectively balances fuel economy, battery durability, and fuel cell durability objectives. The degradation mechanisms of battery and fuel cell under extreme working conditions are considered in the PMP optimization. A comprehensive evaluation framework is established with degradation and energy consumption models to serve as a reward for deep reinforcement learning to balance fuel economy and power sources' lifetime. Simulation results show that the proposed EMS framework reduces FC degradation by 18.4% and battery degradation by 71.1% compared to traditional PMP-based EMS under the NEDC driving condition. Hardware-in-the-loop (HIL) testing demonstrates that the proposed EMS framework has the potential for real-time application, with an average relative error between experiment and simulation of approximately 0.0203. This research highlights the significance of the proposed EMS framework in ensuring the reliable operation of FCHEVs with enhanced performance and reduced cost.

Identifying the Automated Vehicle's Driving Policy in the Vicinity of Pedestrians

The era of automation has already been launched in the field of transportation, expected to increase road capacity and safety levels by reducing and eliminating crashes while the environmental impacts are also anticipated to be positive. Various studies have tried to analyze the behaviour of automated vehicles and their interaction with the surrounding traffic while concerning pedestrians, research is still limited. The present work aims to enrich existing research by modelling the behaviour of an automated vehicle when it interacts with a pedestrian with the intention to cross the road. For this purpose, vehicle and pedestrian trajectories from a virtual experiment are analyzed and the principles of inverse reinforcement learning are used for developing the model. The results are further discussed along with suggestions for future work.

A Mini review on Effectiveness of Reinforcement learning in Stroke Rehabilitation

This review focuses on the Development of Reinforcement based therapies in the field of Rehabilitation of Post -Stroke Patients. The process through which a system's (or organism's) reaction to a stimulus is increased by reward and diminished by punishment (Positive and Negative Reinforcement, respectively) is referred to as reinforcement learning. [1] A plausible set of methods for recreating this reward distribution from experience has been discovered following recent breakthroughs in machine learning. Integration of Reinforcement learning through a Physiotherapeutic perspective is presented in this review. The Brain-Machine interface is a relatively newer as well as potential field which can be used for stroke rehabilitation based on the principles of reinforcement learning which is a subset of Machine Learning.

Effort Reinforces Learning.

Humans routinely learn the value of actions by updating their expectations based on past outcomes - a process driven by reward prediction errors (RPEs). Importantly, however, implementing a course of action also requires the investment of effort. Recent work has revealed a close link between the neural signals involved in effort exertion and those underpinning reward-based learning, but the behavioral relationship between these two functions remains unclear. Across two experiments, we tested healthy male and female human participants (N = 140) on a reinforcement learning task in which they registered their responses by applying physical force to a pair of hand-held dynamometers. We examined the effect of effort on learning by systematically manipulating the amount of force required to register a response during the task. Our key finding, replicated across both experiments, was that greater effort increased learning rates following positive outcomes and decreased them following negative outcomes, which corresponded to a differential effect of effort in boosting positive RPEs and blunting negative RPEs. Interestingly, this effect was most pronounced in individuals who were more averse to effort in the first place, raising the possibility that the investment of effort may have an adaptive effect on learning in those less motivated to exert it. By integrating principles of reinforcement learning with neuroeconomic approaches to value-based decision-making, we show that the very act of investing effort modulates one's capacity to learn, and demonstrate how these functions may operate within a common computational framework.SIGNIFICANCE STATEMENT Recent work suggests that learning and effort may share common neurophysiological substrates. This raises the possibility that the very act of investing effort influences learning. Here, we tested whether effort modulates teaching signals in a reinforcement learning paradigm. Our results showed that effort resulted in more efficient learning from positive outcomes and less efficient learning from negative outcomes. Interestingly, this effect varied across individuals, and was more pronounced in those who were more averse to investing effort in the first place. These data highlight the importance of motivational factors in a common framework of reward-based learning, which integrates the computational principles of reinforcement learning with those of value-based decision-making.

Reinforcement learning method for target hunting control of multi‐robot systems with obstacles

Aiming at the target encirclement problem of multi-robot systems, a target hunting control method based on reinforcement learning is proposed. First, the Markov game modeling for the multi-robot system is carried out. According to the task of hunting, potential energy models are designed to meet the requirements of arriving at the desired state and avoiding obstacles. The multi-robot reinforcement learning algorithm guided by the potential energy models is presented to perform the hunting, where reinforcement learning principles are combined with the model control. Secondly, based on the potential energy models, the target-tracking hunting strategy and the target-circumnavigation hunting strategy are established. In the former, the consensus tracking of multi-robot systems is achieved by designing the velocity potential energy function. And in the latter, virtual circumnavigation points are added to construct the potential energy function, which realizes the desired circumnavigation. Finally, the effectiveness of target hunting control based on the multi-robot reinforcement learning method is verified by simulation.

An improved marine predators algorithm for the optimal design of hybrid renewable energy systems

Microgrid technologies are exciting energy sources that are economically feasible for current and future applications in light of increased energy demand and the depletion of traditional sources. This article focuses on the latest metaheuristic algorithm, the marine predators algorithm (MPA), in the field of energy. To investigate a method for reducing the system’s investment costs in Minia, Egypt, the combination of reinforcement learning (RL) with MPA is used to build a new method, Deep-MPA, where RL principles are applied to adjust and enhance the lack of MPA in global searching. The exploration/exploitation ratio is regulated by varying the step size, which affects the efficiency of the MPA. Instead of updating the value of the parameter for all agents in the same manner, RL principles are used to update it on the basis of the current individual state. Additionally, to resolve the common challenge of using RL to determine the appropriate global search parameters for MPA, Deep-MPA is used to design a hybrid renewable energy microgrid system, which includes photovoltaic panels, a wind turbine system, a diesel generator, and battery storage systems. These are some of the criteria and constraints that this system may require to ensure its stability, robustness, performance, and load satisfaction. The proposed Deep-MPA is verified by contrasting the results with different algorithms in the CEC’2017 test. Moreover, Wilcoxon’s test validates the statistical significance of the Deep-MPA. The energy cost is reduced by 6% of total consumption.

5GhNet: an intelligent QoE aware RAT selection framework for 5G-enabled healthcare network.

The COVID-19 outbreak has stimulated the digital transformation of antiquated healthcare system to a smart hospital, enabling the personalised and remote healthcare services. To augment the functionalities of these intelligent healthcare systems, 5G & B5G heterogeneous network has emerged as a robust and reliable solution. But the pivotal challenge for 5G & B5G connectivity solutions is to ensure flexible and agile service orchestration with acknowledged Quality of Experience (QoE). However, the existing radio access technology (RAT) selection strategies are incapacitated in terms of QoE provisioning and Quality of Service (QoS) maintenance. Therefore, an intelligent QoE aware RAT selection architecture based on software-defined wireless networking (SDWN) and edge computing has been proposed for 5G-enabled healthcare network. The proposed model leverages the principles of invalid action masking and multi-agent reinforcement learning to allow faster convergence to QoE optimised RAT selection policy. The analytical evaluation validates that the proposed scheme outperforms the other existing schemes in terms of enhancing personalised user-experience with efficient resource utilisation.

A HYBRID ALGORITHM FOR FORMING THE SHORTEST TRAJECTORY BASED ON THE APPLICATION OF MULTI-AGENT LEARNING WITH REINFORCEMENT, THE SEARCH ALGORITHM A* AND EXCHANGE OF EXPERIENCE

The article presents a hybrid algorithm for the formation of the shortest trajectory for intelligent agents of a multi-agent system, based on the synthesis of methods of the reinforcement learning paradigm, the heuristic search algorithm A*, which has the functions of exchange of experience, as well as the automatic formation of subgroups of agents based on their visibility areas. The experimental evaluation of the developed algorithm was carried out by simulating the task of finding the target state in the maze in the Microsoft Unity environment. The results of the experiment showed that the use of the developed hybrid algorithm made it possible to reduce the time for solving the problem by an average of 12.7 % in comparison with analogs. The differences between the proposed new “hybrid algorithm for the formation of the shortest trajectory based on the use of multi-agent reinforcement learning, search algorithm A* and exchange of experience” from analogs are as follows: – application of the algorithm for the formation of subgroups of subordinate agents based on the “scope” of the leader agent for the implementation of a multi-level hierarchical system for managing a group of agents; – combining the principles of reinforcement learning and the search algorithm A*.

Language statistical learning responds to reinforcement learning principles rooted in the striatum.

Statistical learning (SL) is the ability to extract regularities from the environment. In the domain of language, this ability is fundamental in the learning of words and structural rules. In lack of reliable online measures, statistical word and rule learning have been primarily investigated using offline (post-familiarization) tests, which gives limited insights into the dynamics of SL and its neural basis. Here, we capitalize on a novel task that tracks the online SL of simple syntactic structures combined with computational modeling to show that online SL responds to reinforcement learning principles rooted in striatal function. Specifically, we demonstrate-on 2 different cohorts-that a temporal difference model, which relies on prediction errors, accounts for participants' online learning behavior. We then show that the trial-by-trial development of predictions through learning strongly correlates with activity in both ventral and dorsal striatum. Our results thus provide a detailed mechanistic account of language-related SL and an explanation for the oft-cited implication of the striatum in SL tasks. This work, therefore, bridges the long-standing gap between language learning and reinforcement learning phenomena.

Reinforcement learning about asset variability and correlation in repeated portfolio decisions

Lay investors construct portfolios that are often not efficient and fail to take the correlation of assets into account. The present work examines whether providing people with a learning opportunity makes them sensitive to the correlation between assets. In two studies, where participants repeatedly allocated their endowment to three assets with feedback, participants changed their portfolio over time dependent on the asset correlation. To model learning about relevant characteristics of a portfolio, we developed reinforcement learning models that take learning about asset variability and correlation into account. We demonstrated via out-of-sample predictions that these models explain portfolio allocations better than basic reinforcement learning models, a static 1/N diversification strategy, and the mean–variance model using sample means, variances, and correlation. Hence, experiencing returns can help investors take asset correlations into account. The principles of reinforcement learning are a cognitively plausible and descriptively valid framework for understanding repeated portfolio allocations.