Final Reward Research Articles

Some Probabilistic Interpretations Related to the Next-Generation Matrix Theory: A Review with Examples

The fact that the famous basic reproduction number R0, i.e., the largest eigenvalue of the next generation matrix FV−1, sometimes has a probabilistic interpretation is not as well known as it deserves to be. It is well understood that half of this formula, −V, is a Markovian generating matrix of a continuous-time Markov chain (CTMC) modeling the evolution of one individual on the compartments. It has also been noted that the not well-enough-known rank-one formula for R0 of Arino et al. (2007) may be interpreted as an expected final reward of a CTMC, whose initial distribution is specified by the rank-one factorization of F. Here, we show that for a large class of ODE epidemic models introduced in Avram et al. (2023), besides the rank-one formula, we may also provide an integral renewal representation of R0 with respect to explicit “age kernels” a(t), which have a matrix exponential form.This latter formula may be also interpreted as an expected reward of a probabilistic continuous Markov chain (CTMC) model. Besides the rather extensively studied rank one case, we also provide an extension to a case with several susceptible classes.

Local connection reinforcement learning method for efficient robotic peg-in-hole assembly

Traditional robotic peg-in-hole assembly methods rely on complex contact state analysis. Reinforcement learning (RL) is gradually becoming a preferred method of controlling robotic peg-in-hole assembly tasks. However, the training process of RL is quite time-consuming because RL methods are always globally connected, which means all state components are assumed to be the input of policies for all action components, thus increasing action space and state space to be explored. In this paper, we first define continuous space serialized Shapley value (CS3) and construct a connection graph to clarify the correlativity of action components on state components. Then we propose a local connection reinforcement learning (LCRL) method based on the connection graph, which eliminates the influence of irrelevant state components on the selection of action components. The simulation and experiment results demonstrate that the LCRL method can achieve the same average reward as the traditional RL method in only 49% of episodes. In the final episode, the LCRL method's final reward is 35% higher than that of the traditional RL method, which guarantees the rapidity and stability of the assembly process.

USN: A Robust Imitation Learning Method against Diverse Action Noise

Learning from imperfect demonstrations is a crucial challenge in imitation learning (IL). Unlike existing works that still rely on the enormous effort of expert demonstrators, we consider a more cost-effective option for obtaining a large number of demonstrations. That is, hire annotators to label actions for existing image records in realistic scenarios. However, action noise can occur when annotators are not domain experts or encounter confusing states. In this work, we introduce two particular forms of action noise, i.e., state-independent and state-dependent action noise. Previous IL methods fail to achieve expert-level performance when the demonstrations contain action noise, especially the state-dependent action noise. To mitigate the harmful effects of action noises, we propose a robust learning paradigm called USN (Uncertainty-aware Sample-selection with Negative learning). The model first estimates the predictive uncertainty for all demonstration data and then selects sampleswith high loss based on the uncertainty measures. Finally, it updates the model parameters with additional negative learning on the selected samples. Empirical results in Box2D tasks and Atari games show that USN consistently improves the final rewards of behavioral cloning, online imitation learning, and offline imitation learning methods under various action noises. The ratio of significant improvements is up to 94.44%. Moreover, our method scales to conditional imitation learning with real-world noisy commands in urban driving

Reinforcement learning for intensive care medicine: actionable clinical insights from novel approaches to reward shaping and off-policy model evaluation

BackgroundReinforcement learning (RL) holds great promise for intensive care medicine given the abundant availability of data and frequent sequential decision-making. But despite the emergence of promising algorithms, RL driven bedside clinical decision support is still far from reality. Major challenges include trust and safety. To help address these issues, we introduce cross off-policy evaluation and policy restriction and show how detailed policy analysis may increase clinical interpretability. As an example, we apply these in the setting of RL to optimise ventilator settings in intubated covid-19 patients.MethodsWith data from the Dutch ICU Data Warehouse and using an exhaustive hyperparameter grid search, we identified an optimal set of Dueling Double-Deep Q Network RL models. The state space comprised ventilator, medication, and clinical data. The action space focused on positive end-expiratory pressure (peep) and fraction of inspired oxygen (FiO2) concentration. We used gas exchange indices as interim rewards, and mortality and state duration as final rewards. We designed a novel evaluation method called cross off-policy evaluation (OPE) to assess the efficacy of models under varying weightings between the interim and terminal reward components. In addition, we implemented policy restriction to prevent potentially hazardous model actions. We introduce delta-Q to compare physician versus policy action quality and in-depth policy inspection using visualisations.ResultsWe created trajectories for 1118 intensive care unit (ICU) admissions and trained 69,120 models using 8 model architectures with 128 hyperparameter combinations. For each model, policy restrictions were applied. In the first evaluation step, 17,182/138,240 policies had good performance, but cross-OPE revealed suboptimal performance for 44% of those by varying the reward function used for evaluation. Clinical policy inspection facilitated assessment of action decisions for individual patients, including identification of action space regions that may benefit most from optimisation.ConclusionCross-OPE can serve as a robust evaluation framework for safe RL model implementation by identifying policies with good generalisability. Policy restriction helps prevent potentially unsafe model recommendations. Finally, the novel delta-Q metric can be used to operationalise RL models in clinical practice. Our findings offer a promising pathway towards application of RL in intensive care medicine and beyond.

Embedding multi-agent reinforcement learning into behavior trees with unexpected interruptions

Behavior trees have attracted great interest in computer games and robotic applications. However, it lacks the learning ability for dynamic environments. Previous works combining behavior trees with reinforcement learning either need to construct an independent sub-scenario or train the learning method over the whole game, which is not suited for complex multi-agent games. In this paper, a framework is proposed, named as MARL-BT, that embeds multi-agent reinforcement learning methods into behavior trees. Following the running mechanism of behavior trees, we design the way of collecting samples and the training procedure. Further, we point out a special phenomenon in MARL-BT, i.e., the unexpected interruption, and present an action masking technique to remove its harmful effect on learning performance. Finally, we make extensive experiments on the 11 versus 11 full game in Google Research Football. The introduced MARL-BT framework could get an 11.507% improvement compared to pure BT for certain scenarios. The action masking technique could greatly improve the performance of the learning method, i.e., the final reward is improved around 100% times for a sub-task.

Nash equilibria in risk-sensitive Markov stopping games under communication conditions

<p>This paper analyzes the existence of Nash equilibrium in a discrete-time Markov stopping game with two players. At each decision point, Player Ⅱ is faced with the choice of either ending the game and thus granting Player Ⅰ a final reward or letting the game continue. In the latter case, Player Ⅰ performs an action that affects transitions and receives a running reward from Player Ⅱ. We assume that Player Ⅰ has a constant and non-zero risk sensitivity coefficient, while Player Ⅱ strives to minimize the utility of Player Ⅰ. The effectiveness of decision strategies was measured by the risk-sensitive expected total reward of Player Ⅰ. Exploiting mild continuity-compactness conditions and communication-ergodicity properties, we found that the value function of the game is described as a single fixed point of the equilibrium operator, determining a Nash equilibrium. In addition, we provide an illustrative example in which our assumptions hold.</p>

Reinforcement learning as data-driven optimization technique for GMAW process

Welding optimization is a significant task that contributes to enhancing the final welding quality. However, the selection of an optimal combination of various process parameters poses different challenges. The welding geometry and quality are influenced differently by several process parameters, with some exhibiting opposite effects. Consequently, multiple experiments are typically required to obtain an optimal welding procedure specification (WPS), resulting in the waste of material and costs. To address this challenge, we developed a machine learning model that correlates the process parameters with the final bead geometry, utilizing experimental data. Additionally, we employed a reinforcement learning algorithm, namely stochastic policy optimization (SPO), with the aim to solve different optimization tasks. The first task is a setpoint‐based optimization problem that aims to find the process parameters that minimize the amount of deposited material while achieving the desired minimum level of penetration depth. The second task is an optimization problem without setpoint in which the agent aims to maximize the penetration depth and reduce the bead area. The proposed artificial intelligence-based method offers a viable means of reducing the number of experiments necessary to develop a WPS, consequently reducing costs and emissions. Notably, the proposed approach achieves better results with respect to other state-of-art metaheuristic data-driven optimization methods such as genetic algorithm. In particular, the setpoint‐based optimization problem is solved in 8 min and with a final mean percentage absolute error (MPAE) of 2.48% with respect to the 42 min and the final 3.42% of the genetic algorithm. The second optimization problem is also solved in less time, 30 s with respect to 6 min of GA, with a higher final reward of 5.8 from the proposed SPO algorithm with respect to the 3.6 obtained from GA.

Dynamic expansions of social followings with lotteries and give-aways

The problem of how to attract a robust following on social media is one of the most pressing for influencers. We study a common practice on popular microblogging platforms such as Twitter, of influencers’ expanding their followings by running lotteries and giveaways. We are interested in how the lottery size and the seeding decisions influence the information propagation and the final reward for such a campaign. We construct an information-diffusion model based on a random graph, and show that the market demand curve of the lottery reward via the promotion of the social network is “S”-shaped. This property lays a foundation for finding the optimal lottery size. Second, we observe that (i) dynamic seeding could re-stimulate the spread of information and (ii) with a fixed budget, seeding at two fixed occasions is always better than seeding once at the beginning. This observation motivates us to study the joint optimization of lottery size and adaptive seeding. We model the adaptive seeding problem as a Markov Decision Process. We find the monotonicity of the value functions and trends in the optimal actions, and we show that with adaptive seeding, the reward curve is approximately “S”-shaped with respect to the lottery size.

Online Trajectory Planning Method for Midcourse Guidance Phase Based on Deep Reinforcement Learning

Concerned with the problem of interceptor midcourse guidance trajectory online planning satisfying multiple constraints, an online midcourse guidance trajectory planning method based on deep reinforcement learning (DRL) is proposed. The Markov decision process (MDP) corresponding to the background of a trajectory planning problem is designed, and the key reward function is composed of the final reward and the negative step feedback reward, which lays the foundation for the interceptor training trajectory planning method in the interactive data of a simulation environment; at the same time, concerned with the problems of unstable learning and training efficiency, a trajectory planning training strategy combined with course learning (CL) and deep deterministic policy gradient (DDPG) is proposed to realize the progressive progression of trajectory planning learning and training from satisfying simple objectives to complex objectives, and improve the convergence of the algorithm. The simulation results show that our method can not only generate the optimal trajectory with good results, but its trajectory generation speed is also more than 10 times faster than the hp pseudo spectral convex method (PSC), and can also resist the error influence mainly caused by random wind interference, which has certain application value and good research prospects.

A reinforcement learning-based approach for online bus scheduling

Bus Scheduling Problem (BSP) is vital to save operational cost and ensure service quality. Existing approaches typically generate a bus scheduling scheme in an offline manner and then schedule vehicles according to the scheme. In practice, uncertain events such as traffic congestion occur frequently, which may make the originally planned bus scheduling scheme infeasible. This study proposes a Reinforcement Learning-based Bus Scheduling Approach (RL-BSA) for online bus scheduling. In RL-BSA, each departure time in a bus timetable is regarded as a decision point, and an agent makes a decision at the departure time to select a vehicle to depart at the time. The BSP is modeled as a Markov Decision Process (MDP) for the first time in literature. The state features are devised, which consist of real-time information of vehicles, including remaining working time, remaining driving time, rest time, number of executed trips and vehicle type. A reward function combining a final reward and a step-wise reward is devised. An invalid action masking approach is used to avoid the agent from selecting vehicles not meeting constraints. The agent is trained by interacting with a simulation environment and then the trained agent can schedule vehicles in an online manner. Experiments on real-world BSP instances show that RL-BSA can significantly reduce the number of vehicles used compared with the manual scheduling approach and Adaptive Large Neighborhood Search (ALNS). Under uncertain environment, RL-BSA can cover all departure times in the timetable without increasing the number of vehicles used.

Hidden Otolaryngology Malpractice Claims: An Internal Review From 2000 to 2020.

To report key characteristics and patterns of captive insurance claims not publicly reported in otolaryngology across a large tertiary-level academic health system over the previous 2 decades. Case series. The tertiary care health system. The internal captive insurance database at a tertiary level healthcare system was queried to identify otolaryngology-related malpractice claims regardless of final disposition (settled or dismissed) filed from 2000 to 2020. The date of the incident, date of claim, error type, patient outcome, provider subspecialty, total expenses, disposition, and final reward amount were recorded. Twenty-eight claims were identified. There were 11 (39.3%) claims from 2000 to 2010 and 17 (60.7%) claims from 2011 to 2020. Head and neck surgery was the most frequently implicated subspecialty (n = 9, 32.1% of all cases), followed by general otolaryngology (n = 7, 25.0%), pediatrics (n = 5, 17.9%), skull base/rhinology (n = 4, 14.3%), and laryngology (n = 1, 3.6%). Improper surgical performance was cited in 35.7% of cases (n = 10), followed by failure to diagnose (n = 8, 28.6%), to treat (n = 4, 14.3%), and to obtain informed consent (n = 3, 10.7%). While 2 cases are ongoing, a total of 17/26 (65.4%) cases were settled and 20/26 (76.9%) dismissed some or all parties. Dismissed claims had significantly higher expenses (p = .022) and duration from incident to disposition (p = .013) compared to settled claims. This study expands the malpractice landscape in otolaryngology by including data not readily available through public sources and compares it to national trends. These findings encourage otolaryngologists to better gauge current quality and safety measures that best protect patients from harm.

Resetting in stochastic optimal control

``When in a difficult situation, it is sometimes better to give up and start all over again.'' While this empirical truth has been regularly observed in a wide range of circumstances, quantifying the effectiveness of such a heuristic strategy remains an open challenge. In this paper, we combine the notions of optimal control and stochastic resetting to address this problem. The emerging analytical framework allows one not only to measure the performance of a given restarting policy, but also to obtain the optimal strategy for a wide class of dynamical systems. We apply our technique to a system with a final reward and show that the reward value must be larger than a critical threshold for resetting to be effective. Our approach, analogous to the celebrated Hamilton-Jacobi-Bellman paradigm, provides the basis for the investigation of realistic restarting strategies across disciplines. As an application, we show that the framework can be applied to an epidemic model to predict the optimal lockdown policy.

Offsetting Unequal Competition Through RL-Assisted Incentive Schemes

This article investigates the dynamics of competition among organizations with unequal expertise. Multiagent reinforcement learning (MARL) has been used to simulate and understand the impact of various incentive schemes designed to offset such inequality. We design Touch-Mark, a game based on well-known multiagent particle environment, where two teams (weak and strong) with unequal but changing skill levels compete against each other. For training such a game, we propose a novel controller-assisted MARL algorithm C-MADDPG, which empowers each agent with an ensemble of policies along with a supervised controller that by selectively partitioning the sample space and triggers intelligent role division among the teammates. Using C-MADDPG as an underlying framework, we propose an incentive scheme for the weak team such that the final rewards of both teams become the same. We find that despite the incentive, the final reward of the weak team falls short of the strong team. On inspecting, we realize that an overall incentive scheme for the weak team does not incentivize the weaker agents within that team to learn and improve. To offset this, we now specially incentivize the weaker player to learn and, as a result, observe that the weak team beyond an initial phase performs at par with the stronger team. The final goal of this article has been to formulate a dynamic incentive scheme that continuously balances the reward of the two teams. This is achieved by devising an incentive scheme enriched with an RL agent, which takes minimum information from the environment.

An enhanced searching strategy for multi-agent mobile applications

Multi-agent mobile applications play an essential role in mobile applications and have attracted more and more researchers' attention. Previous work has always focused on multi-agent applications with perfect information. Researchers are usually based on human-designed rules to provide decision-making searching services. However, existing methods for solving perfect-information mobile applications cannot be directly applied to imperfect-information mobile applications. Here, we take the Contact Bridge, a multi-agent application with imperfect information, for the case study. We propose an enhanced searching strategy to deal with multi-agent applications with imperfect information. We design a self-training bidding system model and apply a Recurrent Neural Network (RNN) to model the bidding process. The bridge system model consists of two parts, a bidding prediction system based on imitation learning to get a contract quickly and a visualization system for hands understanding to realize regular communication between players. Then, to dynamically analyze the impact of other players' unknown hands on our final reward, we design a Monte Carlo sampling algorithm based on the bidding system model (BSM) to deal with imperfect information. At the same time, a double-dummy analysis model is designed to efficiently evaluate the results of sampling. Experimental results indicate that our searching strategy outperforms the top rule-based mobile applications.

Autonomous Maneuver Decision of Air Combat Based on Simulated Operation Command and FRV-DDPG Algorithm

With the improvement of UAV performance and intelligence in recent years, it is particularly important for unmanned aerial vehicles (UAVs) to improve the ability of autonomous air combat. Aiming to solve the problem of how to improve the autonomous air combat maneuver decision ability of UAVs so that it can be close to manual manipulation, this paper proposes an autonomous air combat maneuvering decision method based on the combination of simulated operation command and the final reward value deep deterministic policy gradient (FRV-DDPG) algorithm. Firstly, the six-degree-of-freedom (6-DOF) model is established based on the air combat process, UAV motion, and missile motion. Secondly, a prediction method based on the Particle swarm optimization radial basis function (PSO-RBF) is designed to simulate the operation command of the enemy aircraft, which makes the training process more realistic, and then an improved DDPG strategy is proposed, which returns the final reward value to the previous reward value in a certain proportion of time for offline training, which can improve the convergence speed of the algorithm. Finally, the effectiveness of the algorithm is verified by building a simulation environment. The simulation results show that the algorithm can improve the autonomous air combat maneuver decision-making ability of UAVs.

DRL-M4MR: An intelligent multicast routing approach based on DQN deep reinforcement learning in SDN

Traditional multicast routing methods have some problems in constructing a multicast tree. These problems include limited access to network state information, poor adaptability to dynamic and complex changes in the network, and inflexible data forwarding. To address these defects, the optimal multicast routing problem in software-defined networking (SDN) is tailored as a multiobjective optimization problem, and DRL-M4MR, an intelligent multicast routing algorithm based on the deep Q network (DQN) deep reinforcement learning (DRL) method is designed to construct a multicast tree in a software-defined network. First, combining the characteristics of SDN global network-aware information, the multicast tree state matrix, link bandwidth matrix, link delay matrix and link packet loss rate matrix are designed as the state space of the reinforcement learning agent to solve the problem in that the original method cannot make full use of network status information. Second, the action space of the agent is all the links in the network, and the action selection strategy is designed to add the links to the current multicast tree in four cases. Third, single-step and final reward function forms are designed to guide the agent to make decisions to construct the optimal multicast tree. The double network architectures, dueling network architectures and prioritized experience replay are adopted to improve the learning efficiency and convergence of the agent. Finally, after the DRL-M4MR agent is trained, the SDN controller installs the multicast flow entries by reversely traversing the multicast tree to the SDN switches to implement intelligent multicast routing. The experimental results show that, compared with existing algorithms, the multicast tree constructed by DRL-M4MR can obtain better bandwidth, delay, and packet loss rate performance after training, and it can make more intelligent multicast routing decisions in a dynamic network environment. Code and DRL model are available at https://github.com/GuetYe/DRL-M4MR.

Maneuver Decision-Making for Autonomous Air Combat Based on FRE-PPO

Maneuver decision-making is the core of autonomous air combat, and reinforcement learning is a potential and ideal approach for addressing decision-making problems. However, when reinforcement learning is used for maneuver decision-making for autonomous air combat, it often suffers from awful training efficiency and poor performance of maneuver decision-making. In this paper, an air combat maneuver decision-making method based on final reward estimation and proximal policy optimization is proposed to solve the above problems. First, an air combat environment based on aircraft and missile models is constructed, and an intermediate reward and final reward are designed. Second, the final reward estimation is proposed to replace the original advantage estimation function of the surrogate objective of proximal policy optimization to improve the training performance of reinforcement learning. Third, sampling according to the final reward estimation is proposed to improve the training efficiency. Finally, the proposed method is used in a self-play framework to train agents for maneuver decision-making. Simulations show that final reward estimation and sampling according to final reward estimation are effective and efficient.

Analysis of Model-Free Reinforcement Learning Algorithm for Target Tracking

Target tracking is a process that can find points in different domains. In tracking, some places contain prizes (positive or negative values) that the agent does not know at first. Therefore, the agent, which is a system, must learn to get the maximum value with various learning rates. Reinforcement learning is a machine learning technique in which agents learn through interaction with the environment using reward functions and probabilistic dynamics to allow agents to explore and learn about the environment through various iterations. Thus, for each action taken, the agent receives a reward from the environment, which determines positive or negative behavior. The agent's goal is to maximize the total reward received during the interaction. In this case, the agent will study three different modules, namely sidewalk, obstacle, and product, using the Q-learning algorithm. Each module will be training with various learning rates and rewards. Q-learning can work effectively with the highest final reward at a learning rate of 0.8 for 500 rounds with an epsilon of 0.9.

Emergency load-shedding optimization control method based on reinforcement learning assistance

At present, on-line pre-decision and real-time matching control forms are the most widely used emergency control forms in the power system. The safety control system predicts the accidents according to the current operation state of the power grid and sets the emergency control strategy for the predicted accidents to ensure the safety and stability of the power grid when the fault occurs. However, the voltage level controlled by the safety control system is relatively high, which leads to a large granularity of load control. Because load nodes are mainly mixed users of various types, the strategy under large-scale centralized control may include important load nodes. It is necessary to replace the important load with other controllable low voltage nodes in the area to ensure that the load-shedding quantity set by the safety control system can be implemented sufficiently. Most emergency control optimization problems regard the load-shedding cost as a definite and constant value, but in fact, the load-shedding cost will change with the environmental factors. Based on the analysis of the influence of environmental factors on the cost of load shedding, this paper uses the reinforcement learning method to learn historical operation data and adapt to the variability of load-shedding cost to decide the best alternatives. The results of an example show that the proposed method can make replacement decisions in any scene based on adaptive environmental variability and can effectively prevent over-cutting through the design of a precise final reward.

Intermediate Sensory Feedback Assisted Multi-Step Neural Decoding for Reinforcement Learning Based Brain-Machine Interfaces.

Reinforcement-learning (RL)-based brain-machine interfaces (BMIs) interpret dynamic neural activity into movement intention without patients' real limb movements, which is promising for clinical applications. A movement task generally requires the subjects to reach the target within one step and rewards the subjects instantaneously. However, a real BMI scenario involves tasks that require multiple steps, during which sensory feedback is provided to indicate the status of the prosthesis, and the reward is only given at the end of the trial. Actually, subjects internally evaluate the sensory feedback to adjust motor activity. Existing RL-BMI tasks have not fully utilized the internal evaluation from the brain upon the sensory feedback to guide the decoder training, and there lacks an effective tool to assign credit for the multi-step decoding task. We propose first to extract intermediate guidance from the medial prefrontal cortex (mPFC) to assist the learning of multi-step decoding in an RL framework. To effectively explore the neural-action mapping in a large state-action space, a temporal difference (TD) method is incorporated into quantized attention-gated kernel reinforcement learning (QAGKRL) to assign the credit over the temporal sequence of movement, but also discriminate spatially in the Reproducing Kernel Hilbert Space (RKHS). We test our approach on the data collected from the primary motor cortex (M1) and the mPFC of rats when they brain control the cursor to reach the target within multiple steps. Compared with the models which only utilize the final reward, the intermediate evaluation interpreted from the mPFC can help improve the prediction accuracy by 10.9% on average across subjects, with faster convergence and more stability. Moreover, our proposed algorithm further increases 18.2% decoding accuracy compared with existing TD-RL methods. The results reveal the possibility of achieving better multi-step decoding performance for more complicated BMI tasks.