Framework For Sequential Decision Research Articles

A Review of Degradation Models and Remaining Useful Life Prediction for Testing Design and Predictive Maintenance of Lithium-Ion Batteries.

We present a novel decision-making framework for accelerated degradation tests and predictive maintenance that exploits prior knowledge and experimental data on the system's state. As a framework for sequential decision making in these areas, dynamic programming and reinforcement learning are considered, along with data-driven degradation learning when necessary. Furthermore, we illustrate both stochastic and machine learning degradation models, which are integrated in the framework, using data-driven methods. These methods are presented as a valuable tool for designing life-testing experiments and for maintaining lithium-ion batteries.

New Approach to Equitable Intervention Planning to Improve Engagement and Outcomes in a Digital Health Program: Simulation Study.

Digital health programs provide individualized support to patients with chronic diseases and their effectiveness is measured by the extent to which patients achieve target individual clinical outcomes and the program's ability to sustain patient engagement. However, patient dropout and inequitable intervention delivery strategies, which may unintentionally penalize certain patient subgroups, represent challenges to maximizing effectiveness. Therefore, methodologies that optimize the balance between success factors (achievement of target clinical outcomes and sustained engagement) equitably would be desirable, particularly when there are resource constraints. Our objectives were to propose a model for digital health program resource management that accounts jointly for the interaction between individual clinical outcomes and patient engagement, ensures equitable allocation as well as allows for capacity planning, and conducts extensive simulations using publicly available data on type 2 diabetes, a chronic disease. We propose a restless multiarmed bandit (RMAB) model to plan interventions that jointly optimize long-term engagement and individual clinical outcomes (in this case measured as the achievement of target healthy glucose levels). To mitigate the tendency of RMAB to achieve good aggregate performance by exacerbating disparities between groups, we propose new equitable objectives for RMAB and apply bilevel optimization algorithms to solve them. We formulated a model for the joint evolution of patient engagement and individual clinical outcome trajectory to capture the key dynamics of interest in digital chronic disease management programs. In simulation exercises, our optimized intervention policies lead to up to 10% more patients reaching healthy glucose levels after 12 months, with a 10% reduction in dropout compared to standard-of-care baselines. Further, our new equitable policies reduce the mean absolute difference of engagement and health outcomes across 6 demographic groups by up to 85% compared to the state-of-the-art. Planning digital health interventions with individual clinical outcome objectives and long-term engagement dynamics as considerations can be both feasible and effective. We propose using an RMAB sequential decision-making framework, which may offer additional capabilities in capacity planning as well. The integration of an equitable RMAB algorithm further enhances the potential for reaching equitable solutions. This approach provides program designers with the flexibility to switch between different priorities and balance trade-offs across various objectives according to their preferences.

POMDP-BCI: A Benchmark of (Re)Active BCI Using POMDP to Issue Commands.

Past research in Brain-Computer Interfaces (BCI) have presented different decoding algorithms for different modalities. Meanwhile, highly specific decision making processes have been developed for some of these modalities, while others lack such a component in their classic pipeline. The present work proposes a model based on Partially Observable Markov Decission Process (POMDP) that works as a high-level decision making framework for three different active/reactive BCI modalities. We tested our approach on three different BCI modalities using publicly available datasets. We compared the general POMDP model as a decision making process with state of the art methods for each BCI modality. Accuracy, false positive (FP) trials, no-action (NA) trials and average decision time are presented as metrics. Our results show how the presented POMDP models achieve comparable or better performance to the presented baseline methods, while being usable for the three proposed experiments without significant changes. Crucially, it offers the possibility of taking no-action (NA) when the decoding does not perform well. The present work implements a flexible POMDP model that acts as a sequential decision framework for BCI systems that lack such a component, and perform comparably to those that include it. We believe the proposed POMDP framework provides several interesting properties for future BCI developments, mainly the generalizability to any BCI modality and the possible integration of other physiological or brain data pipelines under a unified decision-making framework.

Trust in Robot–Robot Scaffolding

The study of robot trust in humans and other agents is not explored widely despite its importance for the near future human-robot symbiotic societies. Here we propose that robots should trust partners that tend to reduce their computational load, which is analogous to human cognitive load. We test this idea by adopting an interactive visual recalling task. In the first set of experiments, the robot can get help from online instructors with different guiding strategies to decide which one it should trust based on the computational load it experiences during the experiments. The second set of experiments involves robot-robot interactions. Akin to the robot-online instructor case, the Pepper robot is asked to scaffold the learning of a less capable ‘infant’ robot (Nao) with or without being equipped with the cognitive abilities of theory of mind and task experience memory to assess the contribution of these cognitive abilities to scaffolding performance. Overall, the results show that robot trust based on computational/cognitive load within a sequential decision-making framework leads to effective partner selection and robot-robot scaffolding. Thus, using the computational load incurred by the cognitive processing of a robot may serve as an internal signal for assessing the trustworthiness of interaction partners.

Structural Estimation of Partially Observable Markov Decision Processes

Partially Observable Markov Decision Processes (POMDPs) is a well-developed framework for sequential decision making under uncertainty and partial information. This paper considers the (inverse) structural estimation of the primitives of a POMDP based upon data in the form of sequences of observables and implemented actions. We analyze the structural properties of an entropy regularized POMDP and specify conditions under which the model is identifiable without knowledge of the state dynamics. We consider a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">soft</i> policy gradient algorithm to compute a maximum likelihood estimator, and illustrate the approach with an equipment replacement problem.

Generalized Contextual Bandits With Latent Features: Algorithms and Applications.

Contextual bandit is a popular sequential decision-making framework to balance the exploration and exploitation tradeoff in many applications such as recommender systems, search engines, etc. Motivated by two important factors in real-world applications: 1) latent contexts (or features) often exist and 2) feedbacks often have humans in the loop leading to human biases, we formulate a generalized contextual bandit framework with latent contexts. Our proposed framework includes a two-layer probabilistic interpretable model for the feedbacks from human with latent features. We design a GCL-PS algorithm for the proposed framework, which utilizes posterior sampling to balance the exploration and exploitation tradeoff. We prove a sublinear regret upper bound for GCL-PS, and prove a lower bound for the proposed bandit framework revealing insights on the optimality of GCL-PS. To further improve the computational efficiency of GCL-PS, we propose a Markov Chain Monte Carlo (MCMC) algorithm to generate approximate samples, resulting in our GCL-PSMC algorithm. We not only prove a sublinear Bayesian regret upper bound for our GCL-PSMC algorithm, but also reveal insights into the tradeoff between computational efficiency and sequential decision accuracy. Finally, we apply the proposed framework to hotel recommendations and news article recommendations, and show its superior performance over a variety of baselines via experiments on two public datasets.

Transfer learning-based cost function approximation for look-ahead dispatch of power systems

In this paper, a transfer learning-based cost function approximation (TL-CFA) algorithm is proposed for the look-ahead dispatch of power systems. Cost function approximation (CFA) provides an effective sequential decision framework in the stochastic environment, which is suitable for the look-ahead dispatch of power systems with uncertainties. However, the parameters of CFA need to be iteratively updated, which may lead to additional computational burden, and thus hinder its real-time decision-making. Different from the traditional CFA algorithm, a transfer learning (TL) technique is used to describe the relationship between the parameters and stochastic information in the proposed TL-CFA algorithm. Thus, in the look-ahead dispatch process, the parameters of CFA can be directly obtained through the transfer function, which saves the computational time, while almost not losing the accuracy. Case studies are carried out on a modified IEEE 30-bus system and a real 2778-bus system, and the results demonstrate the effectiveness of the proposed approach.

Deep VULMAN: A deep reinforcement learning-enabled cyber vulnerability management framework

Cyber vulnerability management is a critical function of a cybersecurity operations center (CSOC) that helps protect organizations against cyber-attacks on their computer and network systems. Adversaries hold an asymmetric advantage over the CSOC, as the number of deficiencies in these systems is increasing at a significantly higher rate compared to the expansion rate of the security teams to mitigate them. The current approaches in cyber vulnerability management are deterministic and one-time decision-making methods, which do not consider future uncertainties when prioritizing and selecting vulnerabilities for mitigation. These approaches are also constrained by the sub-optimal distribution of resources, providing no flexibility to adjust their response to fluctuations in vulnerability arrivals. We propose a novel framework, Deep VULMAN, consisting of a deep reinforcement learning agent and an integer programming method to fill this gap in cyber vulnerability management process. Our sequential decision-making framework, first, determines the near-optimal amount of resources to be allocated for mitigation under uncertainty for a given system state, and then determines the optimal set of prioritized vulnerability instances for mitigation. Results show that our framework outperforms the current methods in prioritizing the selection of important organization-specific vulnerabilities, on both simulated and real-world vulnerability data, observed over a one-year period.

Optimal Path Planning of Autonomous Marine Vehicles in Stochastic Dynamic Ocean Flows Using a GPU-Accelerated Algorithm

Autonomous marine vehicles play an essential role in many ocean science and engineering applications. Planning time and energy optimal paths for these vehicles to navigate in stochastic dynamic ocean environments are essential to reduce operational costs. In some missions, they must also harvest solar, wind or wave energy (modeled as a stochastic scalar field) and move in optimal paths that minimize net energy consumption. Markov decision processes (MDPs) provide a natural framework for sequential decision making for robotic agents in such environments. However, building a realistic model and solving the modeled MDP becomes computationally expensive in large-scale real-time applications, warranting the need of parallel algorithms and efficient implementation. In this article, we introduce an efficient end-to-end graphical processing unit (GPU)-accelerated algorithm that 1) builds the MDP model (computing transition probabilities and expected one-step rewards) and 2) solves the MDP to compute an optimal policy. We develop methodical and algorithmic solutions to overcome the limited global memory of GPUs by 1) using a dynamic reduced-order representation of the ocean flows; 2) leveraging the sparse nature of the state transition probability matrix; 3) introducing a neighboring subgrid concept; and 4) proving that it is sufficient to use only the stochastic scalar field’s mean to compute the expected one-step rewards for missions involving energy harvesting from the environment, thereby saving memory and reducing the computational effort. We demonstrate the algorithm on a simulated stochastic dynamic environment and highlight that it builds the MDP model and computes the optimal policy 600–1000 times faster than conventional CPU implementations, making it suitable for real-time use.

A sequential decision-making framework with uncertainty quantification for groundwater management

Groundwater contamination causes significant groundwater supply risks that severely affect humans and livestock, especially in the era of climate change. Consequently, the need to develop and optimize remediation strategies such as pump and treat is indispensable. An effective and efficient strategy can reduce the spread of contaminants and cost of clean-up. However, developing such a strategy is challenging due to the inherent uncertainty and sequential nature of the problem. For example, taking actions such as drilling wells is interwoven with information gathering over time. In this paper, we formulate the groundwater remediation planning and decision problem as a partially observable Markov decision process (POMDP). A POMDP formulation allows planning by optimizing the trade-off between information gathering and the performance of possible future scenarios. We use a state-of-the-art POMDP solver called DESPOT for planning remediation strategies. The results show that DESPOT can yield much better remediation strategies than hand crafted heuristics and optimization methods. The superior performance of DESPOT is due to the incorporation of both previous information and future reward into the current decision. We also demonstrate the robustness of the POMDP formulation to measurement error.

Flexible selection framework for secondary frequency regulation units based on learning optimisation method

Grid connections of new energy sources have increased system frequency fluctuations, which requires secondary frequency regulation(SFR) to ensure the safe and stable operation of the system. SFR balances the power deviation between generation and consumption in real time by changing the setpoints of the units. In this paper, we propose a dynamic area control error (ACE) threshold value-based method to rationalise unit selection for SFR tasks. Each conventional unit or virtual generation unit has an independent threshold to aid dispatchers in determining the response priority of each unit to SFR tasks relative to scheduled tasks. In this manner, power generation units can switch their control mode flexibly and track different setpoints using automatic generation control(AGC) and load control technology. We designed a threshold solver based on a sequential decision framework using the deep reinforcement learning method. The solver can dynamically update thresholds based on the real-time system operational state such that the selected units can better meet the current dispatching needs. In the experiment, we tested the performance of the solver in several typical scenarios; the proposed method was found to be effective in maintaining the frequency stability of the system, with lower operation costs.

A dynamic sequential decision-making model on MRI real-time scheduling with simulation-based optimization.

Magnetic resonance imaging (MRI) is widely used in diagnostic medicine and contributes significantly to US health care spending. Scheduling MRI jobs involves uncertainties (e.g., patient arrival time, scanning time, and preparation time) that can lead to excessive delays and high costs in MRI operations. This study addresses real-time decision making in use of MRI scanners based on job assignment and sequencing decisions that override the appointment schedule. The decisions are made using real-time information of the waiting patients, the utilization status of the MRI scanners, and the partially revealed uncertainties of scanning times of current patients. A sequential decision-making framework and a simulation-based solution method are proposed to utilize massive real-time information and match the use of MRI rescheduling in practice. The results are then compared with a real case in a large midwestern academic medical center in the US. This study illustrates that the proposed method reduces patient waiting time by 21.7% and improves utilization of MRI scanners by 23.0%. An optimality gap of 13.6% is provided when compared to off line scheduling methods based on a mixed integer programming (MIP) model. The number of simulation replications in this approach uses the ranking and selection method, which not only reduces solution time, but also provides solution quality guarantees wherein the probability of errors in the proposed method for one day is less than 0.1%. In 100 randomly generated workday experiments, all of the scheduling decisions given by the proposed method perform better than current policy, with an average reduction of 17.93 minutes in each patient's waiting time and an improvement of scanner utilization by 7.20%.

Selective maintenance and inspection optimization for partially observable systems: An interactively sequential decision framework

Selective maintenance is an important condition-based maintenance strategy for multi-component systems, where optimal maintenance actions are identified to maximize the success likelihood of subsequent missions. Most of the existing works on selective maintenance assumed that after each mission, the components’ states can be precisely known without additional efforts. In engineering scenarios, the states of the components in a system need to be revealed by inspections that are usually inaccurate. Inspection activities also consume the limited resources shared with maintenance activities. We, thus, put forth a novel decision framework for selective maintenance of partially observable systems with which maintenance and inspection activities will be scheduled in a holistic and interactively sequential manner. As the components’ states are partially observable and the remaining resources are fully observable, we formulate a finite-horizon Mixed Observability Markov Decision Process (MOMDP) model to support the optimization. In the MOMDP model, both maintenance and inspection actions can be interactively and sequentially planned based on the distributions of components’ states and the remaining resources. To improve the solution efficiency of the MOMDP model, we customize a Deep Value Network (DVN) algorithm in which the maximum mission success probability is approximated. A five-component system and a real-world multi-state coal transportation system are used to demonstrate the effectiveness of the proposed method. It is shown that the probability of the system successfully completing the next mission can be significantly increased by taking inspections into account. The results also demonstrate the computational efficiency of the customized DVN algorithm.

Sequential Resource Planning Decisions in an Epidemic Based on an Innovative Spread Model

Making rapid decisions in intervention resource planning is crucial for mitigating morbidity, mortality, and costs to the societies during epidemic outbreaks. This study presents a data-driven optimization approach for multiperiod resources planning, based on a sequential decision framework, considering up-to-date information and uncertainty in the spread of epidemics. In this method, a new <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$SEI^{3}H^{2}RD$ </tex-math></inline-formula> spread model is constructed to generate the most potential scenarios of an epidemic, based on all the historical information, and risk-averse stochastic programming was proposed to arrive at an optimal resource planning solution. The data-based numerical experiments demonstrate that our approach could control the epidemic by reducing the infected cases and deaths with similar or fewer resources than in the reality. In addition, we also find that the risk-averse design of the objective was able to take a steadier approach to resource planning helped avoid large fluctuations in resource allocations compared to a risk-neutral design. The other insight obtained from these experiments was that a moderate decision interval along with a planning horizon, which is slightly larger than the decision interval, would be a good choice for the sequential planning problem. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In the event of an outbreak of a new infectious disease, the uncertainty of the epidemic parameters limits the ability of the model to provide accurate predictions. However, decision-makers cannot wait for more information to alleviate this uncertainty and must immediately make decisions to control the epidemic. This article proposes a sequential decision-making approach with an innovative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$SEI^{3}H^{2}RD$ </tex-math></inline-formula> spread model and a risk-averse scenario-based stochastic programming to help decision-makers arrive at real-time decisions. We illustrate the performance of this approach using the COVID-19 outbreak in Wuhan. The results show that our model predictions closely fit the real outbreak data and prove that our decision approach could reduce the cost of using the available resources and achieve the goal of controlling the epidemic. The proposed modeling framework can be adopted to study other infectious diseases and provide tangible policy recommendations for controlling outbreaks of such diseases.

Reinforcement Learning for Motion Policies in Mobile Relaying Networks

We consider joint beamforming and relay motion control in mobile relay beamforming networks, operating in a spatio-temporally varying channel environment. A time slotted approach is adopted, where in each slot, the relays implement optimal beamforming and estimate their optimal positions for the next slot. We place the problem of relay motion control in a sequential decision-making framework. We employ Reinforcement Learning (RL) to guide the relay motion, with the goal of maximizing the cumulative Signal-to-Interference+Noise Ratio (SINR) at the destination. First, we present a model based RL approach, which predictively estimates the SINR and accordingly determines the relay motion, based on partial knowledge of the channel model along with channel measurements at the current relay positions. Second, we propose a model-free deep Q-learning approach, which does not rely on channel models. For the deep Q-learning approach, we propose two modified Multilayer Perceptron Neural Networks (MLPs) for approximating the value function Q. The first modification applies a Fourier feature mapping of the state before passing it through the MLP. The second modification constitutes a different neural network architecture that uses sinusoids as activations between layers. Both modifications enable the MLP to better learn the high frequency value function and have a profound effect on convergence speed and SINR performance. Finally, we conduct a comparative analysis of all the presented approaches and provide insights on advantages and drawbacks.

A Value-of-Information Framework for Personalizing the Timing of Surveillance Testing.

Patient surveillance using repeated biomarker measurements presents an opportunity to detect and treat disease progression early. Frequent surveillance testing using biomarkers is recommended and routinely conducted in several diseases, including cancer and diabetes. However, frequent testing involves tradeoffs. Although surveillance tests provide information about current disease status, the complications and costs of frequent tests may not be justified for patients who are at low risk of progression. Predictions based on patients' earlier biomarker values may be used to inform decision making; however, predictions are uncertain, leading to decision uncertainty. We propose the Personalized Risk-Adaptive Surveillance (PRAISE) framework, a novel method for embedding predictions into a value-of-information (VOI) framework to account for the cost of uncertainty over time and determine the time point at which collection of biomarker data would be most valuable. The proposed sequential decision-making framework is innovative in that it leverages the patient's longitudinal history, considers individual benefits and harms, and allows for dynamic tailoring of surveillance intervals by considering the uncertainty in current information and estimating the probability that new information may change treatment decisions, as well as the impact of this change on patient outcomes. When applied to data from cystic fibrosis patients, PRAISE lowers costs by allowing some patients to skip a visit, compared to an "always test" strategy. It does so without compromising expected survival, by recommending less frequent testing among those who are unlikely to be treated at the skipped time point. A VOI-based approach to patient monitoring is feasible and could be applied to several diseases to develop more cost-effective and personalized strategies for ongoing patient care. In many patient-monitoring settings, the complications and costs of frequent tests are not justified for patients who are at low risk of disease progression. Predictions based on patient history may be used to individualize the timing of patient visits based on evolving risk.We propose Personalized Risk-Adaptive Surveillance (PRAISE), a novel method for personalizing the timing of surveillance testing, where prediction modeling projects the disease trajectory and a value-of-information (VOI)-based pragmatic decision-theoretic framework quantifies patient- and time-specific benefit-harm tradeoffs.A VOI-based approach to patient monitoring could be applied to several diseases to develop more personalized and cost-effective strategies for ongoing patient care.

Markov Decision Process Modeling for Multi-stage Optimization of Intervention and Treatment Strategies in Breast Cancer.

The breast cancer is a prevalent problem that undermines quality of patients' lives and causes significant impacts on psychosocial wellness. Advanced sensing provides unprecedented opportunities to develop smart cancer care. The available sensing data captured from individuals enable the extraction of information pertinent to the breast cancer conditions to construct efficient and personalized intervention and treatment strategies. This research develops a novel sequential decision-making framework to determine optimal intervention and treatment planning for breast cancer patients. We design a Markov decision process (MDP) model for both objectives of intervention and treatment costs as well as quality adjusted life years (QALYs) with the data-driven and state-dependent intervention and treatment actions. The state space is defined as a vector of age, health status, prior intervention, and treatment plans. Also, the action space includes wait, prophylactic surgery, radiation therapy, chemotherapy, and their combinations. Experimental results demonstrate that prophylactic mastectomy and chemotherapy are more effective than other intervention and treatment plans in minimizing the expected cancer cost of 25 to 60 years-old patient with in-situ stage of cancer. However, wait policy leads to an optimal quality of life for a patient with the same state. The proposed MDP framework can also be generally applicable to a variety of medical domains that entail evidence-based decision making.

Optimal Stopping and Worker Selection in Crowdsourcing: an Adaptive Sequential Probability Ratio Test Framework

In this study, we solve a class of multiple testing problems under a Bayesian sequential decision framework. Our work is motivated by binary labeling tasks in crowdsourcing, where a requestor needs to simultaneously choose a worker to provide a label and decide when to stop collecting labels, under a certain budget constraint. We begin by using a binary hypothesis testing problem to determine the true label of a single object, and provide an optimal solution by casting it under an adaptive sequential probability ratio test framework. Then, we characterize the structure of the optimal solution, that is, the optimal adaptive sequential design, which minimizes the Bayes risk using a log-likelihood ratio statistic. We also develop a dynamic programming algorithm to efficiently compute the optimal solution. For the multiple testing problem, we propose an empirical Bayes approach for estimating the class priors, and show that the average loss of our method converges to the minimal Bayes risk under the true model. Experiments on both simulated and real data show the robustness of our method, as well as its superiority over existing methods in terms of its labeling accuracy.

Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

Markov decision processes (MDPs) are the defacto framework for sequential decision making in the presence of stochastic uncertainty. A classical optimization criterion for MDPs is to maximize the expected discounted-sum payoff, which ignores low probability catastrophic events with highly negative impact on the system. On the other hand, risk-averse policies require the probability of undesirable events to be below a given threshold, but they do not account for optimization of the expected payoff. We consider MDPs with discounted-sum payoff with failure states which represent catastrophic outcomes. The objective of risk-constrained planning is to maximize the expected discounted-sum payoff among risk-averse policies that ensure the probability to encounter a failure state is below a desired threshold. Our main contribution is an efficient risk-constrained planning algorithm that combines UCT-like search with a predictor learned through interaction with the MDP (in the style of AlphaZero) and with a risk-constrained action selection via linear programming. We demonstrate the effectiveness of our approach with experiments on classical MDPs from the literature, including benchmarks with an order of 106 states.

A Novel Decision-Making Framework for Probabilistic Search

In this paper, we present the search problem formulated as a decision making problem, the search agent decides whether the target is present in the search region, and if so, where it is located. Then we propose a sequential decision-making framework for a search agent to find a lost target within a bounded region with the searcher motion constraints and imperfect observations being considered. This framework enables the theoretical study to the evolution of the search decision, then we propose two effective adaptive search strategies, which have the advantage of requiring less computation than traditional search strategy. Simulation results shows that the proposed adaptive search strategies have a better performance. Finally, the proposed control strategies are analyzed from different perspectives in detail.