Value Of Information Approach Research Articles

Making Drug Approval Decisions in the Face of Uncertainty: Cumulative Evidence versus Value of Information.

The COVID-19 pandemic underscored the criticality and complexity of decision making for novel treatment approval and further research. Our study aims to assess potential decision-making methodologies, an evaluation vital for refining future public health crisis responses. We compared 4 decision-making approaches to drug approval and research: the Food and Drug Administration's policy decisions, cumulative meta-analysis, a prospective value-of-information (VOI) approach (using information available at the time of decision), and a reference standard (retrospective VOI analysis using information available in hindsight). Possible decisions were to reject, accept, provide emergency use authorization, or allow access to new therapies only in research settings. We used monoclonal antibodies provided to hospitalized COVID-19 patients as a case study, examining the evidence from September 2020 to December 2021 and focusing on each method's capacity to optimize health outcomes and resource allocation. Our findings indicate a notable discrepancy between policy decisions and the reference standard retrospective VOI approach with expected losses up to $269 billion USD, suggesting suboptimal resource use during the wait for emergency use authorization. Relying solely on cumulative meta-analysis for decision making results in the largest expected loss, while the policy approach showed a loss up to $16 billion and the prospective VOI approach presented the least loss (up to $2 billion). Our research suggests that incorporating VOI analysis may be particularly useful for research prioritization and treatment implementation decisions during pandemics. While the prospective VOI approach was favored in this case study, further studies should validate the ideal decision-making method across various contexts. This study's findings not only enhance our understanding of decision-making strategies during a health crisis but also provide a potential framework for future pandemic responses. This study reviews discrepancies between a reference standard (retrospective VOI, using hindsight information) and 3 conceivable real-time approaches to research-treatment decisions during a pandemic, suggesting suboptimal use of resources.Of all prospective decision-making approaches considered, VOI closely mirrored the reference standard, yielding the least expected value loss across our study timeline.This study illustrates the possible benefit of VOI results and the need for evidence accumulation accompanied by modeling in health technology assessment for emerging therapies.

Value of Information for Clinical Trial Design: The Importance of Considering All Relevant Comparators.

Value of Information (VOI) analyses calculate the economic value that could be generated by obtaining further information to reduce uncertainty in a health economic decision model. VOI has been suggested as a tool for research prioritisation and trial design as it can highlight economically valuable avenues for future research. Recent methodological advances have made it increasingly feasible to use VOI in practice for research; however, there are critical differences between the VOI approach and the standard methods used to design research studies such as clinical trials. We aimed to highlight key differences between the research design approach based on VOI and standard clinical trial design methods, in particular the importance of considering the full decision context. We present two hypothetical examples to demonstrate that VOI methods are only accurate when (1) all feasible comparators are included in the decision model when designing research, and (2) all comparators are retained in the decision model once the data have been collected and a final treatment recommendation is made. Omitting comparators from either the design or analysis phase of research when using VOI methods can lead to incorrect trial designs and/or treatment recommendations. Overall, we conclude that incorrectly specifying the health economic model by ignoring potential comparators can lead to misleading VOI results and potentially waste scarce research resources.

Assessing whether decisions are more sensitive to preference or prediction uncertainty with a value of information approach

In many decisions, we are not only uncertain about the predicted outcomes of decision alternatives but also about stakeholder preferences regarding these outcomes. Further information collection may reduce uncertainties, but is costly. We present and apply a framework to identify the most decisive uncertainties and prioritize data collection efforts based on value of information (VoI) sensitivity analysis. Preference uncertainty is usually not explicitly considered in VoI analysis or in standard utility theory. Based on the expected expected utility (EEU) concept, we consider uncertain predictions and preferences jointly in decisions and subsequent VoI analysis. We focus on the expected value of partially perfect information (EVPPI) and adapt a fast, given-data algorithm for estimating this metric. The framework is motivated by complex environmental decision problems and we apply it to a hypothetical multi-criteria decision regarding coral reef management with conflicting stakeholder perspectives. The results show that better understanding of stakeholder positions can be as relevant as improving system understanding. For one perspective, preference model parameters had the highest EVPPI, while for another predictive uncertainties of the reef system attributes were more relevant. For two perspectives, the decision was largely insensitive. By considering predictive and preferential uncertainty on an equal footing in VoI analysis, we open up possibilities to design data collection for decision support processes more efficiently.

Assessing Whether Decisions Are More Sensitive to Preference or Prediction Uncertainty with a Value of Information Approach

Assessing Whether Decisions Are More Sensitive to Preference or Prediction Uncertainty with a Value of Information Approach

Technology Development Framework: Moving from Qualitative toward Quantitative Decision-Making

Summary Technology development in the energy sector typically suffers from an ad hoc approach to making decisions about the key questions of “What is the next derisking step?” and “At what scale?” This lack of structure in decision-making occurs despite the technology development process usually being within a stage-gated protocol. This often leads to significant waste of time and resources and plenty of “surprises” about the extent of additional testing required. The approach proposed in this paper is relatively simple to use, scalable, and transferable across industry sectors. Consistent application of the methodology has the potential to bring a quantitative rigor that can lead to more effective and efficient technology development and improve both intellectual and monetary capital allocation. Additionally, the framework provided facilitates a more structured and faster learning curve for people moving into technology development roles. Starting with a high-level understanding of the technology and business case, the methodology identifies possible commercial deployments and associates key technical and commercial risks that can be barriers to commercialization. Then, using a “risk burndown” approach based around an initial assessment of probability of success (POS), different paths to deployment (PTDs) are developed. A value of information (VOI) approach is used to evaluate key metrics. This leads to a diverse set of derisking options with associated trade-offs for informed decision-making. The methodology has been developed and deployed since 2017 across a portfolio of both incremental improvement and game-changing subsurface technologies at Suncor Energy, a large integrated Canadian energy company. More than 25 technologies have been progressed using elements of this workflow with 7 of the more complex and expensive projects using the full quantitative VOI approach. These analyses have been used to inform decision-making on more than $400 million1 of proposed derisking activity spend.

Value of Information in Wireless Sensor Network Applications and the IoT: A Review

Value of Information (VoI) is a concept to assess the usefulness of information for a specific goal, and has in the last decade experienced a growing interest also for Wireless Sensor Network (WSN) applications and the Internet of Things (IoT). By making the value of information explicit in the form of VoI, WSN and IoT applications should be able to better assess which information to spend their constrained resources on. However, the definition of VoI is highly application-dependent, which has led to a fragmented understanding of VoI, and there is a lack of a comprehensive overview. In this structured review, we first categorize application use cases and examine what VoI is used for, and explore the different approaches to defining VoI. We then provide a well-structured and comprehensive discussion of the specific approaches used in the literature to determine VoI, together with examples of use cases. We categorize the different approaches to calculating VoI, describe their properties systematically and distinguish between observed VoI and expected VoI. We also discuss adaptive VoI approaches and point towards future directions within the field.

A Unified Approach for Outliers and Influential Data Detection - The Value of Information in Retrospect.

Identifying influential and outlying data is important as it would guide the effective collection of future data and the proper use of existing information. We develop a unified approach for outlier detection and influence analysis. Our proposed method is grounded in the intuitive value of information concepts and has a distinct advantage in interpretability and flexibility when compared to existing methods: it decomposes the data influence into the leverage effect (expected to be influential) and the outlying effect (surprisingly more influential than being expected); and it applies to all decision problems such as estimation, prediction, and hypothesis testing. We study the theoretical properties of three value of information quantities, establish the relationship between the proposed measures and classic measures in the linear regression setting, and provide real data analysis examples of how to apply the new value of information approach in the cases of linear regression, generalized linear mixed model, and hypothesis testing.

Estimating the Societal Benefits of Satellite Instruments: Application to a Break-even Analysis of the GeoXO Hyperspectral IR Sounder

This paper describes the process, and lessons learned in a preliminary benefits study of a proposed infrared hyperspectral sounder (HSS) for NOAA’s next generation geostationary satellite program (GeoXO). The valuation of government-owned satellite systems providing a public good with a complex array of instruments is a nascent field of study. Many different sensor configurations are possible, but there are restrictions due to physical and budget constraints. Accounting for economic considerations during the design and planning phase for satellite constellations helps to ensure that the most cost effective instruments are selected. To assess whether the HSS instrument should be included on GeoXO, we applied a value of information approach and found the benefits associated with this instrument are likely to substantially outweigh the costs. Value of information studies often focus on data and information that has a direct use case. Estimating benefits for the HSS is especially challenging because data are not used directly by decision makers. Instead these data along with information from other Earth observing (EO) satellites play a key role in producing the inputs necessary for modern numerical hydrometeorological modeling. We describe strategies to assess the marginal (i.e., incremental) contribution of an instrument that is part of a complex information production process. We make several recommendations that, if implemented, would improve the quality of future studies of this kind. This includes (1) a systems approach to observing system planning, (2) improving the design of observing system experiments (OSSE and OSE), and (3) better tracking of the decisions and needs of end-users, especially those external to the agency.

The application of VOI method to shape a strategy of additional exploration

Background. An optimal exploration strategy creates a significant share in value of project on exploration stage. The paper describes an example of solving the following tasks: determining the feasibility of additional exploration drilling; evaluating the value of drilling of one or more exploration wells; determining the optimal placement for exploration wells and drilling order. Authors presenting the modification of VoI (Value of Information) method and its application. Materials and methods. Complex probabilistic models were created summarizing main uncertainties and limitations, both geological, technical and technological. At the first stage three equiprobable geological concepts were made. For each concept probabilistic geological modelling was proceeded and then realizations corresponding to values of reserves P10, P50, and P90 were selected. Further, detailed production forecasts and economic estimates were performed. The analysis used the well pad and the corresponding area for exploration drilling as a unit of calculation. In the article the authors introduced the concept of remaining uncertainty. Application of modified VoI method allowed to form ‘dynamic’ (i.e. depending on exploration wells drilling order) range of areas for additional exploration which provide the best decrease of remaining uncertainty. An additional exploration strategy has been formed, which includes the necessary and sufficient number of wells and their drilling order. A decision tree was created depending on the success or failure of each subsequent exploration well. Results. The use of the modified VoI approach made it possible to achieve the objectives and obtain economical estimates, all of which combined to facilitate the adoption of decisions. As a result, a program for two exploration well drilling was created which would reduce the uncertainty by 90% from its initial value. Conclusions: The adopted VoI method could be applied to fields at the stage of additional exploration as well as to fields at early exploration stage to develop an exploration drilling strategy.

Optimal placement of sampling locations for identification of a two-dimensional space

ABSTRACT In recent years, substantial attention has been given to the optimal placement and planning of sampling or monitoring based on the Value of Information (VoI). The drawback of the VoI approach is its typically high computation cost. The method proposed here can significantly reduce the computation cost of VoI by applying the theory of Gaussian Process Regression. Due to the low computation cost, global optimisation algorithms such as particle swarm optimisation can be used to optimise a set of sampling locations. The proposed method was applied to a soil contamination problem. For comparison purposes, twenty-one subjects (persons) were asked to follow a three-step procedure. In the first step, contamination levels at five fixed sampling locations are given. Based on this information, the contaminated area is estimated by the subject, who is then asked to place five additional sampling sites in order to gather more information. The procedure is repeated three times. The subject is ultimately asked to produce a final estimate of the contaminated area. It was found that the proposed method generally produced better agreement with the true contaminated area than did the participants’ empirical human judgment.

VoI-Based Optimization of Structural Assessment for Spatially Degrading RC Structures

Before implementing a bridge monitoring strategy, a bridge manager would like to know the return on investment. Moreover, in order to spend the available budget as efficiently as possible, the monitoring strategy should be optimized, i.e., the type of measurements but also the time and locations at which these are performed. For this purpose, the Value of Information (VoI) can be used. The VoI represents an estimate of the benefit that can be gained from a monitoring strategy before it is actually implemented. By comparing the VoI of different alternative strategies, the one with the highest VoI can be selected. As such, the VoI is a tool for objective decision-making. The calculation of the VoI is based on pre-posterior analyses, including Bayesian updating of model parameters based on yet unknown monitoring outcomes. When calculating the VoI for an actual case, some challenges arise. First, the calculation of the VoI requires a number of assumptions on different input parameters. Second, the VoI is computed by evaluating life-cycle costs for different possible outcomes of the monitoring strategy, leading to a high computational cost. However, for practical implementations, results are preferably available within an acceptable time span and are robust with respect to the chosen input parameters. In this work, the implementation of the VoI approach for optimization of monitoring strategies is investigated by a problem statement in a case study where a reinforced concrete girder bridge is considered. To perform this optimization, the VoI for different monitoring strategies is compared. The calculation time required for the Bayesian updating of the model parameters based on the available data is limited by using Maximum A Posteriori (MAP) estimates to approximate the posterior distribution. The VoI can be used both to optimize a monitoring strategy or for comparison of different strategies. To limit the number of required (computationally expensive) evaluations of the VoI, optimization of the monitoring strategy itself can be simplified by determining the optimal sensor locations beforehand, based on a different metric than the VoI. For this purpose, the information entropy is used, which expresses the difference between the prior and posterior uncertainty of the model parameters. Finally, the sensitivity of the VoI to different input parameters is investigated.

Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment.

Value of information (VOI) analysis often requires modeling to characterize and propagate uncertainty. In collaboration with a cancer clinical trial group, we integrated a VOI approach to assessing trial proposals. This paper aims to explore the impact of modeling choices on VOI results and to share lessons learned from the experience. After selecting two proposals (A: phase III, breast cancer; B: phase II, pancreatic cancer) for in-depth evaluations, we categorized key modeling choices relevant to trial decision makers (characterizing uncertainty of efficacy, evidence thresholds to change clinical practice, and sample size) and modelers (cycle length, survival distribution, simulation runs, and other choices). Using a $150,000 per quality-adjusted life-year (QALY) threshold, we calculated the patient-level expected value of sample information (EVSI) for each proposal and examined whether each modeling choice led to relative change of more than 10% from the averaged base-case estimate. We separately analyzed the impact of the effective time horizon. The base-case EVSI was $118,300 for Proposal A and $22,200 for Proposal B per patient. Characterizing uncertainty of efficacy was the most important choice in both proposals (e.g. Proposal A: $118,300 using historical data vs. $348,300 using expert survey), followed by the sample size and the choice of survival distribution. The assumed effective time horizon also had a substantial impact on the population-level EVSI. Modeling choices can have a substantial impact on VOI. Therefore, it is important for groups working to incorporate VOI into research prioritization to adhere to best practices, be clear in their reporting and justification for modeling choices, and to work closely with the relevant decision makers, with particular attention to modeling choices.

The value of information for the management of water resources in agriculture: Assessing the economic viability of new methods to schedule irrigation

This study develops a methodology to assess the comparative advantages of new methods to plan irrigation with respect to prevailing existing irrigation practices. The methodology consists of a comparative cost-benefit analysis based on the Value of Information approach that makes it possible to analyse whether an improvement in the information available to farmers generates economic benefits. The method is applied to the problem of comparing computer irrigation models (providing irrigation advice based on measurements, water balance models and weather predictions) and prevailing irrigation practices (at times based on soil and plant observations, or on advanced technologies) in estimating and predicting crop water requirements, in pilot experiments located in four different European regions. The results reveal that the introduction of the alternative method improves the performance of irrigation practices in Mediterranean regions that are characterised by high weather variability and for those crops for which the consequences of failing to meet predictions are relatively low (i.e. tomato instead of maize, drip irrigated crops instead of sprinkler irrigated crops). Under favourable conditions, the use of the alternative technology generates a 0–20% increase in gross margin and a 10–30% water saving with respect to prevailing existing irrigation practices. The study concludes by addressing the conditions that justify the use of advanced information systems to schedule irrigation interventions and by offering some policy recommendations to drive their uptake. These include subsidising research at the evaluation stage and public investments aimed at knowledge creation (weather and shallow water table monitoring stations) and knowledge sharing (counselling) at the adoption stage.

General rules for environmental management to prioritise social ecological systems research based on a value of information approach

Abstract Globally, billions of dollars are invested each year to help understand the dynamics of social ecological systems (SES) in bettering both social and environmental outcomes. However, there is no scientific consensus on which aspect of an SES is most important and urgent to understand; particularly given the realities of limited time and money. Here we use a simulation‐based “value of information” approach to examine where research will deliver the most important information for environmental management in four SESs representing a range of real‐life environmental issues. We find that neither social nor ecological information is consistently the most important: instead, researchers should focus on understanding the primary effects of their management actions. Thus, when managers are undertaking social actions the highest research priority should be understanding the dynamics of social groups. Alternatively, when manipulating ecological systems it will be most important to quantify ecological population dynamics. Synthesis and applications. Our results provide a standard assessment to determine the uncertain social ecological systems (SES) component with the highest expected impact for management outcomes. First, managers should determine the structure of their SES by identifying social and ecological nodes. Second, managers should identify the qualitative nature of the network, by determining which nodes are linked, but not the strength of those interactions. Finally, managers should identify the actions available to them to intervene in the SES. From these steps, managers will be able to identify the SES components that are closest to the management action(s), and it is these nodes and interactions that should receive priority research attention to achieve effective environmental decision making.

Leveraging Cumulative Network Meta-analysis and Value of Information Analysis to Understand the Evolving Value of Medical Research

Leveraging cumulative network meta-analysis (NMA) and value of information (VOI) analysis, this article aims to understand the evolving value of medical research and to identify gaps in the evidence for future research. As an illustration, we identified 31 randomized controlled trials (RCT) from 1980 to 2013 that examined a network of 3 interventions for coronary artery disease: medical therapy (MED), percutaneous coronary intervention (PCI), and coronary artery bypass graft (CABG) surgery. We conducted Bayesian NMA to combine evidence from a new RCT with existing knowledge. Then, using the Duke Databank for Cardiovascular Diseases database, we developed an accelerated failure time model to estimate the joint effects of patient characteristics and treatment choices on survival outcomes. With the estimated coefficients and covariance matrices, we projected survival benefits and its surrounding uncertainty among 50,000 simulated patients treated with MED, PCI, or CABG. The value of resolving residual uncertainty from future trials was quantified through the VOI analysis. We repeated these steps for each published RCT to estimate dynamic changes in VOI estimates. Our cumulative NMA found that CABG conferred a lower, but not statistically significant, mortality than PCI (hazard ratio [HR], 0.90; 95% uncertainty interval, 0.80-1.05). MED had a nonsignificantly higher long-term mortality than PCI (HR, 1.11; 0.98-1.31) but significantly higher than CABG (HR, 1.07; 1.23-1.41). The greatest potential gains from future research would come from additional head-to-head trials between CABG v. PCI with the value of future research equaling 0.27 life years per patient. The combination of cumulative NMA and VOI approaches can improve the efficiency of comparative effectiveness research by using all of the available evidence to determine future research priorities.

Quantifying the value of monitoring species in multi‐species, multi‐threat systems

Abstract Making effective management decisions is challenging in multi‐species, multi‐threat systems because of uncertainty about the effects of different threats on different species. To inform management decisions, we often monitor species to detect spatial or temporal trends that can help us learn about threatening processes. However, which species to monitor and how to monitor to inform the management of threats can be difficult to determine. Value of information (VOI) analysis is an approach for quantifying the value of monitoring to inform management decisions. We developed a novel method that applies VOI analysis to quantify the benefits of different species monitoring strategies in multi‐threat, multi‐species systems. We applied the approach to compare the effectiveness of surveillance monitoring (monitoring species without experimentation) to targeted monitoring (monitoring species with experimentation to learn about a specific threat), and how prior information drives the benefits of these two different strategies and the species to monitor. We also illustrate the approach by applying it to two contrasting case studies for monitoring and managing declining mammals in Western Australia. Our approach shows that surveillance monitoring generally provides far lower benefits than targeted monitoring for managing threats in multi‐species, multi‐threat systems under economic constraints. Our approach also informs the choice of species to monitor and which threats to manage experimentally to most improve threat management outcomes. We show that the key parameters driving these choices include: the budget available for management, prior understanding of which threats cause declines in which species, the relative cost of managing these threats, and the background probability of decline. Our new VOI approach allows the evaluation of monitoring decisions in multi‐species, multi‐threat systems in the face of uncertainty, while explicitly accounting for the improvement in management outcomes. We recommend that managers need to explicitly consider a range of decision parameters when selecting which species to monitor to inform management. Our framework provides an objective way to do this.

Making the best use of experts' estimates to prioritise monitoring and management actions: A freshwater case study

Under limited time and resources, ecological managers are under increasing pressure to demonstrate tangible impact of monitoring activities. Value of Information (VOI) has been advocated as an ideal tool to evaluate whether more data is required to improve expected management outcomes. Yet, despite several recent works explaining its value, VOI remains seldom used in practice. Here we provide an example of a successful ecological application of VOI. We apply VOI to a novel multi-objective freshwater management problem and show how to make the best use of expert data through a robust sensitivity analysis. Unlike previous VOI approaches, our analysis provides statistical confidence to our recommendations. We apply our approach to the recovery of Moira grass (Pseudoraphis spinescens) plains, a threatened vegetation community at the Ramsar-listed Barmah Forest on the Murray River, Australia. Working closely with managers, we discovered that although many threats may impede Moira grass recovery, reducing grazing pressure and applying ideal depth and duration of flooding were most likely to lead to recovery. We found that learning from monitoring can significantly increase the existing extent of Moira grass, although these gains are modest compared to immediate management action. Our study shows how VOI can be used to demonstrate efficient use of limited environmental water to maximise ecological impact and increase transparency when making monitoring or management decisions. More broadly, the study methods will be of interest to any environmental manager who needs to prioritise monitoring and evaluation activities subject to a limited research budget. At a time where researchers and managers are asked to be more accountable for their decision-making, VOI provides a very accessible tool that can speed up the decision of whether to wait and collect more data or act immediately despite uncertainty.

Determination of sample size to support diagnostic inspection of components

A complex engineering system like a nuclear power reactor consists of a large variety and number of engineering components. As a part of a component aging management program, the diagnostic inspections of various component populations are performed to detect the onset of any unanticipated degradation. A prudent selection of the inspection sample size is necessary to optimise inspection cost. Sample size selection is typically based on the traditional statistical hypothesis test, which tends to result in a fairly large sample size. This paper presents an alternate approach to the sample size determination (SSD) problem based on the concept of the value of information (VoI). The paper provides a comparative analysis of the VoI and hypothesis-testing approaches through illustrative examples. The VoI approach is shown to provide a more meaningful way to minimise the cost of inspection as a function of component-replacement cost and losses arising from a failure. The characteristics and advantages of the VoI approach are analysed.

Determination of sample size to support diagnostic inspection of components

A complex engineering system like a nuclear power reactor consists of a large variety and number of engineering components. As a part of a component aging management program, the diagnostic inspections of various component populations are performed to detect the onset of any unanticipated degradation. A prudent selection of the inspection sample size is necessary to optimise inspection cost. Sample size selection is typically based on the traditional statistical hypothesis test, which tends to result in a fairly large sample size. This paper presents an alternate approach to the sample size determination (SSD) problem based on the concept of the value of information (VoI). The paper provides a comparative analysis of the VoI and hypothesis-testing approaches through illustrative examples. The VoI approach is shown to provide a more meaningful way to minimise the cost of inspection as a function of component-replacement cost and losses arising from a failure. The characteristics and advantages of the VoI approach are analysed.

Value of Global Catastrophic Risk (GCR) Information: Cost-Effectiveness-Based Approach for GCR Reduction

In this paper, we develop and illustrate a framework for determining the potential value of global catastrophic risk (GCR) research in reducing uncertainties in the assessment of GCR levels and the effectiveness of risk-reduction options. The framework uses the decision analysis concept of the expected value of perfect information in terms of the cost-effectiveness of GCR reduction. We illustrate these concepts using available information on impact risks from two types of near-Earth objects (asteroids or extinct comets) as well as nuclear war, and consideration of two risk-reduction measures. We also discuss key challenges in extending the calculations to all GCRs and risk-reduction options, as part of an agenda for comprehensive, integrated GCR research. While real-world research would not result in perfect information, even imperfect information could have significant value in informing GCR-reduction decisions. Unlike most value of information approaches, our equation for calculating value of information is based on risk-reduction cost-effectiveness, to avoid implicitly equating lives and dollars, e.g., using a value of statistical life (VSL), which may be inappropriate given the scale of GCRs. Our equation for value of information may be useful in other domains where VSLs would not be appropriate.