Data-worth Analysis Research Articles

An efficient Bayesian data-worth analysis using a multilevel Monte Carlo method

Abstract Improving the understanding of subsurface systems and thus reducing prediction uncertainty requires collection of data. As the collection of subsurface data is costly, it is important that the data collection scheme is cost-effective. Design of a cost-effective data collection scheme, i.e., data-worth analysis, requires quantifying model parameter, prediction, and both current and potential data uncertainties. Assessment of these uncertainties in large-scale stochastic subsurface hydrological model simulations using standard Monte Carlo (MC) sampling or surrogate modeling is extremely computationally intensive, sometimes even infeasible. In this work, we propose an efficient Bayesian data-worth analysis using a multilevel Monte Carlo (MLMC) method. Compared to the standard MC that requires a significantly large number of high-fidelity model executions to achieve a prescribed accuracy in estimating expectations, the MLMC can substantially reduce computational costs using multifidelity approximations. Since the Bayesian data-worth analysis involves a great deal of expectation estimation, the cost saving of the MLMC in the assessment can be outstanding. While the proposed MLMC-based data-worth analysis is broadly applicable, we use it for a highly heterogeneous two-phase subsurface flow simulation to select an optimal candidate data set that gives the largest uncertainty reduction in predicting mass flow rates at four production wells. The choices made by the MLMC estimation are validated by the actual measurements of the potential data, and consistent with the standard MC estimation. But compared to the standard MC, the MLMC greatly reduces the computational costs.

ANOVA-based transformed probabilistic collocation method for Bayesian data-worth analysis

Abstract Bayesian theory provides a coherent framework in quantifying the data worth of measurements and estimating unknown parameters. Nevertheless, one common problem in Bayesian methods is the considerably high computational cost since a large number of model evaluations is required in the likelihood evaluation. To address this issue, a new surrogate modeling method, i.e., ANOVA (analysis of variance)-based transformed probabilistic collocation method (ATPCM), is developed in this work. To cope with the strong nonlinearity, the model responses are transformed to the arrival times, which are then approximated with a set of low-order ANOVA components. The validity of the proposed method is demonstrated by synthetic numerical cases involving water and heat transport in the vadose zone. It is shown that, the ATPCM is more efficient than the existing surrogate modeling methods (e.g., PCM, ANOVA-based PCM and TPCM). At a very low computational cost, the ATPCM-based Bayesian data-worth analysis provides a quantitative metric in comparing different monitoring plans, and helps to improve the parameter estimation. Although the flow and heat transport in vadose zone is considered in this work, the proposed method can be equally applied in any other hydrologic problems.

Extending Data Worth Analyses to Select Multiple Observations Targeting Multiple Forecasts.

Hydrological models are often set up to provide specific forecasts of interest. Owing to the inherent uncertainty in data used to derive model structure and used to constrain parameter variations, the model forecasts will be uncertain. Additional data collection is often performed to minimize this forecast uncertainty. Given our common financial restrictions, it is critical that we identify data with maximal information content with respect to forecast of interest. In practice, this often devolves to qualitative decisions based on expert opinion. However, there is no assurance that this will lead to optimal design, especially for complex hydrogeological problems. Specifically, these complexities include considerations of multiple forecasts, shared information among potential observations, information content of existing data, and the assumptions and simplifications underlying model construction. In the present study, we extend previous data worth analyses to include: simultaneous selection of multiple new measurements and consideration of multiple forecasts of interest. We show how the suggested approach can be used to optimize data collection. This can be used in a manner that suggests specific measurement sets or that produces probability maps indicating areas likely to be informative for specific forecasts. Moreover, we provide examples documenting that sequential measurement election approaches often lead to suboptimal designs and that estimates of data covariance should be included when selecting future measurement sets.

Toward Increased Use of Data Worth Analyses in Groundwater Studies.

Article impact statement: Groundwater data will always be sparse–that is the nature of our science. Determining how, where, and when to collect data in a more informed manner is the best way to advance conceptual understanding and management of groundwater systems.

ITOUGH2: A multiphysics simulation-optimization framework for analyzing subsurface systems

iTOUGH2 is a simulation-optimization framework for the TOUGH suite of nonisothermal multiphase flow models and related simulators of geophysical, geochemical, and geomechanical processes. After appropriate parameterization of subsurface structures and their properties, iTOUGH2 runs simulations for multiple parameter sets and analyzes the resulting output for parameter estimation through automatic model calibration, local and global sensitivity analyses, data-worth analyses, and uncertainty propagation analyses. Development of iTOUGH2 is driven by scientific challenges and user needs, with new capabilities continually added to both the forward simulator and the optimization framework. This review article provides a summary description of methods and features implemented in iTOUGH2, and discusses the usefulness and limitations of an integrated simulation-optimization workflow in support of the characterization and analysis of complex multiphysics subsurface systems.

A python framework for environmental model uncertainty analysis

We have developed pyEMU, a python framework for Environmental Modeling Uncertainty analyses, open-source tool that is non-intrusive, easy-to-use, computationally efficient, and scalable to highly-parameterized inverse problems. The framework implements several types of linear (first-order, second-moment (FOSM)) and non-linear uncertainty analyses. The FOSM-based analyses can also be completed prior to parameter estimation to help inform important modeling decisions, such as parameterization and objective function formulation. Complete workflows for several types of FOSM-based and non-linear analyses are documented in example notebooks implemented using Jupyter that are available in the online pyEMU repository. Example workflows include basic parameter and forecast analyses, data worth analyses, and error-variance analyses, as well as usage of parameter ensemble generation and management capabilities. These workflows document the necessary steps and provides insights into the results, with the goal of educating users not only in how to apply pyEMU, but also in the underlying theory of applied uncertainty quantification.

Data-worth analysis through probabilistic collocation-based Ensemble Kalman Filter

We propose a new and computationally efficient data-worth analysis and quantification framework keyed to the characterization of target state variables in groundwater systems. We focus on dynamically evolving plumes of dissolved chemicals migrating in randomly heterogeneous aquifers. An accurate prediction of the detailed features of solute plumes requires collecting a substantial amount of data. Otherwise, constraints dictated by the availability of financial resources and ease of access to the aquifer system suggest the importance of assessing the expected value of data before these are actually collected. Data-worth analysis is targeted to the quantification of the impact of new potential measurements on the expected reduction of predictive uncertainty based on a given process model. Integration of the Ensemble Kalman Filter method within a data-worth analysis framework enables us to assess data worth sequentially, which is a key desirable feature for monitoring scheme design in a contaminant transport scenario. However, it is remarkably challenging because of the (typically) high computational cost involved, considering that repeated solutions of the inverse problem are required. As a computationally efficient scheme, we embed in the data-worth analysis framework a modified version of the Probabilistic Collocation Method-based Ensemble Kalman Filter proposed by Zeng et al. (2011) so that we take advantage of the ability to assimilate data sequentially in time through a surrogate model constructed via the polynomial chaos expansion. We illustrate our approach on a set of synthetic scenarios involving solute migrating in a two-dimensional random permeability field. Our results demonstrate the computational efficiency of our approach and its ability to quantify the impact of the design of the monitoring network on the reduction of uncertainty associated with the characterization of a migrating contaminant plume.

Practical notes on local data‐worth analysis

AbstractThese notes discuss the usefulness, limitations, and potential pitfalls of using sensitivity indices as a means to evaluate data worth and to guide the formulation and solution of inverse problems. A sensitivity analysis examines changes in model output variables with respect to changes in model input parameters. It appears straightforward to use this information to select influential parameters that should be subjected to estimation by inverse modeling and to identify the observations that contain information about these parameters and thus may be useful as calibration points. However, the results of such a sensitivity analysis do not account for parameter correlations and redundancies in observations and may not properly separate between calibration and prediction targets if used as criteria that guide inverse modeling; they may thus yield misleading recommendations about parameter identifiability and data worth. These issues (and some remedies) are discussed using an illustrative example, in which we examine the value of data sets potentially used for the calibration of a geothermal reservoir model. These notes highlight the importance of carefully formulating the objectives of a simulation study, which controls the setup of the inverse problem and related data needs.

Bayesian model averaging to explore the worth of data for soil‐plant model selection and prediction

AbstractA Bayesian model averaging (BMA) framework is presented to evaluate the worth of different observation types and experimental design options for (1) more confidence in model selection and (2) for increased predictive reliability. These two modeling tasks are handled separately because model selection aims at identifying the most appropriate model with respect to a given calibration data set, while predictive reliability aims at reducing uncertainty in model predictions through constraining the plausible range of both models and model parameters. For that purpose, we pursue an optimal design of measurement framework that is based on BMA and that considers uncertainty in parameters, measurements, and model structures. We apply this framework to select between four crop models (the vegetation components of CERES, SUCROS, GECROS, and SPASS), which are coupled to identical routines for simulating soil carbon and nitrogen turnover, soil heat and nitrogen transport, and soil water movement. An ensemble of parameter realizations was generated for each model using Monte‐Carlo simulation. We assess each model's plausibility by determining its posterior weight, which signifies the probability to have generated a given experimental data set. Several BMA analyses were conducted for different data packages with measurements of soil moisture, evapotranspiration (ETa), and leaf area index (LAI). The posterior weights resulting from the different BMA runs were compared to the weight distribution of a reference run with all data types to investigate the utility of different data packages and monitoring design options in identifying the most appropriate model in the ensemble. We found that different (combinations of) data types support different models and none of the four crop models outperforms all others under all data scenarios. The best model discrimination was observed for those data where the competing models disagree the most. The data worth for reducing prediction uncertainty depends on the prediction to be made. LAI data have the highest utility for predicting ETa, while soil moisture data are better for predicting soil water drainage. Our study illustrates, that BMA provides an objective framework for data worth analysis with respect to both model discrimination and model calibration for a wide range of applications.

Data Worth of Measurement Quantities and Uncertainty Considerations in Lithium-Ion Battery Modeling

Because of their favorable properties like, e.g., high energy density, lithium-ion batteries are ubiquitous in modern societies. Still, some security aspects, like thermal runaway, are not yet fully solved [1]. As a first step towards an early-alert system that can warn the user of an imminent thermal runaway we investigate some of the stochastic aspects of thermal battery model predictions.Numerical system modeling is inherently challenged by system uncertainties. These uncertainties stem from incomplete knowledge about parameter values, error-prone measurements and the imperfect mathematical description of the real system. The latter results in differences between predicted model output and the behavior of the real system. This phenomenon is called model structural error.We present a minimally intrusive methodology that allows to handle model errors of an existing thermal-electrochemical battery model in a stochastically rigorous way. Moreover, we investigate the explanatory power of different measurement quantities on the temperature profile inside of the battery.An existing, thermal-electrochemical battery model (DENIS [2]) of a LiFePO4 cell is extended by an additional, stochastically motivated model for the model structural error. The parameters for this error model are determined by means of a Monte Carlo maximum likelihood approach. That is, the residuals between experimental discharge curves at low temperatures and the model predictions are used to evaluate the likelihood of different error model parameter sets.Based on this stochastic system description a particle filter is implemented. Instead of a single model run, several parallel runs are computed. This yields a probability density function of the temperature distribution inside of the battery. Online measurement data can be used to update the system state and to decrease the uncertainty of future predictions. In comparison to Kalman filters, the use of a particle filter allows for a wide choice of error models beyond Gaussian and linear system assumptions.The development of this methodology is extended by a data worth analysis. By application of the PreDIA (“Preposterior Data Impact Assessor”) method [3] we investigate the explanatory power of different measurement quantities on the temperature profile inside of the battery. Voltage, discharge current and temperature measurements are considered and for the latter also different sensor locations are examined. The dynamics of the stochastic system description and the relative data worth of the different measurement quantities are interpreted and an outlook towards an early-alert system that builds upon the presented methodology is presented.[1] Bandhauer, Todd M., Srinivas Garimella, and Thomas F. Fuller. "A critical review of thermal issues in lithium-ion batteries." Journal of the Electrochemical Society 158.3 (2011): R1-R25.[2] Neidhardt, Jonathan P., et al. "A Flexible Framework for Modeling Multiple Solid, Liquid and Gaseous Phases in Batteries and Fuel Cells." Journal of The Electrochemical Society 159.9 (2012): A1528-A1542.[3] Leube, P. C., A. Geiges, and W. Nowak. "Bayesian assessment of the expected data impact on prediction confidence in optimal sampling design." Water Resources Research 48.2 (2012).

Using predictive uncertainty analysis to optimise tracer test design and data acquisition

Summary Tracer injection tests are regularly-used tools to identify and characterise flow and transport mechanisms in aquifers. Examples of practical applications are manifold and include, among others, managed aquifer recharge schemes, aquifer thermal energy storage systems and, increasingly important, the disposal of produced water from oil and shale gas wells. The hydrogeological and geochemical data collected during the injection tests are often employed to assess the potential impacts of injection on receptors such as drinking water wells and regularly serve as a basis for the development of conceptual and numerical models that underpin the prediction of potential impacts. As all field tracer injection tests impose substantial logistical and financial efforts, it is crucial to develop a solid a-priori understanding of the value of the various monitoring data to select monitoring strategies which provide the greatest return on investment. In this study, we demonstrate the ability of linear predictive uncertainty analysis (i.e. “data worth analysis”) to quantify the usefulness of different tracer types (bromide, temperature, methane and chloride as examples) and head measurements in the context of a field-scale aquifer injection trial of coal seam gas (CSG) co-produced water. Data worth was evaluated in terms of tracer type, in terms of tracer test design (e.g., injection rate, duration of test and the applied measurement frequency) and monitoring disposition to increase the reliability of injection impact assessments. This was followed by an uncertainty targeted Pareto analysis, which allowed the interdependencies of cost and predictive reliability for alternative monitoring campaigns to be compared directly. For the evaluated injection test, the data worth analysis assessed bromide as superior to head data and all other tracers during early sampling times. However, with time, chloride became a more suitable tracer to constrain simulations of physical transport processes, followed by methane. Temperature data was assessed as the least informative of the solute tracers. However, taking costs of data acquisition into account, it could be shown that temperature data when used in conjunction with other tracers was a valuable and cost-effective marker species due to temperatures low cost to worth ratio. In contrast, the high costs of acquisition of methane data compared to its muted worth, highlighted methanes unfavourable return on investment. Areas of optimal monitoring bore position as well as optimal numbers of bores for the investigated injection site were also established. The proposed tracer test optimisation is done through the application of common use groundwater flow and transport models in conjunction with publicly available tools for predictive uncertainty analysis to provide modelers and practitioners with a powerful yet efficient and cost effective tool which is generally applicable and easily transferrable from the present study to many applications beyond the case study of injection of treated CSG produced water.

Multimodel Bayesian analysis of data-worth applied to unsaturated fractured tuffs

To manage water resource and environmental systems effectively requires suitable data. The worth of collecting such data depends on their potential benefit and cost, including the expected cost (risk) of failing to take an appropriate decision. Evaluating this risk calls for a probabilistic approach to data-worth assessment. Recently we [39] developed a multimodel approach to optimum value-of-information or data-worth analysis based on model averaging within a maximum likelihood Bayesian framework. Adopting a two-dimensional synthetic example, we implemented our approach using Monte Carlo (MC) simulations with and without lead order approximations, finding that the former approach was almost equally accurate but computationally more efficient. Here we apply our methodology to pneumatic permeability data from vertical and inclined boreholes drilled into unsaturated fractured tuff near Superior, Arizona. In an attempt to improve computational efficiency, we introduce three new approximations that require less computational effort and compare results with those obtained by the original Monte Carlo method. The first approximation disregards uncertainty in model parameter estimates, the second does so for estimates of potential new data, and the third disregards both uncertainties. We find that only the first approximation yields reliable quantitative assessments of reductions in predictive uncertainty brought about by the collection of new data. We conclude that, whereas parameter uncertainty may sometimes be disregarded for purposes of analyzing data worth, the same does not generally apply to uncertainty in estimates of potential new data.

Bayesian analysis of data-worth considering model and parameter uncertainties

The rational management of water resource systems requires an understanding of their response to existing and planned schemes of exploitation, pollution prevention and/or remediation. Such understanding requires the collection of data to help characterize the system and monitor its response to existing and future stresses. It also requires incorporating such data in models of system makeup, water flow and contaminant transport. As the collection of subsurface characterization and monitoring data is costly, it is imperative that the design of corresponding data collection schemes be cost-effective, i.e., that the expected benefit of new information exceed its cost. A major benefit of new data is its potential to help improve one’s understanding of the system, in large part through a reduction in model predictive uncertainty and corresponding risk of failure. Traditionally, value-of-information or data-worth analyses have relied on a single conceptual-mathematical model of site hydrology with prescribed parameters. Yet there is a growing recognition that ignoring model and parameter uncertainties render model predictions prone to statistical bias and underestimation of uncertainty. This has led to a recent emphasis on conducting hydrologic analyses and rendering corresponding predictions by means of multiple models. We describe a corresponding approach to data-worth analyses within a Bayesian model averaging (BMA) framework. We focus on a maximum likelihood version (MLBMA) of BMA which (a) is compatible with both deterministic and stochastic models, (b) admits but does not require prior information about the parameters, (c) is consistent with modern statistical methods of hydrologic model calibration, (d) allows approximating lead predictive moments of any model by linearization, and (e) updates model posterior probabilities as well as parameter estimates on the basis of potential new data both before and after such data become actually available. We describe both the BMA and MLBMA versions theoretically and implement MLBMA computationally on a synthetic example with and without linearization.

Data-worth analysis for multiobjective optimal design of pump-and-treat remediation systems

The design and the management of pump-and-treat (PAT) remediation systems for contaminated aquifers under uncertain hydrogeological settings and parameters often involve decisions that trade off cost optimality against reliability. Both design objectives can be improved by planning site characterization programs that reduce subsurface parameter uncertainty. However, the cost for subsurface investigation often weighs heavily upon the budget of the remedial action and must thus be taken into account in the trade-off analysis. In this paper, we develop a stochastic data-worth framework with the purpose of estimating the economic opportunity of subsurface investigation programs. Since the spatial distribution of hydraulic conductivity is most often the major source of uncertainty, we focus on the direct sampling of hydraulic conductivity at prescribed locations of the aquifer. The data worth of hydraulic conductivity measurements is estimated from the reduction of the overall management cost ensuing from the reduction in parameter uncertainty obtained from sampling. The overall cost is estimated as the expected value of the cost of installing and operating the PAT system plus penalties incurred due to violations of cleanup goals and constraints. The crucial point of the data-worth framework is represented by the so-called pre-posterior analysis. Here, the tradeoff between decreasing overall costs and increasing site-investigation budgets is assessed to determine a management strategy proposed on the basis of the information available at the start of remediation. The goal of the pre-posterior analysis is to indicate whether the proposed management strategy should be implemented as is, or re-designed on the basis of additional data collected with a particular site-investigation program. The study indicates that the value of information is ultimately related to the estimates of cleanup target violations and decision makers’ degree of risk-aversion.

Calculating the optimal number of contaminant samples by means of data worth analysis

AbstractThis is a study of Bayesian data worth analysis in environmental clean‐up applications. Its focus is on calculating the worth of simultaneously taking several (soil) samples in a small homogeneous area and on finding the optimal number of samples to take, by relating the reduction in risk cost from sampling, that is, data worth, to the cost of the samples. Even though the cost of one sample may be higher than the risk cost reduction it provides, this study shows that several samples may be cost‐efficient. This is mainly due to two factors: one is that the unit sample cost often decreases as the number of samples increase; another, more important, factor is that the data worth of several samples typically is higher than the worth of fewer samples. Copyright © 2006 John Wiley & Sons, Ltd.

Uncertainty and data worth analysis for the hydraulic design of funnel‐and‐gate systems in heterogeneous aquifers

Hydraulic failure of a funnel‐and‐gate system may occur when the contaminant plume bypasses the funnels rather than being captured by the gate. We analyze the uncertainty of capturing the plumes by funnel‐and‐gate systems in heterogeneous aquifers. Restricting the analysis to two‐dimensional, steady state flow, we characterize plume capture by the values of the stream function at the boundaries of the plume and the funnels. On the basis of the covariance of the log conductivity distribution we compute the covariance matrix of the relevant stream function values by a matrix‐based first‐order second‐moment method, making use of efficient matrix‐multiplication techniques. From the covariance matrix of stream function values, we can approximate the probability that the plume is bypassing the funnels. We condition the log conductivity field to measurements of the log conductivity and the hydraulic head. Prior to performing additional measurements, we estimate their worth by the expected reduction in the variance of stream function differences. In an application to a hypothetical aquifer, we demonstrate that our method of uncertainty propagation and our sampling strategy enable us to discriminate between cases of success and failure of funnel‐and‐gate systems with a small number of additional samples.

Using influence diagrams for data worth analysis

Decision-making under uncertainty describes most environmental remediation and waste management problems. Inherent limitations in knowledge concerning contaminants, environmental fate and transport, remedies, and risks force decision-makers to select a course of action based on uncertain and incomplete information. Because uncertainties can be reduced by collecting additional data., uncertainty and sensitivity analysis techniques have received considerable attention. When costs associated with reducing uncertainty are considered in a decision problem, the objective changes; rather than determine what data to collect to reduce overall uncertainty, the goal is to determine what data to collect to best differentiate between possible courses of action or decision alternatives. Environmental restoration and waste management requires cost-effective methods for characterization and monitoring, and these methods must also satisfy regulatory requirements. Characterization and monitoring activities imply that, sooner or later, a decision must be made about collecting new field data. Limited fiscal resources for data collection should be committed only to those data that have the most impact on the decision at lowest possible cost. Applying influence diagrams in combination with data worth analysis produces a method which not only satisfies these requirements but also gives rise to an intuitive representation of complex structures not possible in the more traditional decision tree representation. This paper demonstrates the use of influence diagrams in data worth analysis by applying to a monitor-and-treat problem often encountered in environmental decision problems.