Uncertainty Metrics Research Articles

Dose planning variations related to delineation variations in MRI-guided brachytherapy for locally advanced cervical cancer

PurposeTo examine the variability in prescribed dose due to contouring variations in intracavitary image-guided adaptive brachytherapy for cervical cancer. To identify correlations between dosimetric outcomes and delineation uncertainty metrics. Methods and MaterialsA data set from an EMBRACE sub-study on contouring uncertainties was used, consisting of magnetic resonance images of six patients with cervical cancer delineated by 10 experienced observers (target volumes and organs at risk). Two gold standard contours were generated, an expert consensus and the simultaneous truth and performance level estimation. Plans were individually optimised to all of the contour sets (12 in total). Plans were applied to the gold standard contour sets, and dose volume histogram parameters including D90, D98 and D2cm3 were determined. The variability between plans was assessed. Dose volume histogram parameters and delineation uncertainty metrics were correlated using the Spearman's non-parametric rank correlation. ResultsThere is a dosimetric variability between observers, patients and the gold standard contour used for analysis. Approximately 3 Gy D90 EQD210 variability (SD) was observed for the CTVHR and 1.2-3.6 Gy D2cm3 EQD23 for the organs at risk. The maximum geometric dimensions of the delineations are most commonly correlated with dosimetry changes. Although the correlations are similar across gold standards, the direction of these correlations differs, indicating that the dosimetric outcomes are dependent on the contour that the plan is optimised to. ConclusionThis study highlights the dosimetric differences interobserver uncertainty in contouring can have for cervical cancer brachytherapy. The importance of carefully choosing a gold standard from which to benchmark is reiterated.

Spatial Flow-Field Approximation Using Few Thermodynamic Measurements—Part II: Uncertainty Assessments

AbstractIn this second part of our two-part paper, we provide a detailed, frequentist framework for propagating uncertainties within our multivariate linear least squares model. This permits us to quantify the impact of uncertainties in thermodynamic measurements—arising from calibrations and the data acquisition system—and the correlations therein, along with uncertainties in probe positions. We show how the former has a much larger effect (relatively) than uncertainties in probe placement. We use this non-deterministic framework to demonstrate why the well-worn metric for assessing spatial sampling uncertainty falls short of providing an accurate characterization of the effect of a few spatial measurements. In other words, it does not accurately describe the uncertainty associated with sampling a non-uniform pattern with a few circumferentially scattered rakes. To this end, we argue that our data-centric framework can offer a more rigorous characterization of this uncertainty. Our paper proposes two new uncertainty metrics: one for characterizing spatial sampling uncertainty and another for capturing the impact of measurement imprecision in individual probes. These metrics are rigorously derived in our paper and their ease in computation permits them to be widely adopted by the turbomachinery community for carrying out uncertainty assessments.

Robust identification of volumetric heat capacity and analysis of thermal response tests by Bayesian inference with correlated residuals

Bayesian inference has tremendous potential for thermal response test analysis, as it provides uncertainty metrics that are useful for the design of ground-coupled heat pump systems. The inference process is computationally heavy and has so far been limited to a few thermal parameters and under the unrealistic assumption of residuals’ independence. In this work, a new closed-form expression of the likelihood and an improved artificial neural network are used to speed up Bayesian inference and consider the strong temporal correlation of the residuals. This efficient strategy allowed the robust inference of the joint distribution of five parameters. Using data measured during a real test of 168 h, this work shows that it is possible to robustly identify the volumetric heat capacity of the ground and grout with an uncertainty of 16.3 and 13.8%, a significant improvement. For the specific data used, it is shown that with independence assumption, some parameters are clearly unrealistic, a problem not encountered when the correlation of the residuals is considered. The impact of the interpretation model, of the test duration and of the sampling frequency was also assessed and illustrated by the sizing of a ground heat exchanger. Results reveal that joint identification of some thermal parameters cannot be achieved reliably by the finite line source model, that duration of thermal response tests should be at least 72 h to avoid large uncertainties on the parameters, and that recording temperature every 2 min degrades the identification of the volumetric heat capacity.

Performance characteristics of automated clinical chemistry analyzers using commercial assay reagents contributing to quality assurance and clinical decision in a hospital laboratory

Background: Clinical laboratories provide essential diagnostic services that are essential in clinical decision making, contributing to the quality of healthcare. The performance of two Siemens ADVIA 1800 analyzers was characterized in a hospital Biochemistry laboratory in order to evaluate the analytical characteristics of such automated analyzer systems using nonoriginal assay reagents attempting to support laboratory quality service and crucial clinical decision making. Methods: We independently completed performance validation studies including trueness, precision, sensitivity as well as measurement of uncertainty and sigma metrics calculation for 25 biochemical parameters. Results: Trueness expressed as bias was less than 20% for both ADVIA 1800 analyzers. Within run and total precisions expressed as CV% were ≤9.85% on both analyzers for most parameters studied with few exceptions (Mg, TB, DB, Cl, HDL and UA) observed either in low or in high level samples and between the two analyzers. LoB, LoD and LoQ values produced by the two analyzers were comparable except Cl. Uncertainty values produced by the two analyzers were comparable with no significant differences. Quality performance of reagent assays was studied using the sigma metrics system. The sigma values were plotted on normalized method decision charts for graphical representation of assay performances for each analyzer. Conclusions: The two ADVIA systems, independently evaluated, showed consistent performance characteristics with certain discrepancies by several reagents. Sigma analysis was helpful for revealing the quality performance of non-original reagents supporting the need for strict assessment of quality assurance and in some instances optimization/improvement of assay methods.

Uncertainty Assessment in Multitemporal Land Use/Cover Mapping with Classification System Semantic Heterogeneity

Land use/cover (LUC) data are commonly relied on to provide land surface information in a variety of applications. However, the exchange and joint use of LUC information from different datasets can be challenging due to semantic differences between common classification systems (CSs). In this paper, we propose an uncertainty assessment schema to capture the semantic translation uncertainty between heterogeneous LUC CSs and evaluate the data label uncertainty of multitemporal LUC mapping results caused by uncertainty propagation. The semantic translation uncertainty between CSs is investigated using a dynamic semantic reference system (DSRS) model and semantic similarity analysis. An object-based unsupervised change detection algorithm is adopted to determine the probability of changes in land patches, and novel uncertainty metrics are proposed to estimate the patch label uncertainty in LUC maps. The proposed uncertainty assessment schema was validated via experiments on four LUC datasets, and the results confirmed that semantic uncertainty had great impact on data reliability and that the uncertainty metrics could be used in the development of uncertainty controls in multitemporal LUC mapping by referring to uncertainty assessment results. We anticipate our findings will be used to improve the applicability and interoperability of LUC data products.

Exhibiting Uncertainty: Visualizing Data Quality Indicators for Cultural Collections

Uncertainty is a standard condition under which large parts of art-historical and curatorial knowledge creation and communication are operating. In contrast to standard levels of data quality in non-historical research domains, historical object and knowledge collections contain substantial amounts of uncertain, ambiguous, contested, or plainly missing data. Visualization approaches and interfaces to cultural collections have started to represent data quality and uncertainty metrics, yet all existing work is limited to representations for isolated metadata dimensions only. With this article, we advocate for a more systematic, synoptic and self-conscious approach to uncertainty visualization for cultural collections. We introduce omnipresent types of data uncertainty and discuss reasons for their frequent omission by interfaces for galleries, libraries, archives and museums. On this basis we argue for a coordinated counter strategy for uncertainty visualization in this field, which will also raise the efforts going into complex interface design and conceptualization. Building on the PolyCube framework for collection visualization, we showcase how multiple uncertainty representation techniques can be assessed and coordinated in a multi-perspective environment. As for an outlook, we reflect on both the strengths and limitations of making the actual wealth of data quality questions transparent with regard to different target and user groups.

Monetary Policy, Uncertainty, and Domestic Shock Transmission to Capital Flow in Turkey

This paper reports findings on multiple important phenomena in capital flow literature: internal and external shock transmission to foreign direct investment and foreign portfolio investment. Operating on Turkey as a case study, I disaggregate quarterly gross capital inflows (FDI and FPI) and examine their time-varying responses to shocks in advanced economies' monetary policy and uncertainty surrounding it, Central Bank of Turkey's monetary policy, and several key domestic macroeconomic determinants. To this end, I employ a recently developed Structural Vector Autoregression technique that uses a data-driven non-Gaussian approach and find that FPI inflows are considerably more sensitive to permanent unexpected monetary policy shocks originating both at home and abroad. Complementary results from uncertainty metrics indicate-to varying extents-that Turkey is a beneficiary of capital flow amidst heightened policy tension in the US. I also visit the push-vs.-pull debate in capital flow literature via Forecast Error Variance Decomposition and discover that Turkey falls decisively in the pull category. The Turkey-specific indicators most crucial in attracting foreign capital are the performance of the currency and CBRT's policy positioning. Lastly, using signal-processing techniques in the time-frequency domain, I report that higher FPI inflows have generally led FDI inflows in times of global economic turbulence, but that phenomenon is undergoing a reversal in the recent past.

Lithological mapping in the Central African Copper Belt using Random Forests and clustering: Strategies for optimised results

The Trident project is located in the Domes region of the Central African Copper Belt and hosts a number of mineralised systems including the Sentinel (Ni) and Enterprise (Cu) deposits. The project has received extensive systematic geochemical soil sampling in addition to high resolution airborne geophysical coverage. This data-rich environment enables experimentation with machine learning strategies which aim to produce or refine geological maps from limited direct observations.In this study we present a series of three case studies that test lithological classification using the supervised Random Forests algorithm. These studies inform the situations encountered in mineral exploration including early stage lithology mapping and more mature stage map refinement. We also present a fourth study, using the unsupervised algorithms k-means and Self-Organising Maps, to identify clusters, potentially associated with lithology in absence of a priori geological information. Our case studies are most relevant to the situation where the geology of a prospect is largely concealed beneath extensive cover rocks, with some rock types being poorly expressed or even absent in outcrop.We find that sampling from limited outcrop produces a RF lithology prediction that is likely to be incorrect. We demonstrate that balancing sample size through a combination of decimation and bootstrapping can improve results. Additionally, we identify some important indicators in both the predicted geology and uncertainty metrics which could alert an explorer to an inability of their training data to make accurate predictions and to the presence of lithological classes not expressed in outcrop. Sampling from a mature lithology map enables further map refinement and acts as an objective audit of the existing product. Information entropy (H) is calculated as a metric to describe quantitatively the uncertainty associated with classification, provide valuable information on the geological complexity of the mapped region and highlight areas which are potentially misclassified. Clusters obtained using the k-means algorithm produced a result more consistent with lithology in this instance and was faster; however Self Organising Maps remains attractive due to the production of additional metrics to assess algorithm performance. Clustering could be used either in the development of a first pass interpretation, or in the critical appraisal and subsequent refinement of existing interpretations.

Quantifying statistical uncertainty in metrics of sleep disordered breathing

BackgroundThe apnea-hypopnea index (AHI) (or one of its derivatives) is the primary clinical metric for characterizing sleep disordered breathing—the value of which with respect to a threshold determines severity of diagnosis and eligibility for treatment reimbursement. The index value, however, is taken as a perfect point estimate, with no measure of statistical uncertainty. Thus, current practice does not robustly account for variability in diagnosis/eligibility due to chance. In this paper, we quantify the statistical uncertainty associated with respiratory event indices for sleep disordered breathing and the effect of uncertainty on treatment eligibility. MethodsWe develop an empirical estimate of uncertainty using a non-parametric bootstrap on the interevent times, as well as a theoretical Poisson estimate reflecting the current formulation of the AHI. We then apply these methods to estimate AHI uncertainty for 2049 subjects (954/1095 M/F, age: mean 69 ± 9.1) from the Multi-Ethnic Study of Atherosclerosis (MESA). Results and ConclusionsThe mean 95% empirical confidence interval width was 11.500 ± 6.208 events per hour and the mean 95% theoretical Poisson confidence interval width was 5.998 ± 2.897 events per hour, suggesting that uncertainty is likely a major confounding factor within the current diagnostic framework. Of the 278 subjects in the symptomatic population (ESS>10), 27% (76/278) had uncertain diagnoses given the 95% empirical confidence interval. Of the 2049 subjects in the full population, 43% (880/2049) had uncertain diagnoses given the 95% empirical confidence interval. The inclusion of subjects with uncertain diagnoses increases the number of eligible patients by 21.3% for the symptomatic population and by 84.8% for the full population. The exclusion of subjects with uncertain diagnoses given the 95% empirical confidence interval decreases the number of eligible patients by 12.4% for the symptomatic population and by 34.8% for full population. Additional analyses suggest that it is practically infeasible to gain diagnostic statistical significance through additional testing for a broad range of borderline cases. Overall, these results suggest that AHI uncertainty is a vital additional piece of information that would greatly benefit clinical practice, and that the inclusion of uncertainty in epidemiological analysis might help improve the ability for researchers to robustly link AHI with co-morbidities and long-term outcomes.

Quantification of Uncertainty in Peptide-MHC Binding Prediction Improves High-Affinity Peptide Selection for Therapeutic Design.

The computational identification of peptides that can bind the major histocompatibility complex (MHC) with high affinity is an essential step in developing personal immunotherapies and vaccines. We introduce PUFFIN, a deep residual network-based computational approach that quantifies uncertainty in peptide-MHC affinity prediction that arises from observational noise and the lack of relevant training examples. With PUFFIN's uncertainty metrics, we define binding likelihood, the probability a peptide binds to a given MHC allele at a specified affinity threshold. Compared to affinity point estimates, we find that binding likelihood correlates better with the observed affinity and reduces false positives inhigh-affinity peptide design. When applied to examine an existing peptide vaccine, PUFFIN identifies an alternative vaccine formulation with higher binding likelihood. PUFFIN is freely available for download at http://github.com/gifford-lab/PUFFIN.

Uncertainty in soil moisture retrievals: An ensemble approach using SMOS L-band microwave data

The uncertainty of surface soil moisture (SM) retrievals from satellite brightness temperature (TB) observations depends primarily on the choice of radiative transfer model (RTM) parameters, prior SM information and TB inputs. This paper studies the sensitivity of several established and experimental SM retrieval products from the Soil Moisture Ocean Salinity (SMOS) mission to these choices at 11 reference sites, located in 7 watersheds across the United States (US). Different RTM parameter sets cause large biases between retrievals. Whereas typical RTM parameter sets are calibrated for SM retrievals, it is shown that a parameter set carefully optimized for TB forward modeling can also be used for retrieving SM. It is also shown that the inclusion of dynamic prior SM estimates in a Bayesian retrieval scheme can strongly improve SM retrievals, regardless of the choice of RTM parameters. The second part of this paper evaluates the ensemble uncertainty metrics for SM retrievals obtained by propagating a wide range of RTM parameters through the RTM, and the relationship with time series metrics obtained by comparing SM retrievals with in situ data. As expected for bounded variables, the total spread in the ensemble SM retrievals is smallest for wet and dry SM values and highest for intermediate SM values. After removal of the strong long-term SM bias associated with the RTM parameter values for individual ensemble members, the remaining anomaly ensemble SM spread shows higher values when SM deviates further from its long-term mean SM. This reveals higher-order biases (e.g. differences in variances) in the retrieval error, which should be considered when characterizing retrieval error. The time-average anomaly ensemble SM spread of 0.037 m3/m3 approximates the actual time series unbiased root-mean-square-difference of 0.042 m3/m3 between ensemble mean retrievals and in situ data across the reference sites.

The carbon dioxide, methane and nitrogen high-pressure sorption properties of South African bituminous coals

In this paper, we report the CO2, CH4 and N2 sorption isotherms of four dry South African bituminous coals at pressures up to 16 MPa at 55 °C. The sorption capacities of the samples with respect to the adsorbate gases decreased in the order: CO2 > CH4 ≈ N2 by weight, and CO2 > CH4 > N2 by volume. A new model, based on a hybrid Dubinin-Radushkevich and Henry law approach (DR-HH) provided substantially better fits to the sorption isotherm data than the previously used modified DR (M-DR) model. Obtained uncertainty metrics show that the DR-HH model generally returned lower error sum of squares (ESS) and root mean square (RMS) residuals, and higher quality of fit (QOF) compared to the M-DR model. The net heat of sorption, βEs, of the samples for the three adsorbate gases were generally low (8.5–12.8 kJ/mol), but comparable to previous determinations of other coals, indicating that physisorption was the dominating sorption mechanism. The sorption capacities of the samples were found to be rank-dependent as they decreased with increasing vitrinite reflectance and elemental carbon content. The micropore properties of the samples as measured by both CO2 low-pressure gas adsorption (LPGA) and small angle X-ray scattering (SAXS), impacted the sorption properties of the sample more than both the mesopore and macropore properties determined from N2 LPGA, SAXS, and mercury intrusion porosimetry. The sorption capacities of the samples were found to increase with increasing lithotypes abundance, suggesting that lithotype bandings enhances either the fluid transport processes or the micropore properties of the coal matrix. In addition, it has been demonstrated that critical properties of the adsorbate gases influenced their sorption properties.

Validation of the Sentinel Simplified Level 2 Product Prototype Processor (SL2P) for mapping cropland biophysical variables using Sentinel-2/MSI and Landsat-8/OLI data

The Simplified Level 2 Product Prototype Processor (SL2P) for estimating Leaf Area index (LAI), fraction of vegetation cover (fCover) and Canopy Water Content (CWC) from Sentinel-2/MSI and Landsat-8/OLI data was validated over an agricultural region. In-situ data collected during the SMAP Validation Experiment 2016 field campaign were used as a reference. SL2P processor performance varied substantially between crop type and biophysical variable. Over all crops, SL2P underestimated in-situ LAI and CWC measurements when using either MSI (slope (bias) of 0.70 (−0.37) for LAI and 0.42 (−0.37 kg/m2) for CWC) or OLI (slope (bias) of 0.59 (−1.21) for LAI and 0.24 (−0.23 kg/m2) for CWC) data. The accuracy of SL2P fCover estimates, over all crops, was higher (slope (bias) of 0.99 (1.84%) using MSI and 0.93 (−3.75%) using OLI). The RMSE between biophysical variables estimated using SL2P from MSI (OLI) in comparison to in-situ data was 0.98 (1.63) for LAI, 11.39% (10.95%) for fCover and 0.66 kg/m2 (0.96 kg/m2) for CWC. Slightly better results are generally obtained using locally calibrated vegetation indices models, when compared to SL2P estimates using the corresponding sensor data. Uncertainty metrics of vegetation biophysical variables derived from both MSI and OLI, when compared to interpolated in-situ data time series, are found comparable to results obtained for cross-validation suggesting the possibility of using interpolated in-situ data time series for validating decametric resolution remote sensing products sparsely sampled in time.

Bayesian QuickNAT: Model uncertainty in deep whole-brain segmentation for structure-wise quality control

We introduce Bayesian QuickNAT for the automated quality control of whole-brain segmentation on MRI T1 scans. Next to the Bayesian fully convolutional neural network, we also present inherent measures of segmentation uncertainty that allow for quality control per brain structure. For estimating model uncertainty, we follow a Bayesian approach, wherein, Monte Carlo (MC) samples from the posterior distribution are generated by keeping the dropout layers active at test time. Entropy over the MC samples provides a voxel-wise model uncertainty map, whereas expectation over the MC predictions provides the final segmentation. Next to voxel-wise uncertainty, we introduce four metrics to quantify structure-wise uncertainty in segmentation for quality control. We report experiments on four out-of-sample datasets comprising of diverse age range, pathology and imaging artifacts. The proposed structure-wise uncertainty metrics are highly correlated with the Dice score estimated with manual annotation and therefore present an inherent measure of segmentation quality. In particular, the intersection over union over all the MC samples is a suitable proxy for the Dice score. In addition to quality control at scan-level, we propose to incorporate the structure-wise uncertainty as a measure of confidence to do reliable group analysis on large data repositories. We envisage that the introduced uncertainty metrics would help assess the fidelity of automated deep learning based segmentation methods for large-scale population studies, as they enable automated quality control and group analyses in processing large data repositories.

Transport Scaling of Dune Dimensions in Shallow Flows

AbstractDune dimensions in sand‐bedded rivers are often thought to scale with flow depth (h), with height (H) scaling as 1/6h and length (L) as 5h, even though substantial scatter about the relations has been observed. Transport stage has been shown to affect bedform geometry, but this control is often ignored in favor of depth scaling relations. Here, we use a series of flume experiments to systematically test controls on dune dimensions and variability. Experiments involved three sets of runs under five constant transport stages, ranging threshold to washout conditions, at three different flow depths. The mobile bed was repeatedly scanned during a 10‐hr equilibrium period to derive mean values and quantify the variability. The results show that dune‐depth scaling is not consistent because of a transport stage effect. Dune height increases with transport stage until a point when H decreases. Length remains nearly constant with transport stage until further increases in transport stage leads to lengthening. In general, dunes grow higher when significant bedload transport occurs but become flatter and longer in the presence of substantial suspension. Ultimately, dunes scale with transport stage, which is a function of slope, grain size, and h. The results are used to derive transport stage relations to guide predictions of dune dimensions in rivers and reconstructions of paleoflows based on dimensions estimated from cross strata. The relations incorporate the nonlinear response of dune dimensions with transport stage and provide metrics of uncertainty to include in predictions.

A Global MODIS Water Vapor Database for the Operational Atmospheric Correction of Historic and Recent Landsat Imagery

Analysis Ready Data (ARD) have undergone the most relevant pre-processing steps to satisfy most user demands. The freely available software FORCE (Framework for Operational Radiometric Correction for Environmental monitoring) is capable of generating Landsat ARD. An essential step of generating ARD is atmospheric correction, which requires water vapor data. FORCE relies on a water vapor database obtained from the Moderate Resolution Imaging Spectroradiometer (MODIS). However, two major drawbacks arise from this strategy: (1) The database has to be compiled for each study area prior to generating ARD; and (2) MODIS and Landsat commissioning dates are not well aligned. We have therefore compiled an application-ready global water vapor database to significantly increase the operational readiness of ARD production. The free dataset comprises daily water vapor data for February 2000 to July 2018 as well as a monthly climatology that is used if no daily value is available. We systematically assessed the impact of using this climatology on surface reflectance outputs. A global random sample of Landsat 5/7/8 imagery was processed twice (i) using daily water vapor (reference) and (ii) using the climatology (estimate), followed by computing accuracy, precision, and uncertainty (APU) metrics. All APU measures were well below specification, thus the fallback usage of the climatology is generally a sound strategy. Still, the tests revealed that some considerations need to be taken into account to help quantify which sensor, band, climate, and season are most or least affected by using a fallback climatology. The highest uncertainty and bias is found for Landsat 5, with progressive improvements towards newer sensors. The bias increases from dry to humid climates, whereas uncertainty increases from dry and tropic to temperate climates. Uncertainty is smallest during seasons with low variability, and is highest when atmospheric conditions progress from a dry to a wet season (and vice versa).

Uncertainty metric in model-based eddy current inversion using the adaptive Monte Carlo method

The accuracy of model-based inversion largely depends on the gap between simulated and observed results. However, the gap does not disappear even if a wide range of calibrations or compensations are made. In essence, the gap varies statistically due to random noises and poorly known parameters, causing predicted results to change in a random manner. In this work, an adaptive Monte Carlo method is proposed to evaluate the uncertainty of model-based inversion for eddy current nondestructive evaluation. Using the presented Monte Carlo method, the influences of excitation frequencies on uncertainty metric were identified, and uncertainty metric was calculated when the conductivity, thickness and liftoff distance were inferred for characterization of a plate. The presented Monte Carlo based method allows efficient and cost-effective assessment of uncertainty and could enable us to identify the factors that play a dominant role in the performances of model-based inversion.

A quantitative uncertainty metric controls error in neural network-driven chemical discovery.

Machine learning (ML) models, such as artificial neural networks, have emerged as a complement to high-throughput screening, enabling characterization of new compounds in seconds instead of hours. The promise of ML models to enable large-scale chemical space exploration can only be realized if it is straightforward to identify when molecules and materials are outside the model's domain of applicability. Established uncertainty metrics for neural network models are either costly to obtain (e.g., ensemble models) or rely on feature engineering (e.g., feature space distances), and each has limitations in estimating prediction errors for chemical space exploration. We introduce the distance to available data in the latent space of a neural network ML model as a low-cost, quantitative uncertainty metric that works for both inorganic and organic chemistry. The calibrated performance of this approach exceeds widely used uncertainty metrics and is readily applied to models of increasing complexity at no additional cost. Tightening latent distance cutoffs systematically drives down predicted model errors below training errors, thus enabling predictive error control in chemical discovery or identification of useful data points for active learning.

Relating Ocean Condition Forecasts to the Process of End-User Decision Making: A Case Study of the Oregon Commercial Fishing Community

This case study is in response to a recognized need to transform short-term regional ocean condition forecast information into useful data products for a range of end users, considering their perceptions of uncertainty and risk associated with these forecasts. It demonstrates the value of user engagement in achieving long-term goals for data providers. Commercial fishermen from Oregon are selected as key information users due to the physically risky and economically uncertain nature of their profession, their expertise at navigating the marine environment, and their important economic and cultural role at the Oregon coast. Semistructured interviews (n = 16) are used to clarify the processes that govern decision making, in terms of risk perception and comfort with uncertainty. The results characterize a community "mental model" in regard to ocean use and ocean forecasts. Findings reveal that commercial fishermen consume and interpret forecast data in a nonlinear fashion by combining multiple sources and data types and with a heavy reliance on real-time data. Our assessment is that improving accuracy at temporal and spatial scales that are relevant to decision making, improving the accessibility of forecasts, and increasing forecast lead time could potentially add more value to forecasts than quantifying and communicating the types of uncertainty metrics used within the scientific community.

Disentangling uncertainties from niche modeling in freshwater ecosystems

Predictions by ecological niche models (ENM) are affected by several sources of uncertainty including the modeling methods and type of variables employed. The predictive uncertainty has been often assessed in terrestrial ecosystems, but it is still unknown how freshwater variables affect the performance of ENMs, contributing to unreliable predictions for aquatic species. Here, we used the ecologically and economically relevant Amazon giant catfish (Brachyplatystoma filamentosum) as a model species to assess uncertainties on ENM predictions in freshwater ecosystems. Specifically, we assessed uncertainty by coupling ENM predictions using five modeling methods and four sets of freshwater environmental variables. Our results indicate that the modeling methods and secondarily the variables account for significant uncertainty in predicting freshwater species distribution using ENM. Areas with high environmental suitability such as the Amazon large rivers and nearby areas presented high uncertainty for the methods component, and lower uncertainties for freshwater variables. Moreover, freshwater variables accounted also for uncertainties in metrics of models’ performance. Whereas Topographic variables better predicted presences (higher sensitivities and lower omission errors), Land cover and Soil variables better predicted pseudo-absences (higher specificities and lower commission errors). The Hydroclimatic variables had better accuracy metrics values (AUC and TSS) but also generated the greatest uncertainty for the final models. When included variables from all groups, ENMs presented low uncertainties and good accuracy. In sum, our findings suggest the importance of measuring and mapping the uncertainties of ENMs using freshwater environmental database.