Uncertainty Metrics Research Articles

A nearest neighbor-based active learning method and its application to time series classification

• Locally estimated prediction uncertainty and sampling utility metrics are introduced for active learning. • An effective batch-mode active learning algorithm is introduced based on the proposed metrics. • Experimental results on two time series data demonstrate the effectiveness of the proposed method. Although the one nearest neighbor approach is widely used in time series classification , its successful performance requires enough labeled data, which is often difficult to obtain due to a high labeling cost. This article considers a practical classification scenario in which labeled data are scant but unlabeled data are plenty, and a limited budget for the annotating task is provided. For an effective classification with limited resources, we propose a nearest neighbor-based sampling strategy for active learning. The proposed approach uses highly local information to measure the uncertainty and utility of an unlabeled instance and is applicable to extremely sparse labeled data. Furthermore, we extend the proposed approach to batch mode active learning to select a batch of informative samples at each sampling iteration. Experimental results on the WAFER and ECG5000 data sets demonstrate the effectiveness of the proposed algorithm as compared with other nearest neighbor-based approaches.

Uncertainty in runup predictions on natural beaches using XBeach nonhydrostatic

The wave time series that forces phase-resolving models is a source of model uncertainty that can propagate into wave runup predictions when the wave phase information is unknown. The effect of beach morphology on the propagation of this intrinsic uncertainty related to wave randomness (Ui) is largely unexplored. Here, we quantify the importance of uncertainty in wave runup at a dissipative (TS), intermediate (DU) and reflective (DE) beach using the phase-resolving model XBeach nonhydrostatic. Intrinsic uncertainty is evaluated with respect to the effect of free model parameters and of an evolving bed. Uncertainty contributions due to model equations and numerical methods are not assessed. The model is forced at each beach with 100 different wave time series corresponding to the same storm condition. Computed uncertainty metrics reveal that Ui is most important at TS, where extreme runup (R2%) of a single simulation can deviate up to 31% from the 100-simulation ensemble average. Intrasite differences in R2% can be attributed to the predominance of low-frequency swash (SLF), which is more sensitive to Ui than high-frequency swash and setup. Additional simulations using the suggested range of breaking and bed friction coefficients demonstrate that Ui produces more variability in SLF than the two most important model parameters, whereas the parametric range is more important for high-frequency wave height. Furthermore, morphodynamic simulations show that Ui produces a larger variability in R2% and SLF than an evolving bed under a 7-hr storm. Not surprisingly, Ui causes larger variability in morphologic than hydrodynamic predictions due to morphologic feedback. Trends can reverse from swash zone accretion to erosion (TS) by using a wave time series with differently distributed phase. Ensemble simulations are recommended for forecasts based on wave time series with randomly distributed phase.

Antarctic Geothermal Heat Flow Model: Aq1

AbstractWe present a refined map of geothermal heat flow for Antarctica, Aq1, based on multiple observables. The map is generated using a similarity detection approach by attributing observables from geophysics and geology to a large number of high‐quality heat flow values (N = 5,792) from other continents. Observables from global, continental, and regional datasets for Antarctica are used with a weighting function that allows the degree of similarity to increase with proximity and how similar the observables are. The similarity detection parameters are optimized through cross correlation. For each grid cell in Antarctica, a weighted average heat flow value and uncertainty metrics are calculated. The Aq1 model provides higher spatial resolution in comparison to previous results. High heat flow is shown in the Thwaites Glacier region, with local values over 150 mW m−2. We also map elevated values over 80 mW m−2 in Palmer Land, Marie Byrd Land, Victoria Land and Queen Mary Land. Very low heat flow is shown in the interior of Wilkes Land and Coats Land, with values under 40 mW m−2. We anticipate that the new geothermal heat flow map, Aq1, and its uncertainty bounds will find extended use in providing boundary conditions for ice sheet modeling and understanding the interactions between the cryosphere and solid Earth. The computational framework and open architecture allow for the model to be reproduced, adapted and updated with additional data, or model subsets to be output at higher resolution for regional studies.

Using Spatial Validity and Uncertainty Metrics to Determine the Relative Suitability of Alternative Suites of Oceanographic Data for Seabed Biotope Prediction. A Case Study from the Barents Sea, Norway

The use of habitat distribution models (HDMs) has become common in benthic habitat mapping for combining limited seabed observations with full-coverage environmental data to produce classified maps showing predicted habitat distribution for an entire study area. However, relatively few HDMs include oceanographic predictors, or present spatial validity or uncertainty analyses to support the classified predictions. Without reference studies it can be challenging to assess which type of oceanographic model data should be used, or developed, for this purpose. In this study, we compare biotope maps built using predictor variable suites from three different oceanographic models with differing levels of detail on near-bottom conditions. These results are compared with a baseline model without oceanographic predictors. We use associated spatial validity and uncertainty analyses to assess which oceanographic data may be best suited to biotope mapping. Our results show how spatial validity and uncertainty metrics capture differences between HDM outputs which are otherwise not apparent from standard non-spatial accuracy assessments or the classified maps themselves. We conclude that biotope HDMs incorporating high-resolution, preferably bottom-optimised, oceanography data can best minimise spatial uncertainty and maximise spatial validity. Furthermore, our results suggest that incorporating coarser oceanographic data may lead to more uncertainty than omitting such data.

On the Usefulness of Uncertainty Sentiment in Twitter for Financial Markets

We examine the volatility dynamics and predictive potency of Twitter-based uncertainty metrics by [1] against major financial assets. Employing a latent factor structure model driven by stochastic volatility in high dimensions, we document a lack of co-movement between uncertainty sentiment on Twitter and market volatility. We also record diminutive predictive power of the same indices, especially in comparison with the more established fear index VIX.

Economic-State Variation in Uncertainty-Yield Dynamics

Abstract We show there is a much stronger negative, dynamic relation between changes in economic uncertainty and Treasury yields over weaker economic times since at least 1990. We document this economic-state variation in uncertainty-yield dynamics for weekly and monthly change horizons, for nominal yields and real-yield proxies, for multiple economic-state identification methods, and for different economic uncertainty metrics. We present additional findings that suggest short-term fluctuations in precautionary-savings and consumption-smoothing forces are more impactful on interest rate dynamics during weaker economic times, especially relying on surveys of expected economic growth and inflation. Received February 8, 2019; editorial decision August 24, 2020 by Editor Nikolai Roussanov. Authors have furnished an Internet Appendix, which is available on the Oxford University Press Web site next to the link to the final published paper online.

HISTORICAL FORECASTING OF INTEREST RATE MEAN AND VOLATILITY OF THE UNITED STATES: IS THERE A ROLE OF UNCERTAINTY?

Uncertainty is known to have negative impact on financial markets through its effects on investors’ decisions. In the wake of the “Great Recession”, quite a few recent studies have highlighted the role of uncertainty in predicting in-sample movements of interest rates. Since in-sample predictability does not guarantee out-of-sample forecasting gains, in this paper, we used historical daily and monthly data for the US to forecast mean and volatility of interest rate. The results show that changes in uncertainty measure movements fail to add any significant statistical gains to the forecast of interest rates for the US. In other words, policy makers in the US are not likely to improve their accuracy of future movements of the policy rate’s mean and volatility by incorporating information derived from changes in metrics of uncertainty.

Active learning for segmentation based on Bayesian sample queries

Segmentation of anatomical structures is a fundamental image analysis task for many applications in the medical field. Deep learning methods have been shown to perform well, but for this purpose large numbers of manual annotations are needed in the first place, which necessitate prohibitive levels of resources that are often unavailable. In an active learning framework of selecting informed samples for manual labeling, expert clinician time for manual annotation can be optimally utilized, enabling the establishment of large labeled datasets for machine learning. In this paper, we propose a novel method that combines representativeness with uncertainty in order to estimate ideal samples to be annotated, iteratively from a given dataset. Our novel representativeness metric is based on Bayesian sampling, by using information-maximizing autoencoders. We conduct experiments on a shoulder magnetic resonance imaging (MRI) dataset for the segmentation of four musculoskeletal tissue classes. Quantitative results show that the annotation of representative samples selected by our proposed querying method yields an improved segmentation performance at each active learning iteration, compared to a baseline method that also employs uncertainty and representativeness metrics. For instance, with only 10% of the dataset annotated, our method reaches within 5% of Dice score expected from the upper bound scenario of all the dataset given as annotated (an impractical scenario due to resource constraints), and this gap drops down to a mere 2% when less than a fifth of the dataset samples are annotated. Such active learning approach to selecting samples to annotate enables an optimal use of the expert clinician time, being often the bottleneck in realizing machine learning solutions in medicine.

Variability in Observation-based Onroad Emission Constraints from a Near-road Environment.

This study uses Las Vegas near-road measurements of carbon monoxide (CO) and nitrogen oxides (NOx) to test the consistency of onroad emission constraint methodologies. We derive commonly used CO to NOx ratios (ΔCO:ΔNOx) from cross-road gradients and from linear regression using ordinary least squares (OLS) regression and orthogonal regression. The CO to NOx ratios are used to infer NOx emission adjustments for a priori emissions estimates from EPA’s MOtor Vehicle Emissions Simulator (MOVES) model assuming unbiased CO. The assumption of unbiased CO emissions may not be appropriate in many circumstances but was implemented in this analysis to illustrate the range of NOx scaling factors that can be inferred based on choice of methods and monitor distance alone. For the nearest road estimates (25m), the cross-road gradient and ordinary least squares (OLS) agree with each other and are not statistically different from the MOVES-based emission estimate while ΔCO:ΔNOx from orthogonal regression is significantly higher than the emitted ratio from MOVES. Using further downwind measurements (i.e., 115m and 300m) increases OLS and orthogonal regression estimates of ΔCO:ΔNOx but not cross-road gradient ΔCO:ΔNOx. The inferred NOx emissions depend on the observation-based method, as well as the distance of the measurements from the roadway and can suggest either that MOVES NOx emissions are unbiased or that they should be adjusted downward by between 10% and 47%. The sensitivity of observation-based ΔCO:ΔNOx estimates to the selected monitor location and to the calculation method characterize the inherent uncertainty of these methods that cannot be derived from traditional standard-error based uncertainty metrics.

An ecological approach to climate change-informed tree species selection for reforestation

Accounting for climate change in reforestation practices has the potential to be one of the most efficacious adaptation strategies for maintaining future forest ecosystem services. There is a rich literature projecting spatial shifts in climatic suitability for tree species and strong scientific evidence for the necessity of assisted migration. However, there has been limited translation of this research into operational reforestation, due in part to mismatches to the information needs of practitioners. Here, we describe a practitioner-focused climate change informed tree species selection (CCISS) model to support reforestation decisions in British Columbia (BC). CCISS projects the climate change redistribution of bioclimate units from the multi-scaled Biogeoclimatic Ecosystem Classification (BEC) system with machine-learning for 90 modelled futures. It leverages the reforestation knowledge from BEC to make site-specific species projections of reforestation feasibility with climate change uncertainty metrics. We present 21st-century feasibility projections for a comprehensive set of tree species native to western North America. Some general trends are evident: augmentation of the number of feasible species in sub-boreal regions due to the rapid expansion of feasibility for temperate species; attrition at low elevations in southern BC due to declines in the feasibility of native species with little compensation by non-native species; and turnover at mid-elevations as declining feasibility for subalpine species is compensated by uphill expansion of climatic feasibility for submontane species. Edaphic (soil) factors are important; feasibility declines are higher on relatively dry sites than on wetter sites for most species. Our analysis emphasizes that changes in feasibility are species-specific, spatially variable, and influenced by edaphic site factors. By employing the multi-scaled BEC system that currently informs operational reforestation, CCISS facilitates translation of research into actionable guidance for practitioners.

An Artificial Intelligence framework for bidding optimization with uncertainty in multiple frequency reserve markets

The global ambitions of a carbon-neutral society necessitate a stable and robust smart grid that capitalizes on frequency reserves of renewable energy. Frequency reserves are resources that adjust power production or consumption in real time to react to a power grid frequency deviation. Revenue generation motivates the availability of these resources for managing such deviations. However, limited research has been conducted on data-driven decisions and optimal bidding strategies for trading such capacities in multiple frequency reserves markets. We address this limitation by making the following research contributions. Firstly, a generalized model is designed based on an extensive study of critical characteristics of global frequency reserves markets. Secondly, three bidding strategies are proposed, based on this market model, to capitalize on price peaks in multi-stage markets. Two strategies are proposed for non-reschedulable loads, in which case the bidding strategy aims to select the market with the highest anticipated price, and the third bidding strategy focuses on rescheduling loads to hours on which highest reserve market prices are anticipated. The third research contribution is an Artificial Intelligence (AI) based bidding optimization framework that implements these three strategies, with novel uncertainty metrics that supplement data-driven price prediction. Finally, the framework is evaluated empirically using a case study of multiple frequency reserves markets in Finland. The results from this evaluation confirm the effectiveness of the proposed bidding strategies and the AI-based bidding optimization framework in terms of cumulative revenue generation, leading to an increased availability of frequency reserves.

Spatial Aggregation Entropy: A Heterogeneity and Uncertainty Metric of Spatial Aggregation

The well-known modifiable areal unit problem (MAUP) has received much attention for a long time. There still exists, however, no unified understanding and solution to the MAUP. There is not even a statistic that quantifies the effects of the MAUP. This article proposes a new metric, namely, spatial aggregation entropy (SAE), based on which the spatial heterogeneity and uncertainty of aggregated data of spatial density are defined. The SAE quantifies the changes in spatial heterogeneity and uncertainty caused by spatial aggregation. The SAE is proven to satisfy scale additivity and spatial additivity, which makes it able to quantify the MAUP effects of spatial heterogeneity and uncertainty. Furthermore, spatial additivity extends SAE to local spatial aggregation entropy (LSAE). I distinguish two types of spatial density that are associated and unassociated with area and construct their SAE mathematical formulations. In the case study, the population density and proportion of Wayne County and the state of California are explored. I calculate the SAE and LSAE of transscale spatial aggregations for the studied attributes to demonstrate their validity. The thematic maps of LSAE are made to illustrate the distribution of local spatial heterogeneity changes. Furthermore, the article compares spatial heterogeneity of the population density with its Moran’s I at different scales. Both theoretical and case studies demonstrate that the SAE could well measure spatial heterogeneity and uncertainty changes of the spatial aggregation that reflect their MAUP effects.

BEAVRS: An integral full core multi-physics PWR benchmark with measurements and uncertainties

The BEAVRS benchmark was proposed in 2013 to serve as a non-proprietary benchmark based on measured reactor data to validate high-fidelity reactor simulation methods. In its third version release, the benchmark now includes an open source repository to provide users all source files related to BEAVRS. This version will also contain complete documentation of the uncertainty quantification work conducted. As part of this process, this paper elaborates on a systematic approach to quantifying the uncertainty that arises from using radial tilt-corrected data to compare BEAVRS data to simulated data. These uncertainty metrics are combined with model fitting uncertainties from fitting simulation model trends to BEAVRS data in order to compute overall time-dependent uncertainty of fission reaction rates. It is found that axially-integrated fission chamber time-dependent 95% uncertainty values are 1.8% and 1.9% for Cycle 1 and Cycle 2 respectively, which match well with results using traditional methods for uncertainty quantification.

Active and incremental learning for semantic ALS point cloud segmentation

Supervised training of a deep neural network for semantic segmentation of point clouds requires a large amount of labelled data. Nowadays, it is easy to acquire a huge number of points with high density in large-scale areas using current LiDAR and photogrammetric techniques. However it is extremely time-consuming to manually label point clouds for model training. In this paper, we propose an active and incremental learning strategy to iteratively query informative point cloud data for manual annotation and the model is continuously trained to adapt to the newly labelled samples in each iteration. We evaluate the data informativeness step by step and effectively and incrementally enrich the model knowledge. The data informativeness is estimated by two data dependent uncertainty metrics (point entropy and segment entropy) and one model dependent metric (mutual information). The proposed methods are tested on two datasets. The results indicate the proposed uncertainty metrics can enrich current model knowledge by selecting informative samples, such as considering points with difficult class labels and choosing target objects with various geometries in the labelled training pool. Compared to random selection, our metrics provide valuable information to significantly reduce the labelled training samples. In contrast with training from scratch, the incremental fine-tuning strategy significantly save the training time.

CLIMCAPS observing capability for temperature, moisture, and trace gases from AIRS/AMSU and CrIS/ATMS

Abstract. The Community Long-term Infrared Microwave Combined Atmospheric Product System (CLIMCAPS) retrieves vertical profiles of temperature, water vapor, greenhouse and pollutant gases, and cloud properties from measurements made by infrared and microwave instruments on polar-orbiting satellites. These are AIRS/AMSU on Aqua and CrIS/ATMS on Suomi NPP and NOAA20; together they span nearly 2 decades of daily observations (2002 to present) that can help characterize diurnal and seasonal atmospheric processes from different time periods or regions across the globe. While the measurements are consistent, their information content varies due to uncertainty stemming from (i) the observing system (e.g., instrument type and noise, choice of inversion method, algorithmic implementation, and assumptions) and (ii) localized conditions (e.g., presence of clouds, rate of temperature change with pressure, amount of water vapor, and surface type). CLIMCAPS quantifies, propagates, and reports all known sources of uncertainty as thoroughly as possible so that its retrieval products have value in climate science and applications. In this paper we characterize the CLIMCAPS version 2.0 system and diagnose its observing capability (ability to retrieve information accurately and consistently over time and space) for seven atmospheric variables – temperature, H2O, CO, O3, CO2, HNO3, and CH4 – from two satellite platforms, Aqua and NOAA20. We illustrate how CLIMCAPS observing capability varies spatially, from scene to scene, and latitudinally across the globe. We conclude with a discussion of how CLIMCAPS uncertainty metrics can be used in diagnosing its retrievals to promote understanding of the observing system and the atmosphere it measures.

Uncertainty‐assisted deep vision structural health monitoring

AbstractComputer vision leveraging deep learning has achieved significant success in the last decade. Despite the promising performance of the existing deep vision inspection models, the extent of models’ reliability remains unknown. Structural health monitoring (SHM) is a crucial task for the safety and sustainability of structures, and thus, prediction mistakes can have fatal outcomes. In this paper, we use Bayesian inference for deep vision SHM models where uncertainty can be quantified using the Monte Carlo dropout sampling. Three independent case studies for cracks, local damage identification, and bridge component detection are investigated using Bayesian inference. Aside from better prediction results, the two uncertainty metrics, variations in softmax probability and entropy, are shown to have good correlations with misclassifications. However, modifying the decision or triggering human intervention can be challenging based on raw uncertainty outputs. Therefore, the concept of surrogate models is proposed to develop the models for uncertainty‐assisted segmentation and prediction quality tagging. The former refines the segmentation mask and the latter is used to trigger human interventions. The proposed framework can be applied to future deep vision SHM frameworks to incorporate model uncertainty in the inspection processes.

Uncertainty in temperature and sea level datasets for the Pleistocene glacial cycles: Implications for thermal state of the subsea sediments

Abstract Temperature and sea level changes in the Pleistocene are uncertain. This leads to uncertainty in the associated response of the thermal state of the subsea sediments. We quantified the upper bound of the latter uncertainty in idealised simulations with a model for thermophysical processes in the sediments. At the coast and at the shallow and intermediate–depth shelves and except during relatively isolated time intervals, this bound for permafrost base depth and for the methane hydrate stability zone (MHSZ) characteristics (depth of its bottom boundary and its thickness) is ≤45% provided that the geothermal heat flux (GHF) is not larger than 80 mW m−2. These values are much smaller than the uncertainty metrics for the forcing data, which are typically ≥65%. However, for the intermediate shelf with a larger geothermal heat flux and for the deep shelf irrespective of GHF, different forcing time series may even lead to qualitatively different behaviour of the sediment thermophysical characteristics. We found that prescription of sea level changes plays a crucial role in uncertainty of the simulated subsea permafrost and MHSZ in the deep shelf sediments. In addition, we also quantified uncertainty for estimated apparent response time scales. The relative uncertainty for permafrost base depth and hydrate stability zone thickness time scales is ≤20% for most cases. We found no systematic dependence of our results on accounting for millennium–scale temperature variability provided that timescales of the order of 104 yr are resolved by forcing datasets.

A new uncertainty estimation approach with multiple datasets and implementation for various precipitation products

Abstract. Ensemble estimates based on multiple datasets are frequently applied once many datasets are available for the same climatic variable. An uncertainty estimate based on the difference between the ensemble datasets is always provided along with the ensemble mean estimate to show to what extent the ensemble members are consistent with each other. However, one fundamental flaw of classic uncertainty estimates is that only the uncertainty in one dimension (either the temporal variability or the spatial heterogeneity) can be considered, whereas the variation along the other dimension is dismissed due to limitations in algorithms for classic uncertainty estimates, resulting in an incomplete assessment of the uncertainties. This study introduces a three-dimensional variance partitioning approach and proposes a new uncertainty estimation (Ue) that includes the data uncertainties in both spatiotemporal scales. The new approach avoids pre-averaging in either of the spatiotemporal dimensions and, as a result, the Ue estimate is around 20 % higher than the classic uncertainty metrics. The deviation of Ue from the classic metrics is apparent for regions with strong spatial heterogeneity and where the variations significantly differ in temporal and spatial scales. This shows that classic metrics underestimate the uncertainty through averaging, which means a loss of information in the variations across spatiotemporal scales. Decomposing the formula for Ue shows that Ue has integrated four different variations across the ensemble dataset members, while only two of the components are represented in the classic uncertainty estimates. This analysis of the decomposition explains the correlation as well as the differences between the newly proposed Ue and the two classic uncertainty metrics. The new approach is implemented and analysed with multiple precipitation products of different types (e.g. gauge-based products, merged products and GCMs) which contain different sources of uncertainties with different magnitudes. Ue of the gauge-based precipitation products is the smallest, while Ue of the other products is generally larger because other uncertainty sources are included and the constraints of the observations are not as strong as in gauge-based products. This new three-dimensional approach is flexible in its structure and particularly suitable for a comprehensive assessment of multiple datasets over large regions within any given period.

Uncertainty Quantification in Planetary Thermal History Models: Implications for Hypotheses Discrimination and Habitability Modeling

Abstract Multiple hypotheses/models have been put forward regarding Earth’s cooling history. Searching for life beyond Earth has brought these models into a new light as they connect to an energy source that life can tap. Discriminating between different cooling models and adapting them to aid in the assessment of planetary habitability has been hampered by a lack of uncertainty quantification. Here, we provide an uncertainty quantification that accounts for a range of interconnected model uncertainties. This involved calculating over a million individual model evolutions to determine uncertainty metrics. Accounting for uncertainties means that model results must be evaluated in a probabilistic sense, even though the underlying models are deterministic. The uncertainty analysis was used to quantify the degree to which different models can satisfy observational constraints on the Earth’s cooling. For the Earth’s cooling history, uncertainty leads to ambiguity—multiple models, based on different hypotheses, can match observations. This has implications for using such models to forecast conditions for exoplanets that share Earth characteristics but are older than the Earth, i.e., ambiguity has implications for modeling the long-term life potential of terrestrial planets. Even for the most earthlike planet we know of, the Earth itself, model uncertainty and ambiguity leads to large forecast spreads. Given that Earth has the best data constraints, we should expect larger spreads for models of terrestrial planets, in general. The uncertainty analysis provided here can be expanded by coupling planetary cooling models to climate models and propagating uncertainty between them to assess habitability from a probabilistic view.

Forecast uncertainty‐based performance degradation diagnosis of solar PV systems

In this study, the authors are interested in estimating how much a PV system underperforms than expected by exploiting forecast uncertainty. For this, they first study a forecast accuracy-related forecast uncertainty metric using the ensemble method based on the dropout technique, which is widely used in deep learning forecasting models. Given the forecast accuracy-related uncertainty metric, the rationale of the authors' approach is that forecast accuracy is likely to decrease compared to the normal case of similar uncertainty metric values if any performance degradation happens. It is because similar uncertainty metric values are likely to show similar forecast accuracy. Therefore, they generate a standard table by simulating possible performance degradation cases and conduct the performance degradation diagnosis by looking up the standard table based on the uncertainty metric. From the experiments, in the case of persistent degradation, they show that their approach estimates the performance degradation with the estimation error of around 1% while an uncertainty-unaware approach shows the estimation error of up to 5%. In the case of temporal degradation, their approach shows the estimation error of around 3%, while the uncertainty-unaware approach does not show meaningful result.