Underestimation Of Variance Research Articles

Effects of model-overfit on model-assisted forest inventory in boreal forests with remote sensing data

Abstract While remote sensing can be an effective tool in building a forest inventory, field measurements and model fitting can be both expensive and challenging. One strategy to reduce forest inventory costs is to leverage forest inventory models fitted to a different population (external models), although the effectiveness of external models is poorly understood. One concern is that models may predict well to the sample data, but poorly to the population—which is termed ‘overfitting’. The effect of overfit may be especially problematic in attempts to predict for a different population (a forest area not covered by any sample plots). Assessing overfit is difficult and its consequence for estimation are not well understood, especially in the context of prediction using external models. This study assesses how overfitting affects model-assisted forest inventory estimation when using internal and external models. We used field and remotely sensed data (Sentinel-2 images and airborne laser scanning data) from two forest areas in Finland. We evaluated four modeling approaches: ordinary least square regression (OLS), random forest, k-nearest neighbors, and gaussian process regression. Both analytical and bootstrap variance estimators were used to evaluate model-assisted estimation performance. Internal models, especially OLS, were the most affected by model overfitting, leading to bias in the population means and underestimation of variance. Estimates using external models provided unbiased means and realistic intervals except in the case of deliberate excessive overfitting. The bootstrap variance estimator was found to be more robust to overfit than the analytical variance estimator for the internal model, but was not helpful for the external model. Internal models should be parsimonious to generalize well to the population and avoid bias. The bootstrap estimator of variance is recommended for internal models, especially if there is concern about model overfitting.

Investigation on the production data frequency for assimilation with ensemble smoother

Underestimating posterior variances is a critical problem in using iterative ensemble smoothers for data assimilation in reservoir models. This issue limits the method’s capacity to assimilate large amounts of data and results in inaccurate evaluations of uncertainty for future performance predictions. This underestimation may be attributed to various factors, such as the parametrization of the problem, the characterization of the prior and data uncertainties and deficiencies in the ability of the forward model to reproduce the system’s dynamic behavior. Unfortunately, limitations inherent in the ensemble smoother formulation, mainly related to using small ensembles, also contribute to variance underestimation. To address this issue, we devise a synthetic reservoir data assimilation problem that isolates variance underestimation to the ensemble smoother. Using this test problem, we explore the impact of altering the production data setup to counteract the premature loss of ensemble variance. Our tests indicate that reducing data frequency is a relatively straightforward and robust approach to implement in practice. In addition, we propose implementing a distinct truncation scheme for singular values, which helps to mitigate the undesirable variance loss. Furthermore, we present six supplementary test problems, including four actual field cases, to provide additional empirical evidence that it is possible to reduce the variance loss without compromising the data-match quality.

AutoQS v1: automatic parametrization of QuickSampling based on training images analysis

Abstract. Multiple-point geostatistics are widely used to simulate complex spatial structures based on a training image. The practical applicability of these methods relies on the possibility of finding optimal training images and parametrization of the simulation algorithms. While methods for automatically selecting training images are available, parametrization can be cumbersome. Here, we propose to find an optimal set of parameters using only the training image as input. The difference between this and previous work that used parametrization optimization is that it does not require the definition of an objective function. Our approach is based on the analysis of the errors that occur when filling artificially constructed patterns that have been borrowed from the training image. Its main advantage is to eliminate the risk of overfitting an objective function, which may result in variance underestimation or in verbatim copy of the training image. Since it is not based on optimization, our approach finds a set of acceptable parameters in a predictable manner by using the knowledge and understanding of how the simulation algorithms work. The technique is explored in the context of the recently developed QuickSampling algorithm, but it can be easily adapted to other pixel-based multiple-point statistics algorithms using pattern matching, such as direct sampling or single normal equation simulation (SNESIM).

Bayesian Analysis of Multivariate Matched Proportions with Sparse Response

Multivariate matched proportions (MMP) data appear in a variety of contexts including post-market surveillance of adverse events in pharmaceuticals, disease classification, and agreement between care providers. It consists of multiple sets of paired binary measurements taken on the same subject. While recent work proposes methods to address the complexities of MMP data, the issue of sparse response, where no or very few “yes” responses are recorded for one or more sets, is unaddressed. The presence of sparse response sets results in the underestimation of variance components, loss of coverage, and lowered power in existing methods. Bayesian methods, which have not previously been considered for MMP data, provide a useful framework when sparse responses are present. In particular, the Bayesian probit model in combination with mean model prior specifications provides an elegant solution to the problem of variance underestimation. We examine a multivariate probit-based approach using hierarchical horseshoe-like priors along with a Bayesian functional principal component analysis (FPCA) to model the latent covariance. We show that our approach performs well on MMP data with sparse responses and outperforms existing methods. In a re-examination of a study on the system of care (SOC) framework for children with mental and behavioral disorders, we are able to provide a more complete picture of the relationships in the data. Our analysis provides additional insights into the functioning on the SOC that a previous univariate analysis missed.

Exploring the effects of physical and social networks on urban water system’s supply-demand dynamics through a hybrid agent-based modeling framework

The urban water system involves complex interactions between physical and social factors, such as physical infrastructures and information communications. Among these, the water supply network (WSN) and water consumers network (WCN) are critical determinants of water supply and demand, which are featured with prominent structural heterogeneity and interrelationships. The structures of WSN and WCN have been traditionally modeled in isolation under the assumption of structural homogeneity. This work proposes a new agent-based modeling (ABM) framework with a local-world fitness network to explore the effects of structural heterogeneity and WSN-WCN interrelationship on water supply–demand dynamics. A case study in Lhasa reveals that, the traditionally assumed fully-connected water supply system and structurally homogeneous water user network may result in simultaneous overestimation of water demand and underestimation of inter-region variances in water demand and groundwater table. Our analysis provides new insights into the disentanglement of the complex interdependencies of WSN and WCN and their associated structural heterogeneities, which could be extended to the modeling of other complex and interrelated networks.

Mesh-free semi-quantitative variance underestimation elimination method in Monte Caro algorithm

Mesh-free semi-quantitative variance underestimation elimination method in Monte Caro algorithm

Missing Step Count Data? Step Away From the Expectation–Maximization Algorithm

In studies that compare physical activity between groups of individuals, it is common for physical activity to be quantified by step count, which is measured by accelerometers or other wearable devices. Missing step count data often arise in these settings and can lead to bias or imprecision in the estimated effect if handled inappropriately. Replacing each missing value in accelerometer data with a single value using the Expectation–Maximization (EM) algorithm has been advocated in the literature, but it can lead to underestimation of variances and could seriously compromise study conclusions. We compare the performance in terms of bias and variance of two missing data methods, the EM algorithm and Multiple Imputation (MI), through a simulation study where data are generated from a parametric model to reflect characteristics of a trial on physical activity. We also conduct a reanalysis of the 2019 MOVE-IT trial. The EM algorithm leads to an underestimate of the variance of effects of interest, in both the simulation study and the reanalysis of the MOVE-IT trial. MI should be the preferred approach to handling missing data in accelerometer, which provides valid point and variance estimates.

Capturing and utilizing the random feature in Monte Carlo fission source distributions

The Monte Carlo algorithm commonly adopts the power iteration (PI) process in criticality calculations. Previous studies have shown that random error term in the PI process significantly influences the source convergence diagnosis and variance underestimation phenomenon. This paper uses the Sliced Wasserstein (SW) distance of the fission source distributions (FSDs) to estimate the error terms in the PI process. Three mesh-free SW-based methods are proposed to capture the random error term's 1m feature for the OECD source convergence fissile slab model, sphere array model, and the BEAVRS model. Then, a mesh-free source convergence auto-diagnosis algorithm is introduced based on the methods. The algorithm sets the batch size at five stages. The 1m feature of the random error term is utilized to achieve auto-diagnosis at the target batch size. Numerical calculation results show that the auto-diagnosis algorithm is practical and efficient for source convergence diagnosis and acceleration, which can automatically save 58 % to 81 % of the computational time to reach a converged fission source in models with the dominance ratio of around 0.99.

When Fixed and Random Effects Mismatch: Another Case of Inflation of Evidence in Non-Maximal Models

Mixed-effects models that include both fixed and random effects are widely used in the cognitive sciences because they are particularly suited to the analysis of clustered data. However, testing hypotheses about fixed effects in the presence of random effects is far from straightforward and a set of best practices is still lacking. In the target article, van Doorn et al. (Computational Brain & Behavior, 2022) examined how Bayesian hypothesis testing with mixed-effects models is impacted by particular model specifications. Here, I extend their work to the more complex case of multiple correlated predictors, such as a predictor of interest and a covariate. I show how non-maximal models can display ‘mismatches’ between fixed and random effects, which occur when a model includes random slopes for the effect of interest, but fails to include them for those predictors that correlate with the effect of interest. Bayesian model comparisons with synthetic data revealed that such mismatches can lead to an underestimation of random variance and to inflated Bayes factors. I provide specific recommendations for resolving mismatches of this type: fitting maximal models, eliminating correlations between predictors, and residualising the random effects. Data and code are publicly available in an OSF repository at https://osf.io/njaup.

Rigorous calibration of homodyne detection efficiency for continuous-variable quantum key distribution.

We propose a rigorous calibration method for homodyne detection efficiency, which combines all the factors that affect detection efficiency to calibrate together through the actual homodyne detection. With this method, the transmittance converted from electronic noise in the one-time calibration method of the shot noise can be attributed to the detection inefficiency. Thus, a trusted detection noise-free model for continuous-variable quantum key distribution (CV-QKD) can be established, which simplifies the calibration of shot noise while having the same performance as the trusted detection noise model. We demonstrate this calibration method with a balanced detector based on a transimpedance amplifier. Experimental results show that detection efficiency will be overestimated if the integration factor of the detector is overlooked. The overestimation of the detection efficiency leads to an underestimation of modulation variance and excess noise when the modulation variance is monitored by the balanced detector, which opens security loopholes. Our method may prove a necessary method in the calibration of detection efficiency for CV-QKD.

A multiple membership multilevel negative binomial model for intersection crash analysis

Many intersections belong to more than one zone, but most research has not considered the effects of multiple zones in intersection crash analysis. This issue is known as a boundary problem. Unobserved heterogeneity between zones can lead to model misspecification which can result in biased parameter estimates and poor model fitting performance. This study investigated the issue using five years of intersection crash data from the City of Regina, Saskatchewan, Canada. The study developed three multiple membership multilevel negative binomial models to reduce unobserved zonal-level heterogeneity. Each multiple membership multilevel model used a different weight strategy. When the fitting performance of the three multiple membership multilevel models was compared with two additional models, a traditional single level model and a conventional multilevel model, all three multiple membership multilevel models had a better fitting performance. Five individual-level and seven group-level variables were statistically significant (90% confidence level) in all the models with five of the individual-level and four of the group-level variables statistically significant at the 99% confidence level. The multiple membership multilevel models also helped to prevent the underestimation of group-level variance and type I statistical errors that tend to occur with single level models and conventional multilevel models. In particular, the three multiple membership multilevel models produced more accurate results for intersections with a large AADT. As intersections with a large AADT are known to have more crashes, multiple membership multilevel models are likely to be more useful than single level models and conventional multilevel models when selecting intersections for safety improvement. The study recommends the adoption of a multiple membership multilevel model to improve fitting performance and reduce the boundary problem for intersections affected by more than one zone.

Deep Quantile Regression for Uncertainty Estimation in Unsupervised and Supervised Lesion Detection

Despite impressive state-of-the-art performance on a wide variety of machine learning tasks in multiple applications, deep learning methods can produce over-confident predictions, particularly with limited training data. Therefore, quantifying uncertainty is particularly important in critical applications such as lesion detection and clinical diagnosis, where a realistic assessment of uncertainty is essential in determining surgical margins, disease status and appropriate treatment. In this work, we propose a novel approach that uses quantile regression for quantifying aleatoric uncertainty in both supervised and unsupervised lesion detection problems. The resulting confidence intervals can be used for lesion detection and segmentation. In the unsupervised settings, we combine quantile regression with the Variational AutoEncoder (VAE). The VAE is trained on lesion-free data, so when presented with an image with a lesion, it tends to reconstruct a lesion-free version of the image. To detect the lesion, we then compare the input (lesion) and output (lesion-free) images. Here we address the problem of quantifying uncertainty in the images that are reconstructed by the VAE as the basis for principled outlier or lesion detection. The VAE models the output as a conditionally independent Gaussian characterized by its mean and variance. Unfortunately, joint optimization of both mean and variance in the VAE leads to the well-known problem of shrinkage or underestimation of variance. Here we describe an alternative Quantile-Regression VAE (QR-VAE) that avoids this variance shrinkage problem by directly estimating conditional quantiles for the input image. Using the estimated quantiles, we compute the conditional mean and variance for input image from which we then detect outliers by thresholding at a false-discovery-rate corrected p-value. In the supervised setting, we develop binary quantile regression (BQR) for the supervised lesion segmentation task. We show how BQR can be used to capture uncertainty in lesion boundaries in a manner that characterizes expert disagreement.

Adapting Nearest Neighbor for Multiple Imputation: Advantages, Challenges, and Drawbacks

Abstract The U.S. Census Bureau has historically used nearest neighbor (NN) or random hot deck (RHD) imputation to handle missing data for many types of survey data. Using these methods removes the need to parametrically model values in imputation models. With strong auxiliary information, NN imputation is preferred because it produces more precise estimates than RHD. In addition, NN imputation is robust against a misspecified response mechanism if missingness depends on the auxiliary variable, in contrast to RHD which ignores the auxiliary information. A compromise between these two methods is k-NN imputation, which identifies a set of the k closest neighbors (“donor pool”) and randomly selects a single donor from this set. Recently these methods have been used for multiple imputation (MI), enabling variance estimation via the so-called Rubin’s Combining Rules. The Approximate Bayesian Bootstrap (ABB) is a simple-to-implement algorithm that makes the RHD “proper” for MI. In concept, ABB should work to propagate uncertainty for NN MI; bootstrapping respondents mean each nonrespondent’s one “nearest” donor will not be available for every imputation. However, we demonstrate through simulation that NN MI using ABB leads to variance underestimation. This underestimation is somewhat but not entirely attenuated with k-NN imputation. An alternative approach to variance estimation after MI, bootstrapped MI, eliminates the underestimation with NN imputation, but we show that it suffers from overestimation of variance with nonnegligible sampling fractions under both equal and unequal probability sampling designs. We propose a modification to bootstrapped MI to account for nonnegligible sampling fractions. We compare the performance of RHD and the various NN MI methods under a variety of sampling designs, sampling fractions, distribution shapes, and missingness mechanisms.

Fast Scalable Image Restoration Using Total Variation Priors and Expectation Propagation.

This paper presents a scalable approximate Bayesian method for image restoration using Total Variation (TV) priors, with the ability to offer uncertainty quantification. In contrast to most optimization methods based on maximum a posteriori estimation, we use the Expectation Propagation (EP) framework to approximate minimum mean squared error (MMSE) estimates and marginal (pixel-wise) variances, without resorting to Monte Carlo sampling. For the classical anisotropic TV-based prior, we also propose an iterative scheme to automatically adjust the regularization parameter via Expectation Maximization (EM). Using Gaussian approximating densities with diagonal covariance matrices, the resulting method allows highly parallelizable steps and can scale to large images for denoising, deconvolution, and compressive sensing (CS) problems. The simulation results illustrate that such EP methods can provide a posteriori estimates on par with those obtained via sampling methods but at a fraction of the computational cost. Moreover, EP does not exhibit strong underestimation of posteriori variances, in contrast to variational Bayes alternatives.

Imputation Allowing Standard Variance Formulas

Although deletion of cases is still a common method of dealing with item nonresponse, imputation is a major alternative. With traditional methods of imputation, though, the usual variance formulas understate the variance of estimates. This paper proposes that items be imputed from distributions more diffuse than those of the real data, thereby compensating for the underestimation of variance by the usual formulas. The impact on covariances is considered in the design of the method. The method is intended for use by data analysts applying techniques based on functions of first and second moments of means only.

Suitability of fibre-optic distributed temperature sensing for revealing mixing processes and higher-order moments at the forest–air interface

Abstract. The suitability of a fibre-optic distributed temperature sensing (DTS) technique for observing atmospheric mixing profiles within and above a forest was quantified, and these profiles were analysed. The spatially continuous observations were made at a 125 m tall mast in a boreal pine forest. Airflows near forest canopies diverge from typical boundary layer flows due to the influence of roughness elements (i.e. trees) on the flow. Ideally, these complex flows should be studied with spatially continuous measurements, yet such measurements are not feasible with conventional micrometeorological measurements with, for example, sonic anemometers. Hence, the suitability of DTS measurements for studying canopy flows was assessed. The DTS measurements were able to discern continuous profiles of turbulent fluctuations and mean values of air temperature along the mast, providing information about mixing processes (e.g. canopy eddies and evolution of inversion layers at night) and up to third-order turbulence statistics across the forest–atmosphere interface. Turbulence measurements with 3D sonic anemometers and Doppler lidar at the site were also utilised in this analysis. The continuous profiles for turbulence statistics were in line with prior studies made at wind tunnels and large eddy simulations for canopy flows. The DTS measurements contained a significant noise component which was, however, quantified, and its effect on turbulence statistics was accounted for. Underestimation of air temperature fluctuations at high frequencies caused 20 %–30 % underestimation of temperature variance at typical flow conditions. Despite these limitations, the DTS measurements should prove useful also in other studies concentrating on flows near roughness elements and/or non-stationary periods, since the measurements revealed spatio-temporal patterns of the flow which were not possible to be discerned from single point measurements fixed in space.

Partial and complete dependency among data sets has minimal consequence on estimates from integrated population models

Integrated population models (IPMs) are widely used to combine disparate data sets in joint analysis to better understand population dynamics and provide guidance for conservation activities. An often-cited assumption of IPMs is independence among component data sets within the combined likelihood. Dependency among data sets should lead to underestimation of variance and bias because individuals contribute data to>1 data set. In practice, studied individuals often occur in multiple data sets in IPMs (i.e., overlap), which is one way for the independence assumption to be violated. Such cases have the potential to dissuade practitioners and limit application of IPMs to solve emerging ecological problems. We assessed precision and bias of demographic rates estimated from IPMs using a complete gradient (0-100%) of overlap among data sets, wide ranges in demographic rates (e.g., survival 0.1-0.8) and sample sizes (100-1200 individuals) and variable data sources. We compared results from our simulations with those from IPMs constructed using empirical data on tree swallows (Tachycineta bicolor) where data sets either had complete overlap or included different individuals. Contrary to previous investigators, we found no substantive bias or uncertainty in any demographic rate from IPMs derived from data sets with complete overlap. While variability in demographic rates was greater at low sample sizes (i.e., low capture, recapture and survey probabilities), there were negligible differences in the posterior mean or root mean square error of demographic rates among IPMs with strong dependence versus complete independence among data sets. Our simulations suggest IPMs can be designed using only capture-recapture data or harvest and capture-recovery data where population estimates are obtained from the same data as survival and productivity data. While we encourage researchers to carefully consider the modeling approach best-suited for their data sets, our results suggest that dependence among data sets does not generally compromise IPM estimates. Thus, violation of the independence assumption should not dissuade researchers from the application of IPMs in ecological research.

Quantile Regression for Uncertainty Estimation in VAEs with Applications to Brain Lesion Detection.

The Variational AutoEncoder (VAE) has become one of the most popular models for anomaly detection in applications such as lesion detection in medical images. The VAE is a generative graphical model that is used to learn the data distribution from samples and then generate new samples from this distribution. By training on normal samples, the VAE can be used to detect inputs that deviate from this learned distribution. The VAE models the output as a conditionally independent Gaussian characterized by means and variances for each output dimension. VAEs can therefore use reconstruction probability instead of reconstruction error for anomaly detection. Unfortunately, joint optimization of both mean and variance in the VAE leads to the well-known problem of shrinkage or underestimation of variance. We describe an alternative VAE model, Quantile-Regression VAE (QR-VAE), that avoids this variance shrinkage problem by estimating conditional quantiles for the given input image. Using the estimated quantiles, we compute the conditional mean and variance for input images under the Gaussian model. We then compute reconstruction probability using this model as a principled approach to outlier or anomaly detection. We also show how our approach can be used for heterogeneous thresholding of images for detecting lesions in brain images.

An Energy Closure Criterion for Model Reduction of a Kicked Euler–Bernoulli Beam

AbstractReduced order models (ROMs) can be simulated with lower computational cost while being more amenable to theoretical analysis. Here, we examine the performance of the proper orthogonal decomposition (POD), a data-driven model reduction technique. We show that the accuracy of ROMs obtained using POD depends on the type of data used and, more crucially, on the criterion used to select the number of proper orthogonal modes (POMs) used for the model. Simulations of a simply supported Euler–Bernoulli beam subjected to periodic impulsive loads are used to generate ROMs via POD, which are then simulated for comparison with the full system. We assess the accuracy of ROMs obtained using steady-state displacement, velocity, and strain fields, tuning the spatiotemporal localization of applied impulses to control the number of excited modes in, and hence the dimensionality of, the system’s response. We show that conventional variance-based mode selection leads to inaccurate models for sufficiently impulsive loading and that this poor performance is explained by the energy imbalance on the reduced subspace. Specifically, the subspace of POMs capturing a fixed amount (say, 99.9%) of the total variance underestimates the energy input and dissipated in the ROM, yielding inaccurate reduced-order simulations. This problem becomes more acute as the loading becomes more spatio-temporally localized (more impulsive). Thus, energy closure analysis provides an improved method for generating ROMs with energetics that properly reflect that of the full system, resulting in simulations that accurately represent the system’s true behavior.

Multiple membership multilevel model to estimate intersection crashes

Numerous studies have developed intersection crash prediction models to identify crash hotspots and evaluate safety countermeasures. These studies largely considered only micro-level crash contributing factors such as traffic volume, traffic signals, etc. Some recent studies, however, have attempted to include macro-level crash contributing factors, such as population per zone, to predict the number of crashes at intersections. As many intersections are located between multiple zones and thus affected by factors from the multiple zones, the inclusion of macro-level factors requires boundary problems to be resolved. In this study, we introduce an advanced multilevel model, the multiple membership multilevel model (MMMM), for intersection crash analysis. Our objective was to reduce heterogeneity issues between zones in crash prediction model while avoiding misspecification of the model structure. We used five years of intersection crash data (2009–2013) for the City of Regina, Saskatchewan, Canada and identified micro- and macro-level factors that most affected intersection crashes. We compared the fitting performance of the MMMM with that of two existing models, a traditional single model (SM) and a conventional multilevel model (CMM). The MMMM outperformed the SM and CMM in terms of fitting capability. We found that the MMMM avoided both the underestimation of macro-level variance and the type I statistical error that tend to occur when the crash data are analyzed using a SM or CMM. Statistically significant micro-level and macro-level crash contributing factors in Regina included major roadway AADT, four legs, traffic signals, speed, young drivers, and different types of land use.