Bias-variance Research Articles

Genetic Parameter and Hyper-Parameter Estimation Underlie Nitrogen Use Efficiency in Bread Wheat.

Estimation and prediction play a key role in breeding programs. Currently, phenotyping of complex traits such as nitrogen use efficiency (NUE) in wheat is still expensive, requires high-throughput technologies and is very time consuming compared to genotyping. Therefore, researchers are trying to predict phenotypes based on marker information. Genetic parameters such as population structure, genomic relationship matrix, marker density and sample size are major factors that increase the performance and accuracy of a model. However, they play an important role in adjusting the statistically significant false discovery rate (FDR) threshold in estimation. In parallel, there are many genetic hyper-parameters that are hidden and not represented in the given genomic selection (GS) model but have significant effects on the results, such as panel size, number of markers, minor allele frequency, number of call rates for each marker, number of cross validations and batch size in the training set of the genomic file. The main challenge is to ensure the reliability and accuracy of predicted breeding values (BVs) as results. Our study has confirmed the results of bias-variance tradeoff and adaptive prediction error for the ensemble-learning-based model STACK, which has the highest performance when estimating genetic parameters and hyper-parameters in a given GS model compared to other models.

Estimation of prediction error in time series

Summary The accurate estimation of prediction errors in time series is an important problem, which has immediate implications for the accuracy of prediction intervals as well as the quality of a number of widely used time series model selection criteria such as the Akaike information criterion. Except for simple cases, however, it is difficult or even impossible to obtain exact analytical expressions for one-step and multi-step predictions. This may be one of the reasons that, unlike in the independent case (see Efron, 2004), up to now there has been no fully established methodology for time series prediction error estimation. Starting from an approximation to the bias-variance decomposition of the squared prediction error, a method for accurate estimation of prediction errors in both univariate and multivariate stationary time series is developed in this article. In particular, several estimates are derived for a general class of predictors that includes most of the popular linear, nonlinear, parametric and nonparametric time series models used in practice, with causal invertible autoregressive moving average and nonparametric autoregressive processes discussed as lead examples. Simulations demonstrate that the proposed estimators perform quite well in finite samples. The estimates may also be used for model selection when the purpose of modelling is prediction.

The role of workplace friendship in cultural intelligence and quality of work-life: A cross-sectional study

The purpose of this study is to examine the effect of cultural intelligence (CI) and workplace friendship (WPF) on the quality of work-life (QWL) of individual workers and to examine whether WPF mediates the relationship between CI and QWL between employees of the same nationality, but of different ethnic groups working in government-owned organisations. A cross-sectional study was conducted among 161 employees using self-report measures. Data analysis was performed using two statistical procedures (the Harman single factor test and the correlation matrix) to assess the presence of a common-method variance (CMV) bias in the data collected for this study. Theoretical propositions and empirical observations offered plausible explanations for WPF's mediating role in the CI-QWL relationship. The present study confirms the reliability and, to some extent, the validity of the measures in a developing economy context. However, this study also builds empirical evidence that age and gender are fewer differentiating factors in employees' work attitudes and behaviour.

Prediction and model evaluation for space–time data

ABSTRACT Evaluation metrics for prediction error, model selection and model averaging on space–time data are understudied and poorly understood. The absence of independent replication makes prediction ambiguous as a concept and renders evaluation procedures developed for independent data inappropriate for most space–time prediction problems. Motivated by air pollution data collected during California wildfires in 2008, this manuscript attempts a formalization of the true prediction error associated with spatial interpolation. We investigate a variety of cross-validation (CV) procedures employing both simulations and case studies to provide insight into the nature of the estimand targeted by alternative data partition strategies. Consistent with recent best practice, we find that location-based cross-validation is appropriate for estimating spatial interpolation error as in our analysis of the California wildfire data. Interestingly, commonly held notions of bias-variance trade-off of CV fold size do not trivially apply to dependent data, and we recommend leave-one-location-out (LOLO) CV as the preferred prediction error metric for spatial interpolation.

Application of polynomial chaos expansion in sensitivity analysis of towed cable parameters of the underwater towing system

The key to achieving the optimal design of towed cables, maintaining numerical simulation accuracy, and achieving precise control of the towed body lies in sensitivity analysis. However, the traditional global sensitivity analysis method presents challenges such as high calculation costs and low accuracy. To address these issues, this paper introduces polynomial chaos expansion (PCE) to quantitatively analyze the impact of uncertainties in physical and environmental parameters on the position and attitude of the towed cable. Latin hypercube sampling is employed to obtain sample sets of input parameters, and these samples are applied to the lumped mass method to calculate the end position coordinates of the towed cable, which serves as the output response. PCE is utilized to quantitatively compute the Sobol global sensitivity index of the towed cable parameters. The accuracy of the PCE model is verified, and the optimal degree of basis functions is selected using the bias-variance trade-off. The advantages of PCE are demonstrated by comparing it with the Monte Carlo and Morris methods. The results indicate that PCE accurately calculates the global sensitivity index of towed cable parameters even with a limited sample size. Under the condition of a fixed cable length, the position and attitude of the towed cable are sensitive to the current rate, liquid density, cable diameter, normal drag coefficient, and specific gravity. The feasibility and efficiency of PCE applied to the sensitivity analysis of towed cable parameters is verified, and recommendations for the engineering application of towed cables are summarized.

Rapid and biased evolution of canalization during adaptive divergence revealed by dominance in gene expression variability during Arctic charr early development

Adaptive evolution may be influenced by canalization, the buffering of developmental processes from environmental and genetic perturbations, but how this occurs is poorly understood. Here, we explore how gene expression variability evolves in diverging and hybridizing populations, by focusing on the Arctic charr (Salvelinus alpinus) of Thingvallavatn, a classic case of divergence between feeding habitats. We report distinct profiles of gene expression variance for both coding RNAs and microRNAs between the offspring of two contrasting morphs (benthic/limnetic) and their hybrids reared in common conditions and sampled at two key points of cranial development. Gene expression variance in the hybrids is substantially affected by maternal effects, and many genes show biased expression variance toward the limnetic morph. This suggests that canalization, as inferred by gene expression variance, can rapidly diverge in sympatry through multiple gene pathways, which are associated with dominance patterns possibly biasing evolutionary trajectories and mitigating the effects of hybridization on adaptive evolution.

Alternative parameter learning schemes for monitoring process stability

In statistical process control, accurately estimating in-control (IC) parameters is crucial for effective monitoring. This typically requires a Phase I analysis to obtain estimates before monitoring commences. The traditional “fixed” estimate (FE) approach uses these estimates exclusively, while the “adaptive” estimate (AE) approach updates the estimates with each new observation. Such extreme criteria reflect the traditional bias-variance tradeoff in the framework of the sequential parameter learning schemes. This paper proposes an intermediate update rule that generalizes two ad hoc criteria for monitoring univariate Gaussian data, by giving a lower probability to parameter updates when an out-of-control (OC) situation is likely, therefore updating more frequently when there is no evidence of an OC scenario. The simulation study shows that this approach improves the detection power for small and early shifts, which are commonly regarded as a weakness of control charts based on fully online adaptive estimation. The paper also shows that the proposed method performs similarly to the fully adaptive procedure for larger or later shifts. The proposed method is illustrated by monitoring the sudden increase in ICU counts during the 2020 COVID outbreak in New York.

Forecasting stock market volatility under parameter and model uncertainty

We forecast monthly stock market volatility under parameter and model uncertainty. Using a long economic dataset spanning almost a century, we prove that model uncertainty plays a more crucial role than parameter uncertainty in improving volatility predictability. The combination models with model uncertainty, especially dynamic model averaging (DMA), provide very competitive improvements in forecasting accuracy, whose superiority is also reflected in asset allocation and risk hedging. We find two empirical properties of forecast combination: (i) it incorporates information from numerous predictors, helping reduce both the forecast bias and forecast error variance; and (ii) the economic links of the forecasts based on it are significant, and the predictive gains are concentrated in poor economic conditions. Overall, we highlight the importance of considering model uncertainty via forecast combination when investigating the expected stock market volatility.

Quantification of short echo time MRS signals with improved version of QUantitation based on quantum ESTimation algorithm.

Magnetic resonance spectroscopy offers information about metabolite changes in the organism, which can be used in diagnosis. While short echo time proton spectra exhibit more distinguishable metabolites compared with proton spectra acquired with long echo times, their quantification (and providing estimates of metabolite concentrations) is more challenging. They are hampered by a background signal, which originates mainly from macromolecules (MM) and mobile lipids. An improved version of the quantification algorithm QUantitation based on quantum ESTimation (QUEST), with MM prior knowledge (QUEST-MM), dedicated to proton signals and invoking appropriate prior knowledge on MM, is proposed and tested. From a single acquisition, it enables better metabolite quantification, automatic estimation of the background, and additional automatic quantification of MM components, thus improving its applicability in the clinic. The proposed algorithm may facilitate studies that involve patients with pathological MM in the brain. QUEST-MM and three QUEST-based strategies for quantifying short echo time signals are compared in terms of bias-variance trade-off and Cramér-Rao lower bound estimates. The performances of the methods are evaluated through extensive Monte Carlo studies. In particular, the histograms of the metabolite and MM amplitude distributions demonstrate the performances of the estimators. They showed that QUEST-MM works better than QUEST (Subtract approach) and is a good alternative to QUEST when measured MM signal is unavailable or unsuitable. Quantification with QUEST-MM is shown for 1 H in vivo rat brain signals obtained with the SPECIAL pulse sequence at 9.4 T, and human brain signals obtained, respectively, with STEAM at 4 T and PRESS at 3 T. QUEST-MM is implemented in jMRUI and will be available for public use from version 7.1.

A Two-Step Synthetic Control Approach for Estimating Causal Effects of Marketing Events

Marketing researchers are often interested in estimating causal effects when a randomized experiment is infeasible. The synthetic control (SC) method has emerged as a powerful tool in these quasiexperimental settings. It is important to verify the SC parallel pretrends assumption, the testable part of the identifying assumption, because its violation may lead to biased estimates. However, no formal test exists, so researchers have to rely on visual inspection. Even with a formal test, researchers still need to know how to balance the bias-efficiency trade-off for the estimate. We fill this void and advance the two-step synthetic control (TSSC) approach that comprises a formal test for the SC pretrends assumption in the first step and recommends an appropriate method that balances the dual goal of reducing bias and increasing efficiency in the second step. Simulations show that the TSSC approach performs favorably in the bias-variance (bias-efficiency) trade-off. Applying the TSSC approach, we find that New York State’s repeal of the tampon tax caused a positive and significant (2.08%) increase in weekly tampon sales. Using theory, simulations, and empirics, we demonstrate the importance, validity, and usefulness of the TSSC approach. This paper was accepted by Matthew Shum, marketing. Supplemental Material: The data files and online appendix are available at https://doi.org/10.1287/mnsc.2023.4878 .

Neural Belief Propagation for Scene Graph Generation.

Scene graph generation aims to interpret an input image by explicitly modelling the objects contained therein and their relationships. In existing methods the problem is predominantly solved by message passing neural network models. Unfortunately, in such models, the variational distributions generally ignore the structural dependencies among the output variables, and most of the scoring functions only consider pairwise dependencies. This can lead to inconsistent interpretations. In this article, we propose a novel neural belief propagation method seeking to replace the traditional mean field approximation with a structural Bethe approximation. To find a better bias-variance trade-off, higher-order dependencies among three or more output variables are also incorporated into the relevant scoring function. The proposed method achieves the state-of-the-art performance on various popular scene graph generation benchmarks.

On lower bounds for the bias-variance trade-off

On lower bounds for the bias-variance trade-off

Tradeoff of generalization error in unsupervised learning

Abstract Finding the optimal model complexity that minimizes the generalization error (GE) is a key issue of machine learning. For the conventional supervised learning, this task typically involves the bias-variance tradeoff: lowering the bias by making the model more complex entails an increase in the variance. Meanwhile, little has been studied about whether the same tradeoff exists for unsupervised learning. In this study, we propose that unsupervised learning generally exhibits a two-component tradeoff of the GE, namely the model error (ME) and the data error (DE)—using a more complex model reduces the ME at the cost of the DE, with the DE playing a more significant role for a smaller training dataset. This is corroborated by training the restricted Boltzmann machine to generate the configurations of the two-dimensional Ising model at a given temperature and the totally asymmetric simple exclusion process with given entry and exit rates. Our results also indicate that the optimal model tends to be more complex when the data to be learned are more complex.

Asymptotic properties of Spearman’s footrule and Gini’s gamma in bivariate normal model

This paper aims at rejuvenating the two rank correlation coefficients, Spearman’s footrule (SF) and Gini’s gamma (GG), which were forgotten in the literature for a long time due to lack of knowledge concerning their statistical properties. Under the common bivariate normal model, we establish the asymptotic analytical expressions of the mean and variance of SF and GG, and investigate the performances of SF and GG from the aspects of biased effect, approximate variance and asymptotic relative efficiency (ARE). Moreover, we further study the robustness of SF and GG under contaminated normal models. In order to get a deeper understanding of their performances, we also compare SF and GG with Kendall’s tau (KT) and Spearman’s rho (SR), the most widely used rank correlation coefficients, in terms of bias and mean square error (MSE) under both the normal and contaminated normal models. Finally we show an application of SF and GG in the field of signal processing through the example of time-delay estimation. Simulation results indicate that SF and GG outperform SR and KT in some cases. The new findings discovered in this paper enable SF and GG to play complementary roles to KT and SR in practice.

A surrogate-assisted uncertainty-aware Bayesian validation framework and its application to coupling free flow and porous-medium flow

Existing model validation studies in geoscience often disregard or partly account for uncertainties in observations, model choices, and input parameters. In this work, we develop a statistical framework that incorporates a probabilistic modeling technique using a fully Bayesian approach to perform a quantitative uncertainty-aware validation. A Bayesian perspective on a validation task yields an optimal bias-variance trade-off against the reference data. It provides an integrative metric for model validation that incorporates parameter and conceptual uncertainty. Additionally, a surrogate modeling technique, namely Bayesian Sparse Polynomial Chaos Expansion, is employed to accelerate the computationally demanding Bayesian calibration and validation. We apply this validation framework to perform a comparative evaluation of models for coupling a free flow with a porous-medium flow. The correct choice of interface conditions and proper model parameters for such coupled flow systems is crucial for physically consistent modeling and accurate numerical simulations of applications. We develop a benchmark scenario that uses the Stokes equations to describe the free flow and considers different models for the porous-medium compartment and the coupling at the fluid–porous interface. These models include a porous-medium model using Darcy’s law at the representative elementary volume scale with classical or generalized interface conditions and a pore-network model with its related coupling approach. We study the coupled flow problems’ behaviors considering a benchmark case, where a pore-scale resolved model provides the reference solution. With the suggested framework, we perform sensitivity analysis, quantify the parametric uncertainties, demonstrate each model’s predictive capabilities, and make a probabilistic model comparison.

Identifying the structure of within‐team variance in ratings of team constructs

AbstractResearchers frequently assume some degree of consensus among team members' perceptions when aggregating individual survey responses to the team level. Literature reviews have routinely found, however, substantial within‐team variance in referent‐shift consensus measures of team constructs and that team member agreement is often lower than desired. To enhance our understanding of the structure of within‐team variance, using a sample of 20,183 individuals in 4,313 teams, we explored the proportion of variance attributable to different sources (response acquiescence, positivity bias, and rater variance specific to each construct) for five team constructs. We also examined the extent to which within‐team variance changed over time with a subsample of 3,720 individuals in 731 teams. The results indicated that constructs thought to be more observable, including role clarity and monitoring goal progress, appeared to be less prone to idiosyncratic perceiver effects and that the processes were widely experienced by all team members. Conversely, relationship conflict showed higher levels of within‐team variance and team consensus actually decreased over time. These findings indicate that perceptions of relationship conflict are influenced by the individual differences of team members and that conflict may be restricted to dyads or subgroups within teams. Overall, the findings indicate that stable perceiver effects and processes that are limited to a small proportion of team members may have a stronger influence on ratings of some team constructs than previously thought. We conclude with suggestions for theory development, team construct measurement, and advice to understand processes that lead to consensus in perceptions of team constructs.

Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Supervised machine learning (ML) is becoming an influential analytical method in psychology and other social sciences. However, theoretical ML concepts and predictive-modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide an intuitive but thorough primer and introduction to supervised ML for psychologists in four consecutive modules. After introducing the basic terminology and mindset of supervised ML, in Module 1, we cover how to use resampling methods to evaluate the performance of ML models (bias-variance trade-off, performance measures, k-fold cross-validation). In Module 2, we introduce the nonlinear random forest, a type of ML model that is particularly user-friendly and well suited to predicting psychological outcomes. Module 3 is about performing empirical benchmark experiments (comparing the performance of several ML models on multiple data sets). Finally, in Module 4, we discuss the interpretation of ML models, including permutation variable importance measures, effect plots (partial-dependence plots, individual conditional-expectation profiles), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts are provided, with as few mathematical formulas as possible, and followed by code examples using the mlr3 and companion packages in R. Key practical-analysis steps are demonstrated on the publicly available PhoneStudy data set ( N = 624), which includes more than 1,800 variables from smartphone sensing to predict Big Five personality trait scores. The article contains a checklist to be used as a reminder of important elements when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in online materials ( https://osf.io/9273g/ ).

GCL: Contrastive learning instead of graph convolution for node classification

Contrastive learning as an effective representation learning technique has attracted tremendous attention due to its general success in downstream tasks. However, the theoretical explanations and quantitative experimental analyses of its generalization ability are still limited. These issues are pivotal yet challenging for improving both the interpretability and performance of contrastive learning. To address these issues, we first re-examine the least squares bias-variance decomposition and successfully derive GCL, a novel bias-variance decomposition with two optional generalized biases and one generalized variance. GCL is shown to be extendable to common contrastive learning models so that it can be utilized as a unified contrastive learning framework. Meanwhile, a surprising finding that the gradient descent of contrastive loss concerning feature representation is closely related to the message passing mechanism (graph convolution) of Graph Neural Networks (GNNs). The contrastive learning model called GCP is then proposed as a convincing implementation of GCL. GCP has a pure MLP-based structure and employs a conventional cross-entropy to reduce the bias between predictions and ground truth labels, two optional contrastive losses to optimize the variance of the model. Finally, extensive experiments demonstrate that the two biases proposed by GCL have their own merits; GCP achieves comparable or even better performance than GNNs in a more efficient and robust manner, its bias and variance meet the bias-variance tradeoff to some extent.

Optimal adjustment sets for causal query estimation in partially observed biomolecular networks.

Causal query estimation in biomolecular networks commonly selects a 'valid adjustment set', i.e. a subset of network variables that eliminates the bias of the estimator. A same query may have multiple valid adjustment sets, each with a different variance. When networks are partially observed, current methods use graph-based criteria to find an adjustment set that minimizes asymptotic variance. Unfortunately, many models that share the same graph topology, and therefore same functional dependencies, may differ in the processes that generate the observational data. In these cases, the topology-based criteria fail to distinguish the variances of the adjustment sets. This deficiency can lead to sub-optimal adjustment sets, and to miss-characterization of the effect of the intervention. We propose an approach for deriving 'optimal adjustment sets' that takes into account the nature of the data, bias and finite-sample variance of the estimator, and cost. It empirically learns the data generating processes from historical experimental data, and characterizes the properties of the estimators by simulation. We demonstrate the utility of the proposed approach in four biomolecular Case studies with different topologies and different data generation processes. The implementation and reproducible Case studies are at https://github.com/srtaheri/OptimalAdjustmentSet.

Measuring the Hubble constant with cosmic chronometers: a machine learning approach

Local measurements of the Hubble constant (H_0) based on Cepheids e Type Ia supernova differ by approx 5 sigma from the estimated value of H_0 from Planck CMB observations under Lambda CDM assumptions. In order to better understand this H_0 tension, the comparison of different methods of analysis will be fundamental to interpret the data sets provided by the next generation of surveys. In this paper, we deploy machine learning algorithms to measure the H_0 through a regression analysis on synthetic data of the expansion rate assuming different values of redshift and different levels of uncertainty. We compare the performance of different regression algorithms as Extra-Trees, Artificial Neural Network, Gradient Boosting, Support Vector Machines, and we find that the Support Vector Machine exhibits the best performance in terms of bias-variance tradeoff in most cases, showing itself a competitive cross-check to non-supervised regression methods such as Gaussian Processes.