Model Selection Bias Research Articles

Model selection of GLMMs in the analysis of count data in single-case studies: A Monte Carlo simulation.

Generalized linear mixed models (GLMMs) have great potential to deal with count data in single-case experimental designs (SCEDs). However, applied researchers have faced challenges in making various statistical decisions when using such advanced statistical techniques in their own research. This study focused on a critical issue by investigating the selection of an appropriate distribution to handle different types of count data in SCEDs due to overdispersion and/or zero-inflation. To achieve this, I proposed two model selection frameworks, one based on calculating information criteria (AIC and BIC) and another based on utilizing a multistage-model selection procedure. Four data scenarios were simulated including Poisson, negative binominal (NB), zero-inflated Poisson (ZIP), and zero-inflated negative binomial (ZINB). The same set of models (i.e., Poisson, NB, ZIP, and ZINB) were fitted for each scenario. In the simulation, I evaluated 10 model selection strategies within the two frameworks by assessing the model selection bias and its consequences on the accuracy of the treatment effect estimates and inferential statistics. Based on the simulation results and previous work, I provide recommendations regarding which model selection methods should be adopted in different scenarios. The implications, limitations, and future research directions are also discussed.

Evaluation of quantitative bias analysis in epidemiological research: A systematic review from 2010 to mid-2023.

We aimed to demonstrate the use of quantitative bias analysis (QBA), which reveals the effects of systematic error, including confounding, misclassificationand selection bias, on study resultsin epidemiological studies published in the period from 2010 to mid-23. The articles identified through a keyword search using Pubmed and Scopus were included in the study. The articles obtained from this search were eliminated according to the exclusion criteria, and the articles in which QBA analysis was applied were included in the detailed evaluation. It can be said that the application of QBA analysis has gradually increased over the 13-year period. Accordingly, the number of articles in which simple is used as a method in QBA analysis is 9 (9.89%), the number of articles in which the multidimensional approach is used is 10 (10.99%), the number of articles in which the probabilistic approach is used is 60 (65.93%) and the number of articles in which the method is not specified is 12 (13.19%). The number of articles with misclassification bias model is 44 (48.35%), the number of articles with uncontrolled confounder(s) bias model is 32 (35.16%), the number of articles with selection bias model is 7 (7.69%) and the number of articles using more than one bias model is 8 (8.79%). Of the 49 (53.85%) articles in which the bias parameter source was specified, 19 (38.78%) used internal validation, 26 (53.06%) used external validationand 4 (8.16%) used educated guess, data constraintsand hypothetical data.Probabilistic approach was used as a bias method in 60 (65.93%) of the articles, and mostly beta (8 [13.33%)], normal (9 [15.00%])and uniform (8 [13.33%]) distributions were selected. The application of QBA is rare in the literature but is increasing over time. Future researchers should include detailed analyzes such as QBA analysis to obtain inferences with higher evidence value, taking into account systematic errors.

Comparison of model selection and data bias on the prediction performance of purpleback flying squid (Sthenoteuthis oualaniensis) fishing ground in the Northwest Indian Ocean

Purpleback flying squid (Sthenoteuthis oualaniensis, PFS) is one of the critical economically cephalopod species in the northwest Indian Ocean, and the accurate model and dataset selection are critical parameters for predicting and managing the PFS fishing grounds. In this study, the PFS fishery data for the years 2016–2021 was analyzed using the gravity center of fishing grounds method, generalized additive model (GAM), gradient boosted trees (GBT), 3D Convolutional Neural Network (3DCNN), and 3D Convolutional Neural Networks-Convolutional LSTM Network (3DCNN-ConvLSTM) to explore the differences in annual catches, the annual gravity center of fishing ground, model performance, and importance of environmental variables in the case of datasets A (no moonlight days) and B (no moonlight days + bright moonlight days). The results are as follows: 1) Datasets A and B exhibit similar patterns of variation, with annual catches rising and then declining and the annual gravity center of fishing ground moving northeastward overall; 2) The GAM and GBT models had better model performance on dataset B (GAM (average F1-score ± standard deviation): 0.678658 ± 0.014684; GBT: 0.737422 ± 0.011748)than on dataset A (GAM: 0.676802 ± 0.013403; GBT: 0.736547 ± 0.013323), but almost negligible, and the standard deviation of the GAM model on dataset B becomes larger. 3DCNN and 3DCNN-ConvLSTM models perform in contrast, with significantly better F1-score and standard deviations on dataset A (3DCNN: 0.75048 ± 0.019763; 3DCNN-ConvLSTM: 0.740041 ± 0.023927) than on dataset B (3DCNN: 0.746378 ± 0.020337; 3DCNN-ConvLSTM: 0.736927 ± 0.04498); 3) The 3DCNN (optimal prediction of performance) or GBT (Optimal model stability) model is optimal for predicting the PFS fishing grounds; 4) GBT, 3DCNN, and 3DCNN-ConvLSTM model results all showed that the differences in the importance of environmental variables obtained from datasets A and B were significant; 5) Unlike the GBT model, the 3DCNN and 3DCNN-ConvLSTM models were more susceptible to dataset influences, with significant dependence on environmental variables that have large positive and negative sample differences. This study provides rich suggestions for constructing a predictive model for the PFS fishing grounds in the context of climate change. It also provides a new perspective on cleaning up biased data for light fisheries.

The relationship between firm-level climate change exposure, financial integration, cost of capital and investment efficiency

This paper investigates the impactof firm-level climate change exposure on corporate cost of capital, growth opportunities and new investment across 67countries withvarying degrees of financial integration from 2002 to 2021. The analysis documents that firms with high climate change exposure havea negative outlook, face increased cost of capital, and have reduced investment activity. Moreover, firms with climate change exposure are characterised by investment inefficiency and slower speed of adjustment towards the target investment. Thesefindings become more pronounced for companieswhich operate in countrieswith highlevels of financialintegration. Our results are robust to alternative estimation techniques that address model sensitivity, endogeneity, and selection bias issues.

Silicon versus Superbug: Assessing Machine Learning's Role in the Fight against Antimicrobial Resistance.

In his 1945 Nobel Prize acceptance speech, Sir Alexander Fleming warned of antimicrobial resistance (AMR) if the necessary precautions were not taken diligently. As the growing threat of AMR continues to loom over humanity, we must look forward to alternative diagnostic tools and preventive measures to thwart looming economic collapse and untold mortality worldwide. The integration of machine learning (ML) methodologies within the framework of such tools/pipelines presents a promising avenue, offering unprecedented insights into the underlying mechanisms of resistance and enabling the development of more targeted and effective treatments. This paper explores the applications of ML in predicting and understanding AMR, highlighting its potential in revolutionizing healthcare practices. From the utilization of supervised-learning approaches to analyze genetic signatures of antibiotic resistance to the development of tools and databases, such as the Comprehensive Antibiotic Resistance Database (CARD), ML is actively shaping the future of AMR research. However, the successful implementation of ML in this domain is not without challenges. The dependence on high-quality data, the risk of overfitting, model selection, and potential bias in training data are issues that must be systematically addressed. Despite these challenges, the synergy between ML and biomedical research shows great promise in combating the growing menace of antibiotic resistance.

Tracking COVID-19 vaccination expectancies and vaccination refusal in the United States

ABSTRACT To identify factors that predict COVID-19 vaccination refusal and show how expectancies affect vaccination acceptance for non-vaccinated adults, we used a monthly repeated cross-sectional sample from June/2021 to October/2021 to collect data on vaccination behaviors and predictor variables for 2,116 US adults over 50 years of age. Selection bias modeling – which is required when data availability is a result of behavioral choice – predicts two outcomes: (1) no vaccination vs. vaccination for the entire sample and (2) the effects of expectancy indices predicting vaccination Refuser vs. vaccination Accepters for the unvaccinated group. Vaccine refusers were younger and less educated, endorsed common misconceptions about the COVID-19 epidemic, and were Black. Vaccination expectancies were related to vaccination refusal in the unvaccinated eligible group: negative expectancies increased vaccine refusal, while positive expectancies decreased it. We conclude that behavior-related expectancies (as opposed to more stable psychological traits) are important to identify because they are often modifiable and provide a point of intervention, not just for COVID-19 vaccination acceptance but also for other positive health behaviors.

Integrated data-driven modeling and experimental optimization of granular hydrogel matrices

Integrated data-driven modeling and experimental optimization of granular hydrogel matrices

Comparison of Seven Non-Linear Mixed Effect Model-Based Approaches to Test for Treatment Effect.

Analyses of longitudinal data with non-linear mixed-effects models (NLMEM) are typically associated with high power, but sometimes at the cost of inflated type I error. Approaches to overcome this problem were published recently, such as model-averaging across drug models (MAD), individual model-averaging (IMA), and combined Likelihood Ratio Test (cLRT). This work aimed to assess seven NLMEM approaches in the same framework: treatment effect assessment in balanced two-armed designs using real natural history data with or without the addition of simulated treatment effect. The approaches are MAD, IMA, cLRT, standard model selection (STDs), structural similarity selection (SSs), randomized cLRT (rcLRT), and model-averaging across placebo and drug models (MAPD). The assessment included type I error, using Alzheimer's Disease Assessment Scale-cognitive (ADAS-cog) scores from 817 untreated patients and power and accuracy in the treatment effect estimates after the addition of simulated treatment effects. The model selection and averaging among a set of pre-selected candidate models were driven by the Akaike information criteria (AIC). The type I error rate was controlled only for IMA and rcLRT; the inflation observed otherwise was explained by the placebo model misspecification and selection bias. Both IMA and rcLRT had reasonable power and accuracy except under a low typical treatment effect.

Reference and target sample age distribution impacts between model types in dental developmental age estimation

The performance of age estimation methods may vary due to a combination of method- and sample-related factors. Method development and evaluation necessitates understanding what influences these factors have on age estimation outcomes. In the specific context of juvenile dental age estimation, we used a single dataset and complete factorial design to systematically test four potential sources of difference: age distributions of reference and target sample (uniform, unimodal, U-shaped), Bayesian (multivariate Bayesian cumulative probit) vs. classical regression modeling (multivariate adaptive regression splines i.e. MARS), and model selection bias. The dataset consisted of 850 sets of left mandibular molar scores from London children 5-18years old. True age and estimated age intervals in target samples were compared for bias, root-mean-squared error, precision, and accuracy using locally weighted smoothing of performance measures across the age range and means of performance metrics between factor-level combinations. We found interactions of model type, reference distribution, and target distribution. MARS models showed consistent evidence of age mimicry. Central tendency of the reference sample corresponded with increased bias while central tendency of the target sample corresponded with reduced RMSE and reduced precision for both model types. We found evidence of model selection bias, mitigated through averaging model metrics. We conclude that reference and target sample distribution influences and model selection bias are sufficient to cause difference in model performance within a single population. We suggest using Bayesian modeling, drawing uniform reference and target samples, and calculating test error on a hold-out sample to mitigate these challenges in method development.

Smartphone-based hard-braking event detection at scale for road safety services

Road crashes are the sixth leading cause of lost disability-adjusted life-years (Dalys) worldwide. One major challenge in traffic safety research is the sparsity of crashes, which makes it difficult to achieve a fine-grain understanding of crash causations and predict future crash risk in a timely manner. Hard-braking events have been widely used as a safety surrogate due to their relatively high prevalence and ease of detection with embedded vehicle sensors. As an alternative to using sensors fixed in vehicles, this paper presents a scalable approach for detecting hard-braking events using the kinematics data collected from smartphone sensors. We train a Transformer-based machine learning model for hard-braking event detection using concurrent sensor readings from smartphones and vehicle sensors from drivers who connect their phone to the vehicle while navigating in Google Maps. The detection model shows superior performance with a 0.83 Area under the Precision–Recall Curve (Pr-auc), which is 3.8×better than a Gps speed-based heuristic model, and 166.6×better than an accelerometer-based heuristic model. The detected hard-braking events are strongly correlated with crashes from publicly available datasets, supporting their use as a safety surrogate. In addition, we conduct model fairness and selection bias evaluation to ensure that the safety benefits are equally shared. The developed methodology can benefit many safety applications such as identifying safety hot spots at road network level, evaluating the safety of new user interfaces, as well as using routing to improve traffic safety.

Dose/exposure-response modeling in dose titration trials: Overcoming the titration paradox.

Response-based dose individualization or dose titration is a powerful approach to achieve precision dosing. Yet, titration as an individualization strategy is underused in drug development and therefore not reflected in labeling, possibly partly because of the data analysis challenges associated with assessing dose/exposure-response under dose titration, where there is an inherent risk of selection bias because poor responders would get high doses, whereas good responders would get low doses. In a recent article, this issue of selection bias was termed the "titration paradox." In this study, we demonstrate by means of simulation that the titration paradox may be overcome if longitudinal data from dose titration trials is analyzed using a population approach that accounts for the fact that dose/exposure-response relationships differ between individuals. We show that with an appropriate sample size and missing data missing at random, stepwise dose/exposure-response modeling based on data obtained under dose titration is not by definition subject to model selection bias or bias in parameter estimates. We also illustrate the challenges of graphical exploration of data obtained under dose titration and discuss the use of model diagnostic tools with such data. Our study shows that if, at every timepoint in the course of a trial, there is a clear causal relationship between the response and the dose/exposure level, and a population approach is used, it will in many cases be possible to develop, estimate, and appropriately qualify a dose/exposure-response model also for data obtained under dose titration, thus overcoming the titration paradox.

Integrative Bayesian models using Post-selective inference: A case study in radiogenomics.

Integrative analyses based on statistically relevant associations between genomics and a wealth of intermediary phenotypes (such as imaging) provide vital insights into their clinical relevance in terms of the disease mechanisms. Estimates for uncertainty in the resulting integrative models are however unreliable unless inference accounts for the selection of these associations with accuracy. In this paper, we develop selection-aware Bayesian methods, which (1) counteract the impact of model selection bias through a "selection-aware posterior" in a flexible class of integrative Bayesian models post a selection of promising variables via ℓ1 -regularized algorithms; (2) strike an inevitable trade-off between the quality of model selection and inferential power when the same data set is used for both selection and uncertainty estimation. Central to our methodological development, a carefully constructed conditional likelihood function deployed with a reparameterization mapping provides tractable updates when gradient-based Markov chain Monte Carlo (MCMC) sampling is used for estimating uncertainties from the selection-aware posterior. Applying our methods to a radiogenomic analysis, we successfully recover several important gene pathways and estimate uncertainties for their associations with patient survivaltimes.

Smartphone-Based Hard-Braking Event Detection at Scale for Road Safety Services

Road crashes are the sixth leading cause of lost disability-adjusted life-years (DALYs) worldwide. One major challenge in traffic safety research is the sparsity of crashes, which makes it difficult to achieve a fine-grain understanding of crash causations and predict future crash risk in a timely manner. Hard-braking events have been widely used as a safety surrogate due to their relatively high prevalence and ease of detection with embedded vehicle sensors. As an alternative to using sensors fixed in vehicles, this paper presents a scalable approach for detecting hard-braking events using the kinematics data collected from smartphone sensors. We train a Transformer-based machine learning model for hard-braking event detection using concurrent sensor readings from smartphones and vehicle sensors from drivers who connect their phone to the vehicle while navigating in Google Maps. The detection model shows superior performance with a $0.83$ Area under the Precision-Recall Curve (PR-AUC), which is $3.8\times$better than a GPS speed-based heuristic model, and $166.6\times$better than an accelerometer-based heuristic model. The detected hard-braking events are strongly correlated with crashes from publicly available datasets, supporting their use as a safety surrogate. In addition, we conduct model fairness and selection bias evaluation to ensure that the safety benefits are equally shared. The developed methodology can benefit many safety applications such as identifying safety hot spots at road network level, evaluating the safety of new user interfaces, as well as using routing to improve traffic safety.

Locally Robust Semiparametric Estimation

Many economic and causal parameters depend on nonparametric or high dimensional first steps. We give a general construction of locally robust/orthogonal moment functions for GMM, where first steps have no effect, locally, on average moment functions. Using these orthogonal moments reduces model selection and regularization bias, as is important in many applications, especially for machine learning first steps. Also, associated standard errors are robust to misspecification when there is the same number of moment functions as parameters of interest. We use these orthogonal moments and cross‐fitting to construct debiased machine learning estimators of functions of high dimensional conditional quantiles and of dynamic discrete choice parameters with high dimensional state variables. We show that additional first steps needed for the orthogonal moment functions have no effect, globally, on average orthogonal moment functions. We give a general approach to estimating those additional first steps. We characterize double robustness and give a variety of new doubly robust moment functions. We give general and simple regularity conditions for asymptotic theory.

Correcting for sample selection bias in Bayesian distributional regression models

In the presence of non-randomly selected samples, many statistical models, including standard regression models, can fail. In particular, without accounting for the underlying selection process estimates might be biased. Sample selection models can correct this bias when an informative selection process governs the availability of the outcome of interest. A copula based approach is presented for correcting the sample selection bias in Bayesian structured additive distributional regression models. This framework relaxes the distributional assumption on the response of the linear or the generalized linear model and models all distributional parameters as functions of the covariates. Covariate effects are not limited to being purely linear and other effect types, such as smooth functional effects, are available. As a consequence, the approach presented provides increased flexibility with respect to the dependence structure, the available predictor specifications and the choice of the marginal distributions compared to Heckman's classic sample selection model. To facilitate estimation in such a complex model, a fully Bayesian approach based on Markov chain Monte Carlo simulations is developed and the presented methodology is empirically evaluated. Furthermore, the introduced approach is compared to a frequentist competitor and an application on a data set from psychological judge-advisor research is presented.

Mitigating selection bias in organ allocation models

BackgroundThe lung allocation system in the U.S. prioritizes lung transplant candidates based on estimated pre- and post-transplant survival via the Lung Allocation Scores (LAS). However, these models do not account for selection bias, which results from individuals being removed from the waitlist due to receipt of transplant, as well as transplanted individuals necessarily having survived long enough to receive a transplant. Such selection biases lead to inaccurate predictions.MethodsWe used a weighted estimation strategy to account for selection bias in the pre- and post-transplant models used to calculate the LAS. We then created a modified LAS using these weights, and compared its performance to that of the existing LAS via time-dependent receiver operating characteristic (ROC) curves, calibration curves, and Bland-Altman plots.ResultsThe modified LAS exhibited better discrimination and calibration than the existing LAS, and led to changes in patient prioritization.ConclusionsOur approach to addressing selection bias is intuitive and can be applied to any organ allocation system that prioritizes patients based on estimated pre- and post-transplant survival. This work is especially relevant to current efforts to ensure more equitable distribution of organs.

Model Selection and Model Averaging for Mixed-Effects Models with Crossed Random Effects for Subjects and Items

A good deal of experimental research is characterized by the presence of random effects on subjects and items. A standard modeling approach that includes such sources of variability is the mixed-effects models (MEMs) with crossed random effects. However, under-parameterizing or over-parameterizing the random structure of MEMs bias the estimations of the Standard Errors (SEs) of fixed effects. In this simulation study, we examined two different but complementary perspectives: model selection with likelihood-ratio tests, AIC, and BIC; and model averaging with Akaike weights. Results showed that true model selection was constant across the different strategies examined (including ML and REML estimators). However, sample size and variance of random slopes were found to explain true model selection and SE bias of fixed effects. No relevant differences in SE bias were found for model selection and model averaging. Sample size and variance of random slopes interacted with the estimator to explain SE bias. Only the within-subjects effect showed significant underestimation of SEs with smaller number of items and larger item random slopes. SE bias was higher for ML than REML, but the variability of SE bias was the opposite. Such variability can be translated into high rates of unacceptable bias in many replications.

The consequences of checking for zero‐inflation and overdispersion in the analysis of count data

Abstract Count data are ubiquitous in ecology and the Poisson generalized linear model (GLM) is commonly used to model the association between counts and explanatory variables of interest. When fitting this model to the data, one typically proceeds by first confirming that the model assumptions are satisfied. If the residuals appear to be overdispersed or if there is zero‐inflation, key assumptions of the Poison GLM may be violated and researchers will then typically consider alternatives to the Poison GLM. An important question is whether the potential model selection bias introduced by this data‐driven multi‐stage procedure merits concern. Here we conduct a large‐scale simulation study to investigate the potential consequences of model selection bias that can arise in the simple scenario of analysing a sample of potentially overdispersed, potentially zero‐inflated, count data. Specifically, we investigate model selection procedures recently recommended by Blasco‐Moreno et al. (2019) using either a series of score tests or information theoretic criteria to select the best model. We find that, when sample sizes are small, model selection based on preliminary score tests (or information theoretic criteria, e.g. AIC, BIC) can lead to potentially substantial inflation of false positive rates (i.e. type 1 error inflation). When sample sizes are sufficiently large, model selection based on preliminary score tests, is not problematic. Ignoring the possibility of overdispersion and zero‐inflation during data analyses can lead to invalid inference. However, if one does not have sufficient power to test for overdispersion and zero‐inflation, post hoc model selection may also lead to substantial bias. This ‘catch‐22’ suggests that, if sample sizes are small, a healthy skepticism is warranted whenever one rejects the null hypothesis of no association between a given outcome and covariate.

Major shareholders’ trust and market risk: Substituting weak institutions with trust

This study examines the impact of foreign controlling shareholder trust on firm market risk using two measures of total and idiosyncratic risk. An extensive global sample of 12,496 firm-year observations from 43 countries is employed. The results show that firms controlled by foreign trusting shareholders display lower levels of risk in both market measures. Trust appears more important for firms based in countries with a less favourable institutional environment, whereby it varies with the investment horizon of foreign controlling shareholders. The results are robust after controlling for cultural measures, endogeneity, selection bias and alternate model specifications.

To list or not to list, that is the question: new evidence from the performance of Vietnamese commercial banks

PurposeThis paper aims to investigate the decisions of listing for Vietnamese banks and the impact of listing on bank performance.Design/methodology/approachA longitudinal data set of 30 commercial banks in the period of 2006–2018 with various univariate and multivariate tests is used.FindingsThis study found that listing is positively associated with bank profitability. The results are consistent even after the control for potential endogeneity problems by propensity score matching methodology and Heckman selection bias models. Further analysis suggests some new alternative channels for the positive impact, namely, the increased quality of information disclosure, technological development and income diversification of commercial banks after listing.Practical implicationsHence, this paper provides recommendations and policy implications for regulatory bodies regarding the listing of commercial banks in Vietnam.Originality/valueThe contributions to the literature are three-folds. First, this study contributes to a strand of literature on the impact of going public [initial public offering (IPO)/listing] of financial institutions on their performance. While the literature on non-financial firm performance post-going public is ample, few have directly considered the IPO/listing of banks and other financial institutions. Second, in further looking at the impact of listing on bank performance, this study also sheds some light on the new possible channels of the effect and provides evidence of new channels. Then, last but not least, the case of Vietnam could possibly yield interesting results for a transitory stock market. From the evidence, the recommendations and policy implications for a listing of Vietnamese banks are provided.