Model Selection Methods Research Articles

An efficient deep learning method for amino acid substitution model selection.

Amino acid substitution models play an important role in studying the evolutionary relationships among species from protein sequences. The amino acid substitution model consists of a large number of parameters; therefore, it is estimated from hundreds or thousands of alignments. Both general models and clade-specific models have been estimated and widely used in phylogenetic analyses. The maximum likelihood method is normally used to select the best fit model for a specific protein alignment under the study. A number of studies have discussed theoretical concerns as well as computational burden of the maximum likelihood methods in model selection. Recently, machine learning methods have been proposed for selecting nucleotide models. In this paper, we propose a method to measure substitution rates among amino acids (called summary statistics) from protein alignments to efficiently train a deep learning network of so-called ModelDetector for detecting amino acid substitution models. The ModelDetector network was trained from 2,246,400 alignments on a computer with 8 cores (without GPU) in about 3.3 hours. Experiments on simulation data showed that the accuracy of the ModelDetector was comparable with that of the maximum likelihood method ModelFinder. It was orders of magnitude faster than the maximum likelihood method in inferring amino acid substitution models and able to analyze genome alignments with millions of sites in minutes. The results indicate that the deep learning network can play as a promising tool for amino acid substitution model selection.

Modeling of Long-term Afterglow Counterparts to Gravitational Wave Events: The Full View of GRB 170817A

The arrival of gravitational wave astronomy and a growing number of time-domain-focused observatories are set to lead to an increasing number of detections of short gamma-ray bursts (GRBs) launched with a moderate inclination to Earth. Being nearby events, these are also prime candidates for very long-term follow-up campaigns and very long-baseline interferometry, which has implications for multi-messenger modeling, data analysis, and statistical inference methods applied to these sources. Here, we present a comprehensive modeling update that directly incorporates into afterglowpy astrometric observations of the GRB position, Poissonian statistics for faint sources, and modeling of a trans-relativistic population of electrons. We use the revolutionary event GW170817 to demonstrate the impact of these extensions both for the best-fit physics parameters and model selection methods that assess the statistical significance of additional late-time emission components. By including in our analysis the latest Chandra X-ray observations of GRB 170817A, we find only weak evidence (≲2σ) for a new emission component at late times, which makes for a slightly more natural fit to the centroid evolution and prediction for the external medium density.

Improving equivalent scores for clinical neuropsychology: a new method for regression model selection.

Equivalent Scores (ES) represent a statistical gold standard to obtain thresholds for clinical inference from normative data. The procedure to obtain ES requires a preliminary mandatory step: to perform a regression model selection to obtain adjusted scores that take into account age, education, and sex. The current article, starting from theoretical considerations, focuses on this step and proposes a new and improved regression model selection method. Results from data simulation show that the newly proposed method outperforms the current one on a wide range of simulation parameters and conditions, leading to better performance in classifying impaired or unimpaired performances, and more precise ES. The article is associated with an online app and R code to allow to easily apply the method to other normative data. This new model selection procedure can be easily incorporated also with other regression-based norm approaches.

Identification of an additive interaction using parameter regularization and model selection in epidemiology

Background In epidemiology, indicators such as the relative excess risk due to interaction (RERI), attributable proportion (AP), and synergy index (S) are commonly used to assess additive interactions between two variables. However, the results of these indicators are sometimes inconsistent in real world applications and it may be difficult to draw conclusions from them. Method Based on the relationship between the RERI, AP, and S, we propose a method with consistent results, which are achieved by constraining $e^{\theta_3}-e^{\theta_1}-e^{\theta_2}+1=0$, and the interpretation of the results is simple and clear. We present two pathways to achieve this end: one is to complete the constraint by adding a regular penalty term to the model likelihood function; the other is to use model selection. Result Using simulated and real data, our proposed methods effectively identified additive interactions and proved to be applicable to real-world data. Simulations were used to evaluate the performance of the methods in scenarios with and without additive interactions. The penalty term converged to 0 with increasing λ, and the final models matched the expected interaction status, demonstrating that regularized estimation could effectively identify additive interactions. Model selection was compared with classical methods (delta and bootstrap) across various scenarios with different interaction strengths, and the additive interactions were closely observed and the results aligned closely with bootstrap results. The coefficients in the model without interaction adhered to a simplifying equation, reinforcing that there was no significant interaction between smoking and alcohol use on oral cancer risk. Conclusion In summary, the model selection method based on the Hannan-Quinn criterion (HQ) appears to be a competitive alternative to the bootstrap method for identifying additive interactions. Furthermore, when using RERI, AP, and S to assess the additive interaction, the results are more consistent and the results are simple and easy to understand.

A method for temperature sensor and model selection for machine tool thermal error modelling using ANFIS and ANN

AbstractThermal errors account for a significant part of the dimensional errors of components produced by precision machine tools. These errors are commonly compensated using predictions from temperature-based empirical models. The accuracy and robustness of these predictions are affected by the locations of temperature sensors used to obtain the model’s input data. Methods for sensor selection found in literature are often difficult to replicate and automate because they require tuning of several hyperparameters. This work presents a sensor and model selection approach that uses proper orthogonal decomposition (POD) and QR pivoting to select a subset of sensors that have been preinstalled in the machine tool as possible model inputs. These sensors are then sorted according to their correlation with the thermal error being modelled. The final set of inputs and thermal error model structure is chosen using Bayesian Information Criterion (BIC) to limit model overfitting. The approach was tested by modelling the Y-axis thermal error measured from air-cutting experiments performed under different spindle speeds and feed rates. This enabled determination of the structure and the choice inputs for Adaptive Neuro Fuzzy Inference System (ANFIS) and Artificial Neural Network (ANN) thermal error models. The accuracy of these models was compared to that of models trained using inputs selected by conventional approaches: the least absolute shrinkage and selection operator (LASSO), fuzzy c-means clustering (FCM), and principal component regression (PCR). The presented approach had better or comparable results to the conventional approaches while using fewer inputs. The presented approach is also well suited for automation compared to conventional approaches that require expert input.

Evaluation of model-integrated evidence approaches for pharmacokinetic bioequivalence studies using model averaging methods.

Conventional approaches for establishing bioequivalence (BE) between test and reference formulations using non-compartmental analysis (NCA) may demonstrate low power in pharmacokinetic (PK) studies with sparse sampling. In this case, model-integrated evidence (MIE) approaches for BE assessment have been shown to increase power, but may suffer from selection bias problems if models are built on the same data used for BE assessment. This work presents model averaging methods for BE evaluation and compares the power and type I error of these methods to conventional BE approaches for simulated studies of oral and ophthalmic formulations. Two model averaging methods were examined: bootstrap model selection and weight-based model averaging with parameter uncertainty from three different sources, either from a sandwich covariance matrix, a bootstrap, or from sampling importance resampling (SIR). The proposed approaches increased power compared with conventional NCA-based BE approaches, especially for the ophthalmic formulation scenarios, and were simultaneously able to adequately control type I error. In the rich sampling scenario considered for oral formulation, the weight-based model averaging method with SIR uncertainty provided controlled type I error, that was closest to the target of 5%. In sparse-sampling designs, especially the single sample ophthalmic scenarios, the type I error was best controlled by the bootstrap model selection method.

Identifying Brain Network Structure for an fMRI Effective Connectivity Study Using the Least Absolute Shrinkage and Selection Operator (LASSO) Method.

Background: Studying causality relationships between different brain regions using the fMRI method has attracted great attention. To investigate causality relationships between different brain regions, we need to identify both the brain network structure and the influence magnitude. Most current methods concentrate on magnitude estimation, but not on identifying the connection or structure of the network. To address this problem, we proposed a nonlinear system identification method, in which a polynomial kernel was adopted to approximate the relation between the system inputs and outputs. However, this method has an overfitting problem for modelling the input-output relation if we apply the method to model the brain network directly. Methods: To overcome this limitation, this study applied the least absolute shrinkage and selection operator (LASSO) model selection method to identify both brain region networks and the connection strength (system coefficients). From these coefficients, the causality influence is derived from the identified structure. The method was verified based on the human visual cortex with phase-encoded designs. The functional data were pre-processed with motion correction. The visual cortex brain regions were defined based on a retinotopic mapping method. An eight-connection visual system network was adopted to validate the method. The proposed method was able to identify both the connected visual networks and associated coefficients from the LASSO model selection. Results: The result showed that this method can be applied to identify both network structures and associated causalities between different brain regions. Conclusions: System identification with LASSO model selection algorithm is a powerful approach for fMRI effective connectivity study.

Selecting the Best Pharmacokinetic Models for a Priori Model-Informed Precision Dosing with Model Ensembling.

When utilizing population pharmacokinetic (popPK) models for a priori dosage individualization, selecting the best model is crucial to obtain adequate doses. We developed and evaluated several model-selection and ensembling methods, using external evaluation on the basis of therapeutic drug monitoring (TDM) samples to identify the best (set of) models per patient for a priori dosage individualization. PK data and models describing both hospitalized patients (n = 134) receiving continuous vancomycin (26 models) and patients (n = 92) receiving imatinib in an outpatient setting (12 models) are included. Target attainment of four model-selection methods was compared with standard dosing: the best model based on external validation, uninformed model ensembling, model ensembling using a weighting scheme on the basis of covariate-stratified external evaluation, and model selection using covariates in decision trees that were subsequently ensembled. Overall, the use of PK models improved the proportion of patients exposed to concentrations within the therapeutic window for both cohorts. Relative improvement of proportion on target for best model, unweighted, weighted, and decision trees were - 7.0%, 2.3%, 11.4%, and 37.0% (vancomycin method-development); 23.2%, 7.9%, 15.6%, and, 77.2% (vancomycin validation); 40.7%, 50.0%, 59.5%, and 59.5% (imatinib method-development); and 19.0%, 28.5%, 38.0%, and 23.8% (imatinib validation), respectively. The best (set of) models per patient for a priori dosage individualization can be identified using a relatively small set of TDM samples as external evaluation. Adequately performing popPK models were identified while also excluding poor-performing models. Dose recommendations resulted in more patients within the therapeutic range for both vancomycin and imatinib. Prospective validation is necessary before clinical implementation.

Influence of device configuration and noise on a machine learning predictor for the selection of nanoparticle small-angle X-ray scattering models.

Small-angle X-ray scattering (SAXS) is a widely used method for nanoparticle characterization. A common approach to analysing nanoparticles in solution by SAXS involves fitting the curve using a parametric model that relates real-space parameters, such as nanoparticle size and electron density, to intensity values in reciprocal space. Selecting the optimal model is a crucial step in terms of analysis quality and can be time-consuming and complex. Several studies have proposed effective methods, based on machine learning, to automate the model selection step. Deploying these methods in software intended for both researchers and industry raises several issues. The diversity of SAXS instrumentation requires assessment of the robustness of these methods on data from various machine configurations, involving significant variations in the q-space ranges and highly variable signal-to-noise ratios (SNR) from one data set to another. In the case of laboratory instrumentation, data acquisition can be time-consuming and there is no universal criterion for defining an optimal acquisition time. This paper presents an approach that revisits the nanoparticle model selection method proposed by Monge et al. [Acta Cryst. (2024), A80, 202-212], evaluating and enhancing its robustness on data from device configurations not seen during training, by expanding the data set used for training. The influence of SNR on predictor robustness is then assessed, improved, and used to propose a stopping criterion for optimizing the trade-off between exposure time and data quality.

A statistical modelling approach to feedforward neural network model selection

Feedforward neural networks (FNNs) can be viewed as non-linear regression models, where covariates enter the model through a combination of weighted summations and non-linear functions. Although these models have some similarities to the approaches used within statistical modelling, the majority of neural network research has been conducted outside of the field of statistics. This has resulted in a lack of statistically based methodology, and, in particular, there has been little emphasis on model parsimony. Determining the input layer structure is analogous to variable selection, while the structure for the hidden layer relates to model complexity. In practice, neural network model selection is often carried out by comparing models using out-of-sample performance. However, in contrast, the construction of an associated likelihood function opens the door to information-criteria-based variable and architecture selection. A novel model selection method, which performs both input- and hidden-node selection, is proposed using the Bayesian information criterion (BIC) for FNNs. The choice of BIC over out-of-sample performance as the model selection objective function leads to an increased probability of recovering the true model, while parsimoniously achieving favourable out-of-sample performance. Simulation studies are used to evaluate and justify the proposed method, and applications on real data are investigated.

Non-Gaussian Liability Distribution for Depression in the General Population.

Unlike depression sum scores, the underlying risk for depression is typically assumed to be normally distributed across the general population. To assess the true empirical shape of depression risk, we created a continuous-valued estimate of the latent depression density, using the Davidian-Curve Item Response Theory (DC-IRT) and the National Health and Nutrition Examination Survey (NHANES) cohorts from 2005 to 2018 (n = 36,244 on the Nine-item Patient Health Questionnaire; PHQ-9). We conducted simulations to investigate the performance of DC-IRT for large samples and realistic items. The method can recover complex latent-risk distributions even when they are not evident from sum scores. However, estimation accuracy for different sample sizes depends on the method of model selection. In addition to full-data analysis, random samples of a few thousand observations were drawn for analysis. The latent shape of depression was left-skewed and bimodal in both investigations, indicating that the latent-normality assumption does not hold for depression.

Assessing model selection techniques for distributions use in hydrological extremes in the presence of trimming and subsampling

A suitable probability distribution is required to quantify and estimate hydraulic structure design for risk evaluation and management. The inability of model selection criteria to differentiate, in some cases, among candidate distributions used in the analysis of hydrological extremes is often criticised. This study verifies, with the aid of model selection techniques, the potential utility of trimming and subsampling in distinguishing between candidate distributions, which might not be feasible using the traditional goodness of fit method alone, when samples available are small. The performance of the proposed method is evaluated through its application to real and simulated yearly peak rainfall datasets. The proposed approach is then compared with several standard model selection techniques. Results show that the model selection techniques with the aid of subsampling are effective in identifying the true parent distribution for the untrimmed samples given a two-parameter distribution; contrarily, they are inefficient where a distribution with a three-parameter is the parent distribution. However, as trimming is introduced, all model selection methods recognise the true parent distribution for a three-parameter distribution. Overall, utilising trimming and subsampling with the aid of model selection methods yields promising outcomes in the analysis of hydrological extreme frequencies. Drawing from the results of numerical simulation and examination of observed data, the use of trimming and subsampling can be a viable tool in differentiating among candidate distributions used in the investigation of hydrological extreme frequencies.

Novel autoantibodies help diagnose anti-SSA antibody negative Sjögren disease and predict abnormal labial salivary gland pathology

ObjectivesSjögren disease (SjD) diagnosis often requires either positive anti-SSA antibodies or a labial salivary gland biopsy with a positive focus score (FS). One-third of patients with SjD lack anti-SSA antibodies...

Heteroscedastic Gaussian Process Regression for material structure–property relationship modeling

Uncertainty quantification is a critical aspect of machine learning models for material property predictions. Gaussian Process Regression (GPR) is a popular technique for capturing uncertainties, but most existing models assume homoscedastic aleatoric uncertainty (noise), which may not adequately represent the heteroscedastic behavior observed in real-world datasets. Heteroscedasticity arises from various factors, such as measurement errors and inherent variability in material properties. Ignoring heteroscedasticity can lead to lower model performance, biased uncertainty estimates, and inaccurate predictions. Existing Heteroscedastic Gaussian Process Regression (HGPR) models often employ complicated structures to capture input-dependent noise but may lack interpretability. In this paper, we propose an HGPR approach that combines GPR with polynomial regression-based noise modeling to capture and quantify uncertainties in material property predictions while providing interpretable noise models. We demonstrate the effectiveness of our approach on both synthetic and physics-based simulation datasets, including mechanical properties (effective stress) of porous materials. We also introduce an approximated expected log predictive density method for model selection, which eliminates the need for retraining the model during leave-one-out cross-validation, allowing for efficient hyperparameter tuning and model evaluation. By capturing heteroscedastic behavior, enhancing interpretability, and improving model selection, our approach contributes to the development of more robust and reliable machine learning models for material property predictions, enabling informed decision-making in material design and optimization.

Learning under singularity: an information criterion improving WBIC and sBIC

AbstractWe introduce a novel information criterion, termed learning under singularity (LS), designed to enhance the functionality of the widely applicable Bayes information criterion (WBIC) and the singular Bayesian information criterion (sBIC). LS is effective without regularity constraints and demonstrates stability. Watanabe defined a statistical model or a learning machine as regular if the mapping from a parameter to a probability distribution is one-to-one and its Fisher information matrix is positive definite. In contrast, models not meeting these conditions are termed singular. Over the past decade, several information criteria for singular cases have been proposed, including WBIC and sBIC. WBIC is applicable in non-regular scenarios but faces challenges with large sample sizes and redundant estimation of known learning coefficients. Conversely, sBIC is limited in its broader application due to its dependence on maximum likelihood estimates. LS addresses these limitations by enhancing the utility of both WBIC and sBIC. It incorporates the empirical loss from the widely applicable information criterion (WAIC) to represent the goodness of fit to the statistical model, along with a penalty term similar to that of sBIC. This approach offers a flexible and robust method for model selection, free from regularity constraints.

Discussion of the 2023 Hansen Lecture: “Model Selection and Its Important Roles in Surveys”

Abstract The 31st Morris Hansen Lecture gave a broad overview of some fundamental issues regarding the roles of statistical models in surveys, with an emphasis on a method of model selection known as the fence. This discussion provides some general lecture remarks and expands on the possible applicability of the fence methods in small area estimation. Unit-level and area-level small area estimation modeling illustrations are provided. A general variable selection framework for small area estimation is presented, along with recent application studies and ideas for expansion to include the fence methods.

A GARCH model selection and estimation method based on neural network with the loss function of mean square error and model confidence set

AbstractThis paper proposes a method that uses mean square error (MSE) and model confidence set (MCS) as the loss function of back‐propagation neural network (BPNN), aiming to train and find a generalized autoregressive conditional heteroskedastic (GARCH) model that has the best forecasting performance of a time series. Combining MSE and the p‐value of MCS can not only estimate better parameters for the GARCH models but also find the best GARCH model to forecast the volatility of a time series. Meanwhile, we divide a time series into several parts and use each part as the input of the BPNN. Through the BPNN, each part of the time series will be turned into several forecasting values. These values will be used to calculate the MSE and the p‐value of MCS, which will then be used to update the parameters of the BPNN. In the end, we use MCS to choose the best GARCH model among the trained GARCH models and compare this method with maximum likelihood estimation (MLE) and the generalized least squares estimation (GLS). The result shows that the p‐value of MCS of the best model estimated by this method is higher than the p‐value of MCS of the best model estimated by MLE and GLS. According to the theory of MCS, a model that has a larger p‐value does have a better forecasting performance. The method proposed by this paper can provide a potential application of neural network in GARCH model forecasting and estimation.

Evidence-Based Certification of Quantum Dimensions.

Identifying a reasonably small Hilbert space that completely describes an unknown quantum state is crucial for efficient quantum information processing. We introduce a general dimension-certification protocol for both discrete and continuous variables that is fully evidence based, relying solely on the experimental data collected and no other unjustified assumptions whatsoever. Using the Bayesian concept of relative belief, we take the effective dimension of the state as the smallest one such that the posterior probability is larger than the prior, as dictated by the data. The posterior probabilities associated with the relative-belief ratios measure the strength of the evidence provide by these ratios so that we can assess whether there is weak or strong evidence in favor or against a particular dimension. Using experimental data from spectral-temporal and polarimetry measurements, we demonstrate how to correctly assign Bayesian plausible error bars for the obtained effective dimensions. This makes relative belief a conservative and easy-to-use model-selection method for any experiment.

Model Selection Posterior Predictive Model Checking via Limited‐Information Indices for Bayesian Diagnostic Classification Modeling

AbstractRecently, Bayesian diagnostic classification modeling has been becoming popular in health psychology, education, and sociology. Typically information criteria are used for model selection when researchers want to choose the best model among alternative models. In Bayesian estimation, posterior predictive checking is a flexible Bayesian model evaluation tool, which allows researchers to detect Q‐matrix misspecification. However, model selection methods using posterior predictive checking (PPC) for Bayesian DCM are not well investigated. Thus, this research aims to propose a novel model selection approach using posterior predictive checking with limited‐information statistics for selecting the correct Q‐matrix. A simulation study was conducted to examine the performance of the proposed method. Furthermore, an empirical example was provided to illustrate how it can be used in real scenarios.

Model selection of GLMMs in the analysis of count data in single-case studies: A Monte Carlo simulation.

Generalized linear mixed models (GLMMs) have great potential to deal with count data in single-case experimental designs (SCEDs). However, applied researchers have faced challenges in making various statistical decisions when using such advanced statistical techniques in their own research. This study focused on a critical issue by investigating the selection of an appropriate distribution to handle different types of count data in SCEDs due to overdispersion and/or zero-inflation. To achieve this, I proposed two model selection frameworks, one based on calculating information criteria (AIC and BIC) and another based on utilizing a multistage-model selection procedure. Four data scenarios were simulated including Poisson, negative binominal (NB), zero-inflated Poisson (ZIP), and zero-inflated negative binomial (ZINB). The same set of models (i.e., Poisson, NB, ZIP, and ZINB) were fitted for each scenario. In the simulation, I evaluated 10 model selection strategies within the two frameworks by assessing the model selection bias and its consequences on the accuracy of the treatment effect estimates and inferential statistics. Based on the simulation results and previous work, I provide recommendations regarding which model selection methods should be adopted in different scenarios. The implications, limitations, and future research directions are also discussed.