Model Selection Indices Research Articles

Under-Fitting and Over-Fitting: The Performance of Bayesian Model Selection and Fit Indices in SEM

We extended current knowledge by examining the performance of several Bayesian model fit and comparison indices through a simulation study using the confirmatory factor analysis. Our goal was to determine whether commonly implemented Bayesian indices can detect specification errors. Specifically, we wanted to uncover any differences in detecting under-fitting or over-fitting a model. We examined a conventional Bayesian fit index (the posterior predictive p-value), approximate Bayesian fit indices (Bayesian RMSEA, CFI, and TLI), and model comparison indices (BIC and DIC). We varied the type and severity of model mis-specification, sample size, and priors. We focused on the ability of these indices to detect model under- or over-fitting. We provide practical advice for applied researchers regarding how to assess and compare models using these common indices implemented in the Bayesian framework.

Performance of Model Fit and Selection Indices for Bayesian Piecewise Growth Modeling with Missing Data

The Bayesian piecewise growth model (PGM) is a useful class of models for analyzing nonlinear change processes that consist of distinct growth phases. In applications of Bayesian PGMs, it is important to accurately capture growth trajectories and carefully consider knot placements. The presence of missing data is another challenge researchers commonly encounter. To address these issues, one could use model fit and selection indices to detect misspecified Bayesian PGMs, and should give care to the potential impact of missing data on model evaluation. Here we conducted a simulation study to examine the impact of model misspecification and missing data on the performance of Bayesian model fit and selection indices (PPP-value, BCFI, BTLI, BRMSEA, BIC, and DIC), with an additional focus on prior sensitivity. Results indicated that (a) increasing the degree of model misspecification and amount of missing data aggravated the performance of indices in detecting misfit, and (b) different prior specifications had negligible impact on model assessment. We provide practical guidelines for researchers to facilitate effective implementation of Bayesian PGMs.

SOIL QUALITY AND HEIGHT DIAMETER MODEL ASSESSMENT IN THE MANAGEMENT OF GMELINA ARBOREA PLANTATION IN NASARAWA STATE UNIVERSITY, NIGERIA

Gmelina arborea is a very important timber species as it provides wood for furniture, serves as carbon sink and creates micro-climate with decrease soil temperatures enabling smaller niches to be formed within the forest ecosystem. Also, the nutrients composition of forest soil determines its stand structure and composition, growth rate and dominance, including other silvicultural practices. The study assessed soil quality and height-diameter (H-D) models in the management of Gmelina arborea plantation in Nasarawa State University. Thirty (30) temporary sample plots of 0.01 ha size were randomly selected from the sampling frame with 30% sampling intensity. Thereafter, the total height and the Dbh of each tree within the selected plots was measured for height-diameter model assessment. Also, ten (10) soil samples were randomly collected from the thirty (30) selected plots for H-D modeling in order to assess the soil properties of the plantation. Soil samples obtained were analyzed in the Faculty of Agriculture Laboratory to obtain data and further subjected to statistical analysis. The results of height-diameter models revealed that model one (1) with Akaike Information Criterion (62.19), Bayesian Information Criterion (66.39) and Residual Standard Error (0.64) had the lowest model selection indices when compared with other four models applied. Therefore, model one (1) was selected as the best and grand model for Gmelina aborea plantation in the study area. Also, the result of soil quality showed a significant difference between the soil properties in the study area. The Least Significant Difference result also showed that...

Estimating the Multidimensional Generalized Graded Unfolding Model with Covariates Using a Bayesian Approach.

Noncognitive constructs are commonly assessed in educational and organizational research. They are often measured by summing scores across items, which implicitly assumes a dominance item response process. However, research has shown that the unfolding response process may better characterize how people respond to noncognitive items. The Generalized Graded Unfolding Model (GGUM) representing the unfolding response process has therefore become increasingly popular. However, the current implementation of the GGUM is limited to unidimensional cases, while most noncognitive constructs are multidimensional. Fitting a unidimensional GGUM separately for each dimension and ignoring the multidimensional nature of noncognitive data may result in suboptimal parameter estimation. Recently, an R package bmggum was developed that enables the estimation of the Multidimensional Generalized Graded Unfolding Model (MGGUM) with covariates using a Bayesian algorithm. However, no simulation evidence is available to support the accuracy of the Bayesian algorithm implemented in bmggum. In this research, two simulation studies were conducted to examine the performance of bmggum. Results showed that bmggum can estimate MGGUM parameters accurately, and that multidimensional estimation and incorporating relevant covariates into the estimation process improved estimation accuracy. The effectiveness of two Bayesian model selection indices, WAIC and LOO, were also investigated and found to be satisfactory for model selection. Empirical data were used to demonstrate the use of bmggum and its performance was compared with three other GGUM software programs: GGUM2004, GGUM, and mirt.

Using Bayesian Piecewise Growth Curve Models to Handle Complex Nonlinear Trajectories

Bayesian growth curve modeling is a popular method for studying longitudinal data. In this study, we discuss a flexible extension, the Bayesian piecewise growth curve model (BPGCM), which allows the researcher to break up a trajectory into phases joined at change points called knots. By fitting BPGCMs, the researcher can specify three or more phases of growth without concern for model identification. Our goal is to provide substantive researchers with a guide for implementing this important class of models. We present a simple application of Bayesian linear BPGCMs to childrens' math achievement. Our tutorial includes Mplus code, strategies for specifying knots, and how to interpret model selection and fit indices. Extensions of the model are discussed.

Genetic parameters and selection of mango genotypes using the FAI-BLUP multitrait index

Mango (Mangifera indica L.) presents phenotypic diversity in relation to fruit quality. Due to this diversity, studies on performance and evaluation of genotypes help in the selection of new cultivars. The aim of this study was to select superior mango genotypes for fruit quality using a mixed model selection index. The fruit quality of two hundred and forty six different hybrids and four commercial cultivars (Ataulfo, Kent, Palmer and Tommy Atkins) was evaluated in three consecutive harvests. Estimates of variance components and predictions of phenotypic and genotypic values were obtained, as well as principal component analysis, factor analysis and genotype-ideotype distance (FAI-BLUP). Permanent variance among plants represented the largest portion of phenotypic variance for most traits. Estimates of the repeatability coefficient ranged from 0.09 to 0.96, being high for most variables. The selective accuracy predicted by the restricted maximum likelihood model (REML) ranged from 0.48 to 0.99, revealing a good degree of confidence in the inferences. Eigenvalue estimates revealed five principal components responsible for 77.72% of the total genetic variability. The greatest positive genetic gains were for fruit weight (37.29%), soluble solids/titratable acidity ratio (35.02%), b* (23.02%), pulp firmness (20.99%), ventral diameter (12.18%), transverse diameter (11.80%), skin luminosity (8.08%), soluble solids content (6.97%), length (4.28%), c* (3.08%), pulp luminosity (2.18%) and h* (0.88%). Negative genetic gains were observed for titratable acidity (-26.04%) and a* (-12.82%). Twenty-five genotypes were selected as the most promising based on factor analysis and FAI-BLUP index, promoting improvement and genetic gain in several traits. Between commercial cultivars, Ataulfo was the highest ranked genotype (29th), followed by Kent (36th), Palmer (243th) and Tommy Atkins (247th). The genotypes selected should be included in the next stages of the mango breeding program for high-quality fruits under irrigated cultivation in the Brazilian semiarid region.

Full-information item bifactor model for mathematical ability assessment in Chinese compulsory education quality monitoring.

This study focuses on the measurement of mathematical ability in the Chinese Compulsory Education Qualification Monitoring (CCEQM) framework using bifactor theory. First, we propose a full-information item bifactor (FIBF) model for the measurement of mathematical ability. Second, the performance of the FIBF model is empirically studied using a data set from three representative provinces were selected from CCEQM 2015-2017. Finally, Monte Carlo simulations are conducted to demonstrate the accuracy of the model evaluation indices and parameter estimation methods used in the empirical study. The obtained results are as follows: (1) The results for the four used model selection indices (AIC, SABIC, HQ, BIC) consistently showed that the fit of the FIBF model is better than that of the UIRT; (2) All of the estimated general and domain-specific abilities of the FIBF model have reasonable interpretations; (3) The model evaluation indices and parameter estimation methods exhibit excellent accuracy, indicating that the application of the FIBF model is technically feasible in large-scale testing projects.

How to Select the Best Fit Model among Bayesian Latent Growth Models for Complex Data

Bayesian approach is becoming increasingly important as it provides many advantages in dealing with complex data. However, there is no well-defined model selection criterion or index in a Bayesian context. To address the challenges, new indices are needed. The goal of this study is to propose new model selection indices and to investigate their performances in the framework of latent growth mixture models with missing data and outliers in a Bayesian context. We consider latent growth models because they are very flexible in modeling complex data and becoming increasingly popular in statistical, psychological, behavioral, and educational areas. Specifically, this study conducted five simulation studies to cover different cases, including latent growth curve models with missing data, latent growth curve models with missing data and outliers, growth mixture models with missing data and outliers, extended growth mixture models with missing data and outliers, and latent growth models with different classes. Simulation results show that almost all proposed indices can effectively identify the true model. This study also illustrated the application of these model selection indices in real data analysis.

Predictive Utility of Alternate Measures of Physical Activity and Diet for Overweight and Obesity in Low-Income Minority Women.

The purpose is to compare the predictive utility of alternate measures of diet and physical activity for overweight and obesity among low-income minority women. Cross-sectional analysis of baseline data from a cohort study. Three public housing developments in South Los Angeles.Subjects: Adult women (N = 425). Primary outcome-weight status (normal BMI, overweight, or obese). Primary predictors- diet: 24-hour dietary recalls (Healthy Eating Index), dietary screener (intake of specific food groups), and single-item survey question (diet quality); physical activity: accelerometry (minutes/day of moderate-to-vigorous activity), short recall questionnaire (minutes/week of moderate and vigorous activity), and single-item questions (days per week did exercise; self-assessment of overall activity level). Multinomial logistic regression models, controlling for socio-demographic covariates. Models are built up starting with least resource-intensive measures of diet and physical activity (single items) and sequentially adding more resource-intensive measures. Model performance is assessed via information-based model selection indices. Adjusted relative risk for obesity for single-item measures ranged from .61 to .64 for diet (P < .01) and from .80 to .81 for physical activity (P <.05). The added value of resource-intensive measures was negligible for physical activity and at best small for diet. Single-item questions for diet and physical activity can provide valuable information about risk for overweight and obesity in low-income minority women when more resource-intensive assessments are infeasible.

Performance of Model Fit and Selection Indices for Bayesian Structural Equation Modeling with Missing Data

ABSTRACT New model fit and evaluation tools have been introduced into the Bayesian structural equation modeling framework, including Bayesian versions of classic approximate fit measures (RMSEA, CFI, and TLI), as well as a new adjustment of the posterior predictive p-value to properly account for missing data. We examine the performance of these indices for model fit and selection through a simulation study. This study was designed to assess the performance of several indices in the context of model misspecification and missing data across different sample sizes. Findings suggest that Bayesian approximate fit indices may be better suited for model selection than they are as direct measures of model fit. Implications for applied researchers are discussed.

Improving Accuracy and Usage by Correctly Selecting: The Effects of Model Selection in Cognitive Diagnosis Computerized Adaptive Testing.

Decisions on how to calibrate an item bank might have major implications in the subsequent performance of the adaptive algorithms. One of these decisions is model selection, which can become problematic in the context of cognitive diagnosis computerized adaptive testing, given the wide range of models available. This article aims to determine whether model selection indices can be used to improve the performance of adaptive tests. Three factors were considered in a simulation study, that is, calibration sample size, Q-matrix complexity, and item bank length. Results based on the true item parameters, and general and single reduced model estimates were compared to those of the combination of appropriate models. The results indicate that fitting a single reduced model or a general model will not generally provide optimal results. Results based on the combination of models selected by the fit index were always closer to those obtained with the true item parameters. The implications for practical settings include an improvement in terms of classification accuracy and, consequently, testing time, and a more balanced use of the item bank. An R package was developed, named cdcatR, to facilitate adaptive applications in this context.

Bayesian Model Selection Methods for Multilevel IRT Models: A Comparison of Five DIC‐Based Indices

AbstractModel selection is important in any statistical analysis, and the primary goal is to find the preferred (or most parsimonious) model, based on certain criteria, from a set of candidate models given data. Several recent publications have employed the deviance information criterion (DIC) to do model selection among different forms of multilevel item response theory models (MLIRT). The majority of the practitioners use WinBUGS for implementing MCMC algorithms for MLIRT models, and the default version of DIC provided by WinBUGS focused on the measurement‐level parameters only. The results herein show that this version of DIC is inappropriate. This study introduces five variants of DIC as a model selection index for MLIRT models with dichotomous outcomes. Considering a multilevel IRT model with three levels, five forms of DIC are formed: first‐level conditional DIC computed from the measurement model only, which is the index given by many software packages such as WinBUGS; second‐level marginalized DIC and second‐level joint DIC computed from the second‐level model; and top‐level marginalized DIC and top‐level joint DIC computed from the entire model. We evaluate the performance of the five model selection indices via simulation studies. The manipulated factors include the number of groups, the number of second‐level covariates, the number of top‐level covariates, and the types of measurement models (one‐parameter vs. two‐parameter). Considering the computational viability and interpretability, the second‐level joint DIC is recommended for MLIRT models under our simulated conditions.

Detection of significant parameters for shell and tube heat exchanger using polynomial neural network approach

Shell-and-tube heat exchangers (STHXs) are extensively used for numerous industrial applications. The exchange efficiency (EE) of heat exchanger (HX) greatly influences production quality and plant performance. Present paper explores significant parameters to boost up the exchange efficiency of HX. The polynomial neural network (PNN) was applied to optimize HX indicators for maximizing exchange efficiency. Six PNN models were employed to vary the nature of the training data and demand scenarios encountered by the heat exchanger. Model selection index (MSI) determines the models with better accuracy in contrast with other PNN models. It has been found that geometrical indicators: tube area (A), tube layout (Tl), tube pitch (TP) and tube diameter (TD) were found to be most sensitive indicators for enhancing exchange efficiency.

STUDY OF THE TRANSVERSAL DEFORMATION OF CORNEAL STRIP UNDER UNIAXIAL LOADING

Uniaxial test is easy to access and to obtain accuracy data, but it is difficult to acquire two-dimensional deformation information. We investigated the relationship between the two strain components of corneal strip in uniaxial tests, which is the basis for determining of anisotropic strain energy function of cornea via uniaxial tests. Nine rabbits were taken. The left and right corneas were cut along superior-inferior (SI) and nasal-temporal (NT) direction, respectively. For each strip the uniaxial test was carried out, and the tensile displacements, strip images and loads were recorded. Then the stretching strain, the transversal strains and stress were obtained. Optimization based inverse analysis was utilized to find the best among six fitting models that characterizes the relationship between two strain components in uniaxial tests. All models fitted well the experimental data gathered for corneal strips ([Formula: see text]). According to the model selection index, the power model achieved the best performance index: 0.1268 for SI strips and 0.1063 for NT strips versus 0.151 (SI strips) and 0.107 (NT strips) found at most by other models. Thus, it is the most suitable one for describing the relationship between the two strain components of corneal strip during uniaxial stretching.

Developing approaches for linear mixed modeling in landscape genetics through landscape-directed dispersal simulations.

Dispersal can impact population dynamics and geographic variation, and thus, genetic approaches that can establish which landscape factors influence population connectivity have ecological and evolutionary importance. Mixed models that account for the error structure of pairwise datasets are increasingly used to compare models relating genetic differentiation to pairwise measures of landscape resistance. A model selection framework based on information criteria metrics or explained variance may help disentangle the ecological and landscape factors influencing genetic structure, yet there are currently no consensus for the best protocols. Here, we develop landscape‐directed simulations and test a series of replicates that emulate independent empirical datasets of two species with different life history characteristics (greater sage‐grouse; eastern foxsnake). We determined that in our simulated scenarios, AIC and BIC were the best model selection indices and that marginal R 2 values were biased toward more complex models. The model coefficients for landscape variables generally reflected the underlying dispersal model with confidence intervals that did not overlap with zero across the entire model set. When we controlled for geographic distance, variables not in the underlying dispersal models (i.e., nontrue) typically overlapped zero. Our study helps establish methods for using linear mixed models to identify the features underlying patterns of dispersal across a variety of landscapes.

The Efficacy of Common Fit Indices for Enumerating Classes in Growth Mixture Models When Nested Data Structure Is Ignored

Growth mixture model (GMM) is a flexible statistical technique for analyzing longitudinal data when there are unknown heterogeneous subpopulations with different growth trajectories. When individuals are nested within clusters, multilevel growth mixture model (MGMM) should be used to account for the clustering effect. A review of recent literature shows that a higher level of nesting was described in 43% of articles using GMM, none of which used MGMM to account for the clustered data. We conjecture that researchers sometimes ignore the higher level to reduce analytical complexity, but in other situations, ignoring the nesting is unavoidable. This Monte Carlo study investigated whether the correct number of classes can still be retrieved when a higher level of nesting in MGMM is ignored. We investigated six commonly used model selection indices: Akaike information criterion (AIC), consistent AIC (CAIC), Bayesian information criterion (BIC), sample size–adjusted BIC (SABIC), Vuong–Lo–Mendell–Rubin likelihood ratio test (VLMR), and adjusted Lo–Mendell–Rubin likelihood ratio test (ALMR). Results showed that accuracy of class enumeration decreased for all six indices when the higher level is ignored. BIC, CAIC, and SABIC were the most effective model selection indices under the misspecified model. BIC and CAIC were preferable when sample size was large and/or intraclass correlation (ICC) was small, whereas SABIC performed better when sample size was small and/or ICC was large. In addition, SABIC and VLMR/ALMR tended to overextract the number of classes when there are more than two subpopulations and the sample size is large.

Effects of parceling on model selection: Parcel-allocation variability in model ranking.

Research interest often lies in comparing structural model specifications implying different relationships among latent factors. In this context parceling is commonly accepted, assuming the item-level measurement structure is well known and, conservatively, assuming items are unidimensional in the population. Under these assumptions, researchers compare competing structural models, each specified using the same parcel-level measurement model. However, little is known about consequences of parceling for model selection in this context-including whether and when model ranking could vary across alternative item-to-parcel allocations within-sample. This article first provides a theoretical framework that predicts the occurrence of parcel-allocation variability (PAV) in model selection index values and its consequences for PAV in ranking of competing structural models. These predictions are then investigated via simulation. We show that conditions known to manifest PAV in absolute fit of a single model may or may not manifest PAV in model ranking. Thus, one cannot assume that low PAV in absolute fit implies a lack of PAV in ranking, and vice versa. PAV in ranking is shown to occur under a variety of conditions, including large samples. To provide an empirically supported strategy for selecting a model when PAV in ranking exists, we draw on relationships between structural model rankings in parcel- versus item-solutions. This strategy employs the across-allocation modal ranking. We developed software tools for implementing this strategy in practice, and illustrate them with an example. Even if a researcher has substantive reason to prefer one particular allocation, investigating PAV in ranking within-sample still provides an informative sensitivity analysis. (PsycINFO Database Record

Quality of fit measurement in regression quantiles: An elemental set method approach

Little attention has been paid to assess the quality of fit in the quantile regression framework (Noh et al., 2013). As a contribution, I propose a coefficient of determination measure and model selection indices based on the elemental set method.

The flexibility of models of recognition memory: The case of confidence ratings

The normalized maximum likelihood (NML) index is a model-selection index derived from the minimum-description length principle. In contrast to traditional model-selection indices, it also quantifies differences in flexibility between models related to their functional form. We present a new method for computing the NML index for models of categorical data that parameterize multinomial or product-multinomial distributions and apply it to comparing the flexibility of major models of recognition memory for confidence-rating based receiver-operating-characteristic (ROC) data. NML penalties are tabulated for datasets of typical sizes and interpolation functions are fitted that allow one to interpolate NML penalties for datasets with sizes between the tabulated ones. Recovery studies suggest that the NML index performs better than traditional model-selection indices in model selection from ROC data. In an NML-based meta-analysis of 850 ROC datasets, versions of the dual-process signal detection models received most support followed by the finite mixture signal detection model and constrained versions of two-high threshold models.

Model Selection Information Criteria in Latent Class Models with Missing Data and Contingency Question

Latent class analysis (LCA) has been found to have important applications in social and behavioral sciences for modeling categorical response variables, and nonresponse is typical when collecting data. In this study, the nonresponse mainly included “contingency questions” and real “missing data.” The primary objective of this research was to evaluate the effects of some potential factors on model selection indices in LCA with nonresponse data.We simulated missing data with contingency questions and evaluated the accuracy rates of eight information criteria for selecting the correct models. The results showed that the main factors are latent class proportions, conditional probabilities, sample size, the number of items, the missing data rate, and the contingency data rate. Interactions of the conditional probabilities with class proportions, sample size, and the number of items are also significant. From our simulation results, the impact of missing data and contingency questions can be amended by increasing the sample size or the number of items.