Posterior Predictive Model Checking Research Articles

Random-effects substitution models for phylogenetics via scalable gradient approximations.

Phylogenetic and discrete-trait evolutionary inference depend heavily on an appropriate characterization of the underlying character substitution process. In this paper, we present random-effects substitution models that extend common continuous-time Markov chain models into a richer class of processes capable of capturing a wider variety of substitution dynamics. As these random-effects substitution models often require many more parameters than their usual counterparts, inference can be both statistically and computationally challenging. Thus, we also propose an efficient approach to compute an approximation to the gradient of the data likelihood with respect to all unknown substitution model parameters. We demonstrate that this approximate gradient enables scaling of sampling-based inference, namely Bayesian inference via Hamiltonian Monte Carlo, under random-effects substitution models across large trees and state-spaces. Applied to a dataset of 583 SARS-CoV-2 sequences, an HKY model with random-effects shows strong signals of nonreversibility in the substitution process, and posterior predictive model checks clearly show that it is a more adequate model than a reversible model. When analyzing the pattern of phylogeographic spread of 1441 influenza A virus (H3N2) sequences between 14 regions, a random-effects phylogeographic substitution model infers that air travel volume adequately predicts almost all dispersal rates. A random-effects state-dependent substitution model reveals no evidence for an effect of arboreality on the swimming mode in the tree frog subfamily Hylinae. Simulations reveal that random-effects substitution models can accommodate both negligible and radical departures from the underlying base substitution model. We show that our gradient-based inference approach is over an order of magnitude more time efficient than conventional approaches.

Assessment of fit of the time-varying dynamic partial credit model using the posterior predictive model checking method.

Several new models based on item response theory have recently been suggested to analyse intensive longitudinal data. One of these new models is the time-varying dynamic partial credit model (TV-DPCM; Castro-Alvarez etal., Multivariate Behavioral Research, 2023, 1), which is a combination of the partial credit model and the time-varying autoregressive model. The model allows the study of the psychometric properties of the items and the modelling of nonlinear trends at the latent state level. However, there is a severe lack of tools to assess the fit of the TV-DPCM. In this paper, we propose and develop several test statistics and discrepancy measures based on the posterior predictive model checking (PPMC) method (PPMC; Rubin, The Annals of Statistics, 1984, 12, 1151) to assess the fit of the TV-DPCM. Simulated and empirical data are used to study the performance of and illustrate the effectiveness of the PPMC method.

Ant backbone phylogeny resolved by modelling compositional heterogeneity among sites in genomic data

Ants are the most ubiquitous and ecologically dominant arthropods on Earth, and understanding their phylogeny is crucial for deciphering their character evolution, species diversification, and biogeography. Although recent genomic data have shown promise in clarifying intrafamilial relationships across the tree of ants, inconsistencies between molecular datasets have also emerged. Here I re-examine the most comprehensive published Sanger-sequencing and genome-scale datasets of ants using model comparison methods that model among-site compositional heterogeneity to understand the sources of conflict in phylogenetic studies. My results under the best-fitting model, selected on the basis of Bayesian cross-validation and posterior predictive model checking, identify contentious nodes in ant phylogeny whose resolution is modelling-dependent. I show that the Bayesian infinite mixture CAT model outperforms empirical finite mixture models (C20, C40 and C60) and that, under the best-fitting CAT-GTR + G4 model, the enigmatic Martialis heureka is sister to all ants except Leptanillinae, rejecting the more popular hypothesis supported under worse-fitting models, that place it as sister to Leptanillinae. These analyses resolve a lasting controversy in ant phylogeny and highlight the significance of model comparison and adequate modelling of among-site compositional heterogeneity in reconstructing the deep phylogeny of insects.

316 Awardee Talk: Development and Assessment of Predictive Models for Improved Swine Farming

Abstract Prediction of outcomes is critical in swine breeding and management. This necessitates the development of predictive models that address challenges in swine farming. For predictive modeling, there have been significant advances in machine learning. Nevertheless, there are needs to adapt predictive models for specific swine farming problems including genomic prediction and behavior analysis. Furthermore, there is not yet a clear guideline on how to validate novel models in these fields. The overarching goal of this work was to validate a collection of predictive models for improved swine farming with applications to precision management, phenotyping, and breeding. The first study addressed the pig genomic prediction problem. Differential evolution algorithm was utilized to optimize hyperparameters of deep learning (DL) models, which affected the predictive performance of DL. Performance of optimized DL was compared with “best practice” DL architectures selected from literature and baseline DL models with randomly specified hyperparameters. Optimized models showed clear improvement. Despite the success of genomic prediction, phenotyping has become a bottleneck in breeding programs as it is still time-consuming and labor-intensive. The second study aimed at collecting a video dataset to study agonistic behavior of pigs and adapting a state-of-the-art DL pipeline to classify agonistic behaviors of pigs through video analysis. The pipeline was validated through various training-validation data partitions, where the training data were used for model development and the validation data were used for model evaluation. Results showed that splitting the training and validation sets at random led to over-optimistic estimates of model performance. Social behavior continues to be an important topic in animal behavior studies. However, less attention was paid to the modeling of animal social behavior. The last study focused on developing and validating a statistical model for the analysis of social interactions of pigs. Generalized linear mixed models were fitted, and a Bayesian framework was used for parameter estimation and posterior predictive model checking. The predictive performance of the models varied depending on the validation strategy, where three strategies were defined: random cross-validation, block-by-social-group cross-validation, and block-by-focal-animals validation. Collectively, the research items presented how state-of-the-art predictive models can be adapted for and validated in swine farming applications. The findings are also informative to livestock farming in general, as genomic prediction and animal social behavior (of group-housed animals) are popular topics in livestock species.

Analysis of social interactions in group-housed animals using dyadic linear models

Understanding factors affecting social interactions among animals is important for applied animal behavior research. Thus, there is a need to elicit statistical models to analyze data collected from pairwise behavioral interactions. In this study, we propose treating social interaction data as dyadic observations and propose a statistical model for their analysis. We performed posterior predictive checks of the model through different validation strategies: stratified 5-fold random cross-validation, block-by-social-group cross-validation, and block-by-focal-animals validation. The proposed model was applied to a pig behavior dataset collected from 797 growing pigs freshly remixed into 59 social groups that resulted in 10,032 records of directional dyadic interactions. The response variable was the duration in seconds that each animal spent delivering attacks on another group mate. Generalized linear mixed models were fitted. Fixed effects included sex, individual weight, prior nursery mate experience, and prior littermate experience of the two pigs in the dyad. Random effects included aggressor, recipient, dyad, and social group. A Bayesian framework was utilized for parameter estimation and posterior predictive model checking. Prior nursery mate experience was the only significant fixed effect. In addition, a weak but significant correlation between the random aggressor effect and the random recipient effect was obtained when analyzing the attacking duration. The predictive performance of the model varied depending on the validation strategy, with substantially lower performance from the block-by-social-group strategy than other validation strategies. Collectively, this paper demonstrates a statistical model to analyze interactive animal behaviors, particularly dyadic interactions.

Bridging Models of Biometric and Psychometric Assessment: A Three-Way Joint Modeling Approach of Item Responses, Response Times, and Gaze Fixation Counts.

Recently, joint models of item response data and response times have been proposed to better assess and understand test takers' learning processes. This article demonstrates how biometric information such as gaze fixation counts obtained from an eye-tracking machine can be integrated into the measurement model. The proposed joint modeling framework accommodates the relations among a test taker's latent ability, working speed and test engagement level via a person-side variance-covariance structure, while simultaneously permitting the modeling of item difficulty, time-intensity, and the engagement intensity through an item-side variance-covariance structure. A Bayesian estimation scheme is used to fit the proposed model to data. Posterior predictive model checking based on three discrepancy measures corresponding to various model components are introduced to assess model-data fit. Findings from a Monte Carlo simulation and results from analyzing experimental data demonstrate the utility of the model.

Detecting Differential Item Functioning Using Posterior Predictive Model Checking: A Comparison of Discrepancy Statistics

AbstractThis study proposes a new Bayesian differential item functioning (DIF) detection method using posterior predictive model checking (PPMC). Item fit measures including infit, outfit, observed score distribution (OSD), and Q1 were considered as discrepancy statistics for the PPMC DIF methods. The performance of the PPMC DIF method was evaluated via a Monte Carlo simulation manipulating sample size, DIF size, DIF type, DIF percentage, and subpopulation trait distribution. Parametric DIF methods, such as Lord's chi‐square and Raju's area approaches, were also included in the simulation design in order to compare the performance of the proposed PPMC DIF methods to those previously existing. Based on Type I error and power analysis, we found that PPMC DIF methods showed better‐controlled Type I error rates than the existing methods and comparable power to detect uniform DIF. The implications and recommendations for applied researchers are discussed.

Shaky Student Growth? A Comparison of Robust Bayesian Learning Progress Estimation Methods.

Monitoring the progress of student learning is an important part of teachers’ data-based decision making. One such tool that can equip teachers with information about students’ learning progress throughout the school year and thus facilitate monitoring and instructional decision making is learning progress assessments. In practical contexts and research, estimating learning progress has relied on approaches that seek to estimate progress either for each student separately or within overarching model frameworks, such as latent growth modeling. Two recently emerging lines of research for separately estimating student growth have examined robust estimation (to account for outliers) and Bayesian approaches (as opposed to commonly used frequentist methods). The aim of this work was to combine these approaches (i.e., robust Bayesian estimation) and extend these lines of research to the framework of linear latent growth models. In a sample of N = 4970 second-grade students who worked on the quop-L2 test battery (to assess reading comprehension) at eight measurement points, we compared three Bayesian linear latent growth models: (a) a Gaussian model, (b) a model based on Student’s t-distribution (i.e., a robust model), and (c) an asymmetric Laplace model (i.e., Bayesian quantile regression and an alternative robust model). Based on leave-one-out cross-validation and posterior predictive model checking, we found that both robust models outperformed the Gaussian model, and both robust models performed comparably well. While the Student’s t model performed statistically slightly better (yet not substantially so), the asymmetric Laplace model yielded somewhat more realistic posterior predictive samples and a higher degree of measurement precision (i.e., for those estimates that were either associated with the lowest or highest degree of measurement precision). The findings are discussed for the context of learning progress assessment.

A Model Comparison Approach to Posterior Predictive Model Checks in Bayesian Confirmatory Factor Analysis

ABSTRACT Posterior Predictive Model Checking (PPMC) is frequently used for model fit evaluation in Bayesian Confirmatory Factor Analysis (BCFA). In standard PPMC procedures, model misfit is quantified by comparing the location of an ML-based point estimate to the predictive distribution of a statistic. When the point estimate is far from the center posterior predictive distribution, model fit is poor. Not included in this approach, however, is the variability of the Maximum Likelihood (ML)-based point estimates. We propose a new method of PPMC based on comparing posterior predictive distributions of a hypothesized and saturated BCFA model. The method uses the predictive distribution of the saturated model as a reference and the Kolmogorov-Smirnov (KS) statistic to quantify the local misfit of hypothesized models. The results of the simulation study suggest that the saturated model PPMC approach was an accurate method of determining local model misfit and could be used for model comparison. A real data example is also provided in this study.

Assessing Dimensionality of the Ideal Point Item Response Theory Model Using Posterior Predictive Model Checking

Although the use of ideal point item response theory (IRT) models for organizational research has increased over the last decade, the assessment of construct dimensionality of ideal point scales has been overlooked in previous research. In this study, we developed and evaluated dimensionality assessment methods for an ideal point IRT model under the Bayesian framework. We applied the posterior predictive model checking (PPMC) approach to the most widely used ideal point IRT model, the generalized graded unfolding model (GGUM). We conducted a Monte Carlo simulation to compare the performance of item pair discrepancy statistics and to evaluate the Type I error and power rates of the methods. The simulation results indicated that the Bayesian dimensionality detection method controlled Type I errors reasonably well across the conditions. In addition, the proposed method showed better performance than existing methods, yielding acceptable power when 20% of the items were generated from the secondary dimension. Organizational implications and limitations of the study are further discussed.

Bayesian inversion for a fractional Lotka-Volterra model: An application of Canadian lynx vs. snowshoe hares

The Bayesian statistical inversion method is implemented to obtain the parameter estimations for fractional LotkaVolterra models, including the derivative orders, based on analytical solutions obtained by the multi-step homotopy method. For the posterior distributions of the parameter of interest, we used Markov Chain Monte Carlo method through the JAGS package within R software. The posterior predictive model–checking method is implemented to select the best model for a real data set.

The Hellinger Distance within Posterior Predictive Assessment for Investigating Multidimensionality in IRT Models

Under the Bayesian approach, posterior predictive model checking (PPMC) has become a popular tool for fit assessment of item response theory (IRT) models. In this study, we propose the use of the Hellinger distance within PPMC to quantify the distance between the realized and the predictive distribution of the model-based covariance for item pairs. Specifically, the case of multidimensional data analyzed with a unidimensional approach is taken into account. The results of the simulation study show the effectiveness of the method in detecting model misfit and the sensitivity to the trait correlations. An application to real data on tourism perceptions shows the feasibility of the method in practice and especially the capability of detecting potential misfit attributed to specific items.

Identifying model violations under the multispecies coalescent model using P2C2M.SNAPP.

Phylogenetic estimation under the multispecies coalescent model (MSCM) assumes all incongruence among loci is caused by incomplete lineage sorting. Therefore, applying the MSCM to datasets that contain incongruence that is caused by other processes, such as gene flow, can lead to biased phylogeny estimates. To identify possible bias when using the MSCM, we present P2C2M.SNAPP. P2C2M.SNAPP is an R package that identifies model violations using posterior predictive simulation. P2C2M.SNAPP uses the posterior distribution of species trees output by the software package SNAPP to simulate posterior predictive datasets under the MSCM, and then uses summary statistics to compare either the empirical data or the posterior distribution to the posterior predictive distribution to identify model violations. In simulation testing, P2C2M.SNAPP correctly classified up to 83% of datasets (depending on the summary statistic used) as to whether or not they violated the MSCM model. P2C2M.SNAPP represents a user-friendly way for researchers to perform posterior predictive model checks when using the popular SNAPP phylogenetic estimation program. It is freely available as an R package, along with additional program details and tutorials.

An exploratory study on the usability of a multi-CDM through the item-level posterior predictive model checking

An exploratory study on the usability of a multi-CDM through the item-level posterior predictive model checking

A Bayesian Saturated Model Approach to Posterior Predictive Model Checks in Confirmatory Factor Analysis

Data with different sample sizes (N = 50, 500, 2000) were simulated based on two correct CFA models and estimated with four types of mis-specified models (overspecified, underspecified, a model wit...

Multivariate hierarchical frameworks for modeling delayed reporting in count data.

In many fields and applications, count data can be subject to delayed reporting. This is where the total count, such as the number of disease cases contracted in a given week, may not be immediately available, instead arriving in parts over time. For short‐term decision making, the statistical challenge lies in predicting the total count based on any observed partial counts, along with a robust quantification of uncertainty. We discuss previous approaches to modeling delayed reporting and present a multivariate hierarchical framework where the count generating process and delay mechanism are modeled simultaneously in a flexible way. This framework can also be easily adapted to allow for the presence of underreporting in the final observed count. To illustrate our approach and to compare it with existing frameworks, we present a case study of reported dengue fever cases in Rio de Janeiro. Based on both within‐sample and out‐of‐sample posterior predictive model checking and arguments of interpretability, adaptability, and computational efficiency, we discuss the relative merits of different approaches.

Digital Module 06: Bayesian Psychometrics—Posterior Predictive Model Checking https://ncme.elevate.commpartners.com

AbstractDrawing valid inferences from modern measurement models is contingent upon a good fit of the data to the model. Violations of model‐data fit have numerous consequences, limiting the usefulness and applicability of the model. As Bayesian estimation is becoming more common, understanding the Bayesian approaches for evaluating model‐data fit models is critical. In this instructional module, Allison Ames and Aaron Myers provide an overview of Posterior Predictive Model Checking (PPMC), the most common Bayesian model‐data fit approach. Specifically, they review the conceptual foundation of Bayesian inference as well as PPMC and walk through the computational steps of PPMC using real‐life data examples from simple linear regression and item response theory analysis. They provide guidance for how to interpret PPMC results and discuss how to implement PPMC for other model(s) and data. The digital module contains sample data, SAS code, diagnostic quiz questions, data‐based activities, curated resources, and a glossary.

A Hierarchical Framework for Correcting Under-Reporting in Count Data

Tuberculosis poses a global health risk and Brazil is among the top 20 countries by absolute mortality. However, this epidemiological burden is masked by under-reporting, which impairs planning for effective intervention. We present a comprehensive investigation and application of a Bayesian hierarchical approach to modeling and correcting under-reporting in tuberculosis counts, a general problem arising in observational count data. The framework is applicable to fully under-reported data, relying only on an informative prior distribution for the mean reporting rate to supplement the partial information in the data. Covariates are used to inform both the true count-generating process and the under-reporting mechanism, while also allowing for complex spatio-temporal structures. We present several sensitivity analyses based on simulation experiments to aid the elicitation of the prior distribution for the mean reporting rate and decisions relating to the inclusion of covariates. Both prior and posterior predictive model checking are presented, as well as a critical evaluation of the approach. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

Extreme Response Style: A Simulation Study Comparison of Three Multidimensional Item Response Models.

Several multidimensional item response models have been proposed for survey responses affected by response styles. Through simulation, this study compares three models designed to account for extreme response tendencies: the IRTree Model, the multidimensional nominal response model, and the modified generalized partial credit model. The modified generalized partial credit model results in the lowest item mean squared error (MSE) across simulation conditions of sample size (500, 1,000), survey length (10, 20), and number of response options (4, 6). The multidimensional nominal response model is equally suitable for surveys measuring one substantive trait using responses to 10 four-option, forced-choice Likert-type items. Based on data validation, comparison of item MSE, and posterior predictive model checking, the IRTree Model is hypothesized to account for additional sources of construct-irrelevant variance.

A Posterior Predictive Model Checking Method Assuming Posterior Normality for Item Response Theory.

This study investigated the violation of local independence assumptions within unidimensional item response theory (IRT) models. Bayesian posterior predictive model checking (PPMC) methods are increasingly being used to investigate multidimensionality in IRT models. The current work proposes a PPMC method for evaluating local dependence in IRT models that are estimated using full-information maximum likelihood. The proposed approach, which was termed as "PPMC assuming posterior normality" (PPMC-N), provides a straightforward method to account for parameter uncertainty in model fit assessment. A simulation study demonstrated the comparability of the PPMC-N and the Bayesian PPMC approach in the detection of local dependence in dichotomous IRT models.