Cross-classified Data Structure Research Articles

Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching

The cross-classified data structure is ubiquitous in education, psychology, and health outcome sciences. In these areas, assessment instruments that are made up of multiple items are frequently used to measure latent constructs. The presence of both the cross-classified structure and multivariate categorical outcomes leads to the so-called item-level data with cross-classified structure. An example of such data structure is the routinely collected student evaluation of teaching (SET) data. Motivated by the lack of research on multilevel IRT modeling with crossed random effects and the need of an approach that can properly handle SET data, this study proposed a cross-classified IRT model, which takes into account both the cross-classified data structure and properties of multiple items in an assessment instrument. A new variant of the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm was introduced to address the computational complexities in estimating the proposed model. A preliminary simulation study was conducted to evaluate the performance of the algorithm for fitting the proposed model to data. The results indicated that model parameters were well recovered. The proposed model was also applied to SET data collected at a large public university to answer empirical research questions. Limitations and future research directions were discussed.

Individual Mobility across Clusters: The Impact of Ignoring Cross-Classified Data Structures in Discrete-Time Survival Analysis

A multilevel-discrete time survival model may be appropriate for purely hierarchical data, but when data are non-purely hierarchical due to individual mobility across clusters, a cross-classified discrete time survival model may be necessary. The purpose of this research was to investigate the performance of a cross-classified discrete-time survival model and assess the impact of ignoring a cross-classified data structure on the model parameters of a conventional discrete-time survival model and a multilevel discrete-time survival model. A Monte Carlo simulation was used to examine the performance of three discrete-time survival models when individuals are mobile across clusters. Simulation factors included the value of the between-clusters variance, number of clusters, within-cluster sample size, Weibull scale parameter, and mobility rate. The results suggest that substantial relative parameter bias, unacceptable coverage of the 95% confidence intervals, and severely biased standard errors are possible for all model parameters when a discrete-time survival model is used that ignores the cross-classified data structure. The findings presented in this study are useful for methodologists and practitioners in educational research, public health, and other social sciences where discrete-time survival analysis is a common methodological technique for analyzing event-history data.

Propensity Score Matching with Cross-Classified Data Structures: A Comparison of Methods

In the current study, we compare propensity score (PS) matching methods for data with a cross-classified structure, where each individual is clustered within more than one group, but the groups are not hierarchically organized. Through a Monte Carlo simulation study, we compared sequential cluster matching (SCM), preferential within cluster matching (PWCM), greedy matching (GM), and optimal full matching (OFM), using propensity scores from four different models. The results indicated that the four matching methods performed well when PSs were estimated with logistic regression containing both level-1 and level-2 covariates. When the level-2 covariates were omitted in the logistic regression PS model, matching methods resulted in biased treatment effect estimates. However, omission of level-2 covariates did not result in biased estimates when the PS model was a logistic cross-classified random effects model (CCREM). SCM and PWCM outperformed GM and OFM with a logistic CCREM that included level-1 and level-2 covariates.

Multiple imputation approaches for handling incomplete three-level data with time-varying cluster-memberships.

Three‐level data arising from repeated measures on individuals clustered within higher‐level units are common in medical research. A complexity arises when individuals change clusters over time, resulting in a cross‐classified data structure. Missing values in these studies are commonly handled via multiple imputation (MI). If the three‐level, cross‐classified structure is modeled in the analysis, it also needs to be accommodated in the imputation model to ensure valid results. While incomplete three‐level data can be handled using various approaches within MI, the performance of these in the cross‐classified data setting remains unclear. We conducted simulations under a range of scenarios to compare these approaches in the context of an acute‐effects cross‐classified random effects substantive model, which models the time‐varying cluster membership via simple additive random effects. The simulation study was based on a case study in a longitudinal cohort of students clustered within schools. We evaluated methods that ignore the time‐varying cluster memberships by taking the first or most common cluster for each individual; pragmatic extensions of single‐ and two‐level MI approaches within the joint modeling (JM) and the fully conditional specification (FCS) frameworks, using dummy indicators (DI) and/or imputing repeated measures in wide format to account for the cross‐classified structure; and a three‐level FCS MI approach developed specifically for cross‐classified data. Results indicated that the FCS implementations performed well in terms of bias and precision while JM approaches performed poorly. Under both frameworks approaches using the DI extension should be used with caution in the presence of sparse data.

Cross-classified multilevel models improved standard error estimates of covariates in clinical outcomes – a simulation study

ObjectiveTo compare estimates of effect and variability resulting from standard linear regression analysis and hierarchical multilevel analysis with cross-classified multilevel analysis under various scenarios. Study design and settingWe performed a simulation study based on a data structure from an observational study in clinical mental health care. We used a Markov chain Monte Carlo approach to simulate 18 scenarios, varying sample sizes, cluster sizes, effect sizes and between group variances. For each scenario, we performed standard linear regression, multilevel regression with random intercept on patient level, multilevel regression with random intercept on nursing team level and cross-classified multilevel analysis. ResultsApplying cross-classified multilevel analyses had negligible influence on the effect estimates. However, ignoring cross-classification led to underestimation of the standard errors of the covariates at the two cross-classified levels and to invalidly narrow confidence intervals. This may lead to incorrect statistical inference. Varying sample size, cluster size, effect size and variance had no meaningful influence on these findings. ConclusionIn case of cross-classified data structures, the use of a cross-classified multilevel model helps estimating valid precision of effects, and thereby, support correct inferences.

Comparing cross-classified mixed effects and Bayesian structural equations modeling for stimulus sampling designs: A simulation study

Researchers examining a wide range of psychological phenomena, including interpersonal perception, attitude formation, and stereotype activation, apply stimulus sampling designs (SSD). The standard SSD study requires participant raters to provide evaluations of a series of target stimuli (e.g., photographs, media clips, vignettes), and the constituent responses are simultaneously nested within participants and stimuli, yielding a cross-classified data structure. Prior methodological work has illustrated the application of both cross-classified mixed effects, and cross-classified structural equation modeling to accommodate the corresponding dependency structure. Despite their widespread application, little is known about how sample size for both participants and stimuli is associated with inferential power and coverage in SSDs. Even less is known about the feasibility of random slopes, or whether the frequentist (maximum likelihood) and Bayesian (MCMC) estimators differ in accuracy or efficiency under the design conditions typically observed in SSD studies. We conducted a Monte Carlo simulation study to better understand parameter bias, statistical power, and confidence or credible interval coverage, as a function of the number of participant raters and target stimuli, effect size, as well as the presence of random slopes, and modeling framework. Recommendations for future research are provided.

Assessing Differences between Nested and Cross-Classified Hierarchical Models

In multilevel data, cross-classified data structures are common. For example, this occurs when individuals move to different regions in longitudinal data or students go to different secondary schools than their primary school peers. In both cases, the data structure is no longer fully nested. Estimating cross-classified multilevel models is computationally intensive, so researchers have used several shortcuts to decrease run time. We consider how these shortcuts affect parameter estimates. In particular, we compare parameter estimates from fully nested and cross-classified models using a series of Monte Carlo simulations. When the outcome is continuous, we identify systematic differences in estimated standard errors and some differences in the estimated variance components. When the outcome is binary, we also find differences in the estimated coefficients. Accordingly, we caution researchers to avoid fully nested model specifications when cross-classification exists but suggest some limited conditions under which parameter estimates are unlikely to be different.

Trifactor Models for Multiple-Ratings Data

In this study we extend and assess the trifactor model for multiple-ratings data in which two different raters give independent scores for the same responses (e.g., in the GRE essay or to subset of PISA constructed-responses). The trifactor model was extended to incorporate a cross-classified data structure (e.g., items and raters) instead of a strictly hierarchical structure. we present a set of simulations to reflect the incompleteness and imbalance in real-world assessments. The effects of the rate of missingness in the data and of ignoring differences among raters are investigated using two sets of simulations. The use of the trifactor model is also illustrated with empirical data analysis using a well-known international large-scale assessment.

The Impact of Inappropriate Modeling of Cross-Classified Data Structures on Random-Slope Models

The Impact of Inappropriate Modeling of Cross-Classified Data Structures on Random-Slope Models

The Effect of Earthquake Hazards Induced by Natural Gas Mining on Medically Unexplained Physical Symptoms and Psychosocial Problems: A Longitudinal Analysis

Study/Objective To determine whether the chronic threat of exposure to mining-induced earthquakes in the northern part of the Netherlands, is accompanied by a higher prevalence of medically Unexplained Physical Symptoms (MUPS) and psychosocial problems. Background The Groningen natural gas field is among the largest in the world and uniquely located in a populated area. Decades of mining activities burdened the region with increased earthquake hazards, several times higher than 3.0 on the Richter scale (one 3.6, in 2012). Inhabitants were confronted with ground shaking, damaged houses, loss of livelihood, social unrest, and public anger towards the mining company and authorities. In 2013 future earthquakes with a magnitude over 5.0 were considered likely, contributing to lack of positive prospects and chronic stress: factors that are not beneficial from a public health perspective. Methods Data covering a five-year period (2011-2015), was extracted from electronic patient records of general practitioners. Two dependent variables were constructed: MUPS (physical symptoms clustered per organ system) and psychological and social problems (mood and anxiety disorders, suicide and problems in personal relations or work). Statistical analysis was conducted in steps to test differences in prevalence in health issues in the earthquake region and a comparison group; and to determine the predictive value of gender, age, socioeconomic status, and earthquake magnitude and depth, considering the cross-classified data structure with patients nested in general practices and postal codes. Results The results indicate whether the prevalence of MUPS and psychosocial problems was significantly higher in the earthquake region regardless of gender and age, after controlling for socio-economic status and earthquake magnitude and depth. Conclusion The health impact of chronic stress linked to earthquake-threat could be determined apart from other relevant explanatory factors.

Evaluating the Experiences with an Information and Referral Center (IRC) following a Large Air Crash Disaster

Study/Objective To determine whether the chronic threat of exposure to mining-induced earthquakes in the northern part of the Netherlands, is accompanied by a higher prevalence of medically Unexplained Physical Symptoms (MUPS) and psychosocial problems. Background The Groningen natural gas field is among the largest in the world and uniquely located in a populated area. Decades of mining activities burdened the region with increased earthquake hazards, several times higher than 3.0 on the Richter scale (one 3.6, in 2012). Inhabitants were confronted with ground shaking, damaged houses, loss of livelihood, social unrest, and public anger towards the mining company and authorities. In 2013 future earthquakes with a magnitude over 5.0 were considered likely, contributing to lack of positive prospects and chronic stress: factors that are not beneficial from a public health perspective. Methods Data covering a five-year period (2011-2015), was extracted from electronic patient records of general practitioners. Two dependent variables were constructed: MUPS (physical symptoms clustered per organ system) and psychological and social problems (mood and anxiety disorders, suicide and problems in personal relations or work). Statistical analysis was conducted in steps to test differences in prevalence in health issues in the earthquake region and a comparison group; and to determine the predictive value of gender, age, socioeconomic status, and earthquake magnitude and depth, considering the cross-classified data structure with patients nested in general practices and postal codes. Results The results indicate whether the prevalence of MUPS and psychosocial problems was significantly higher in the earthquake region regardless of gender and age, after controlling for socio-economic status and earthquake magnitude and depth. Conclusion The health impact of chronic stress linked to earthquake-threat could be determined apart from other relevant explanatory factors.

A Cross-Classified CFA-MTMM Model for Structurally Different and Nonindependent Interchangeable Methods

ABSTRACTMultirater (multimethod, multisource) studies are increasingly applied in psychology. Eid and colleagues (2008) proposed a multilevel confirmatory factor model for multitrait-multimethod (MTMM) data combining structurally different and multiple independent interchangeable methods (raters). In many studies, however, different interchangeable raters (e.g., peers, subordinates) are asked to rate different targets (students, supervisors), leading to violations of the independence assumption and to cross-classified data structures. In the present work, we extend the ML-CFA-MTMM model by Eid and colleagues (2008) to cross-classified multirater designs. The new C4 model (Cross-Classified CTC[M-1] Combination of Methods) accounts for nonindependent interchangeable raters and enables researchers to explicitly model the interaction between targets and raters as a latent variable. Using a real data application, it is shown how credibility intervals of model parameters and different variance components can be obtained using Bayesian estimation techniques.

The Effects of Nonindependent Rater Sets in Multilevel–Multitrait–Multimethod Models

Multiple ratings are becoming increasingly popular for the assessment of a wide range of behaviors. Teaching evaluation designs and 360-degree feedback often use multiple raters alongside self-ratings. Eid and colleagues (2008) proposed a multilevel structural equation model for the analysis of data stemming from such designs that assumes that raters stem from independent populations of raters. However, it is quite common for raters to rate multiple targets, thus implying a cross-classified data structure. A simulation study was conducted to assess the effects of this rater nonindependence on parameter and standard error estimates in multilevel structural equation models. Results show parameter estimation biases to be small, whereas standard errors pertaining to the Level 1 covariance matrices as well as the mean structure on Level 2 are underestimated.

Standardized Mean Differences in Two-Level Cross-Classified Random Effects Models

Multilevel modeling techniques are becoming more popular in handling data with multilevel structure in educational and behavioral research. Recently, researchers have paid more attention to cross-classified data structure that naturally arises in educational settings. However, unlike traditional single-level research, methodological studies about multilevel effect size have been rare and those that have recently appeared had an emphasis on strictly hierarchical data structure. This article extends the work on multilevel standardized mean differences from strictly hierarchical structure to both fully and partially cross-classified structures. Analytically derived formulae for calculating effect sizes and the corresponding sampling variances (or standard errors) are presented, verified by simulation results, and illustrated with real data examples. Implications for primary research studies and meta-analyses are discussed.

The impact of neighbourhood deprivation on adolescent violent criminality and substance misuse: A longitudinal, quasi-experimental study of the total Swedish population

A number of studies suggest associations between neighbourhood characteristics and criminality during adolescence and young adulthood. However, the causality of such neighbourhood effects remains uncertain. We followed all children born in Sweden from 1975-1989 who lived in its three largest cities by the age of 15 years and for whom complete information was available about individual and contextual factors (N = 303 465). All biological siblings were identified in the sample (N = 179 099). Generalized linear mixed-effects models were used to assess the effect of neighbourhood deprivation on violent criminality and substance misuse between the ages of 15 and 20 years, while taking into account the cross-classified data structure (i.e. siblings in the same families attending different schools and living in different neighbourhoods at age 15). In the crude model, an increase of 1 SD in neighbourhood deprivation was associated with a 57% increase in the odds of being convicted of a violent crime (95% CI 52%-63%). The effect was greatly attenuated when adjustment was made for a number of observed confounders (OR 1.09, 95% CI 1.06-1.11). When we additionally adjusted for unobserved familial confounders, the effect was no longer present (OR 0.96, 95% CI 0.84-1.10). Similar results were observed for substance misuse. The results were not due to poor variability either between neighbourhoods or within families. We found that the adverse effect of neighbourhood deprivation on adolescent violent criminality and substance misuse in Sweden was not consistent with a causal inference. Instead, our findings highlight the need to control for familial confounding in multilevel studies of criminality and substance misuse.

The Consequences of Ignoring Individuals' Mobility in Multilevel Growth Models

In longitudinal multilevel studies, especially in educational settings, it is fairly common that participants change their group memberships over time (e.g., students switch to different schools). Participant’s mobility changes the multilevel data structure from a purely hierarchical structure with repeated measures nested within individuals and individuals nested within clusters to a cross-classified structure with repeated measures cross-classified by both individuals and clusters. If researchers fail to consider the cross-classified data structure and simply use the hierarchical linear models (HLMs) instead of the more appropriate cross-classified random-effects models (CCREMs) to analyze the data, there will be biases in the estimates of variance components and inaccurate statistical inference regarding the fixed effects. In addition, the impact of such model misspecification depends on factors including the rate of mobility and the pattern of mobility.

The Impact of Inappropriate Modeling of Cross-Classified Data Structures

Cross-classified random effects modeling (CCREM) is used to model multilevel data from nonhierarchical contexts. These models are widely discussed but infrequently used in social science research. Because little research exists assessing when it is necessary to use CCREM, 2 studies were conducted. A real data set with a cross-classified structure was analyzed by comparing parameter estimates when ignoring versus modeling the cross-classified data structure. A follow-up simulation study investigated potential factors affecting the need to use CCREM. Results indicated that when the structure is ignored, fixed-effect estimates were unaffected, but standard error estimates associated with the variables modeled incorrectly were biased. Estimates of the variance components also displayed bias, which was related to several study factors.