Multilevel Measurement Model Research Articles

Digital Module 34: Introduction to Multilevel Measurement Modeling

Module AbstractClustered data structures are common in many areas of educational and psychological research (e.g., students clustered in schools, patients clustered by clinician). In the course of conducting research, questions are often administered to obtain scores reflecting latent constructs. Multilevel measurement models (MLMMs) allow for modeling measurement (the relationship of test items to constructs) and the relationships between variables in a clustered data structure. Modeling the two concurrently is important for accurately representing the relationships between items and constructs, and between constructs and other constructs/variables. The barrier to entry with MLMMs can be high, with many equations and less‐documented software functionality. This module reviews two different frameworks for multilevel measurement modeling: (1) multilevel modeling and (2) structural equation modeling. We demonstrate the entire process in R with working code and available data, from preparing the dataset, through writing and running code, to interpreting and comparing output for the two approaches.

Evaluating the relative efficiency among robust estimation methods for multilevel factor analysis with categorical data

Multilevel measurement models are more frequently applied to help answer questions when data arise from hierarchically structured multivariate data. In this simulation study of multilevel factor models, we evaluated the relative efficiency among three estimation methods: robust maximum likelihood, unweighted least squares, and weighted least squares. We found that weighted least squares yielded more or equally efficient parameter estimates under all sample size conditions for all model parameters. The relative efficiency of standard errors was less straightforward where maximum likelihood was more efficient for loadings and residual variances, but weighted least squares was more efficient for the factor covariance matrices. Finally, we give recommendations for estimating multilevel confirmatory factor analysis models and directions for future research.

How Is Knowledge Perceived as Power? A Multilevel Model of Knowledge Power in Innovation Networks.

Inter-organizational power relations have long been considered to be balanced in innovation networks, which are viewed as loosely coupled systems. Some recent studies, however, show that innovation networks are asymmetric and hierarchical, and the power of network actors has become a significant but rarely addressed issue. As knowledge is the most important resource in the network, this paper introduces the concept of knowledge power by combining related research perspectives and conducting some fundamental research on it as follows: (1) knowledge power’s origins are analyzed by proposing the term “activated knowledge” and studying the path through which it is formed over multiple levels of the network; (2) a multilevel framework of characteristics of activated knowledge, which is considered the major determinant of knowledge power, is established, and suggestions are offered for how they impact knowledge power; and (3) a multilevel measurement model for knowledge power is built, and the above propositions are tested by mathematical inference. The purpose of this paper is not only to study knowledge power’s formation, determinants, and measurement but also to offer a comprehensive view, combining multiple network levels and multiple research perspectives, that should be useful to researchers conducting future studies in this field.

Multilevel measurement models for group collective constructs.

Multilevel measurement models for group collective constructs.

Examination of TIMSS Mathematics Data with Multilevel Measurement Models in Respect to Content, Cognitive and Topic Areas

This research study aims to identify TIMMS 8th grade mathematics item groups and the specification of items in which Turkish 8th grade students have signıfıcantly lower level of correct responses compared to all other 8th grade participants. For this purpose, total 260 (82 from 1999, 88 from 2007, and 90 from 2011) items released by International Association for the Evaluation of Educational Achievement (IEA) were grouped according to cognitive, content and sub-content domains. Then, mean correct responses of released items for each participant country were obtained from IEA’s yearly almanac. Finally, data were analyzed by using Multilevel Measurement Models and differences in achievement levels between Turkish 8th graders and their peers from other participating countries were predicted and tested in the context of item groups. Analysis of data showed that performance of Turkish students statistically significantly lower than performance of students from rest of the other participant countries in Number (Content Domain)-Fractions and Decimals (Topic Area)-Knowing (Cognitive Domain) item group. Detailed investigation revealed that students generally fail in procedures in fractions and conversions among fraction, decimal, and percent.

Measuring Collective Efficacy

This article specifies a multilevel measurement model for survey response when data are nested. The model includes a test–retest model of reliability, a confirmatory factor model of interitem reliability with item-specific bias effects, an individual-level model of the biasing effects due to respondent characteristics, and a neighborhood-level model of construct validity. We apply this model for measuring informal social control within collective efficacy theory. Estimating the model on 3,260 respondents nested within 123 Seattle neighborhoods, we find that measures of informal control show reasonable test–retest and interitem reliability. We find support for the hypothesis that respondents’ assessments of whether their neighbors would intervene in specific child deviant acts are related to whether they have observed such acts in the past, which is consistent with a cognitive model of survey response. Finally, we find that, when proper measurement models are not used, the effects of some neighborhood covariates on informal control are biased upward and the effect of informal social control on violence is biased downward.

Single and Multiple Ability Estimation in the SEM Framework: A Noninformative Bayesian Estimation Approach

Latent variable models with many categorical items and multiple latent constructs result in many dimensions of numerical integration, and the traditional frequentist estimation approach, such as maximum likelihood (ML), tends to fail due to model complexity. In such cases, Bayesian estimation with diffuse priors can be used as a viable alternative to ML estimation. This study compares the performance of Bayesian estimation with ML estimation in estimating single or multiple ability factors across 2 types of measurement models in the structural equation modeling framework: a multidimensional item response theory (MIRT) model and a multiple-indicator multiple-cause (MIMIC) model. A Monte Carlo simulation study demonstrates that Bayesian estimation with diffuse priors, under various conditions, produces results quite comparable with ML estimation in the single- and multilevel MIRT and MIMIC models. Additionally, an empirical example utilizing the Multistate Bar Examination is provided to compare the practical utility of the MIRT and MIMIC models. Structural relationships among the ability factors, covariates, and a binary outcome variable are investigated through the single- and multilevel measurement models. The article concludes with a summary of the relative advantages of Bayesian estimation over ML estimation in MIRT and MIMIC models and suggests strategies for implementing these methods.

Assessing Impact, DIF, and DFF in Accommodated Item Scores

This pedagogical article is intended to explain the similarities and differences between the parameterizations of two multilevel measurement model (MMM) frameworks. The conventional two-level MMM that includes item indicators and models item scores (Level 1) clustered within examinees (Level 2) and the two-level cross-classified MMM (in which item scores are cross-classified by two Level 2 classifications including item and examinee) are discussed. A small subset of National Assessment of Educational Progress 4th grade mathematics item scores is used to demonstrate use of the MMMs for assessing facets of the validity of accommodated test scores. The models’ similarities and distinctions are emphasized as well as the flexibility of the models’ extensions.

Distinguishing Differential Testlet Functioning From Differential Bundle Functioning Using the Multilevel Measurement Model

This study extends the multilevel measurement model to handle testlet-based dependencies. A flexible two-level testlet response model (the MMMT-2 model) for dichotomous items is introduced that permits assessment of differential testlet functioning (DTLF). A distinction is made between this study’s conceptualization of DTLF and that of differential bundle functioning (DBF) with an emphasis on the practical utility of decomposing differential functioning into item- and testlet-specific components. A small-scale simulation study is also conducted to assess estimation of the MMMT-2 model and its measures of DIF and DTLF when compared with SIBTEST’s estimation of DIF and DBF. Results support use of the MMMT-2 model for assessing impact, DIF, and DTLF for tests that include testlet-based dichotomous items with equal item discriminations.

Coaching competency and satisfaction with the coach: A multi-level structural equation model

The purpose of this initial predictive validity study was to determine the ability of measures derived from the Athletes' Perceptions of Coaching Competency Scale II – High School Teams (APCCS II-HST) to predict satisfaction with the head coach. Specification of the statistical model was informed by the mediational model of coach–athlete interactions. The technical quality of the satisfaction measure was evaluated before testing the predictive validity of the coaching competency measures. Data were collected from athletes of seven sports. Athlete observations (N = 748) were clustered within teams (G = 74). Multi-group confirmatory factor analyses (CFA) provided evidence for factorial invariance of a reduced version of the satisfaction measure by athlete gender. Multi-level CFA provided evidence of model–data consistency for a reduced version of the satisfaction measure. Multi-level structural equation modelling provided evidence for the ability of latent coaching competency to positively predict latent satisfaction at both the athlete level (technique competency and motivation competency) and the team level (coaching competency) and for close model–data fit. Implications of this study include: that the APCCS II-HST should be viewed as a replacement for the Coaching Competency Scale when the intended population is appropriate; a preliminary multi-level measurement model for satisfaction with one's coach that should be considered as a potential starting point in subsequent studies; and empirical support for a key relationship proposed in the mediational model of coach–athlete interactions.

Confirmatory factor analysis for applied research

With its emphasis on practical and conceptual aspects, rather than mathematics or formulas, this accessible book has established itself as the go-to resource on confirmatory factor analysis (CFA). Detailed, worked-through examples drawn from psychology, management, and sociology studies illustrate the procedures, pitfalls, and extensions of CFA methodology. The text shows how to formulate, program, and interpret CFA models using popular latent variable software packages (LISREL, Mplus, EQS, SAS/CALIS); understand the similarities and differences between CFA and exploratory factor analysis (EFA); and report results from a CFA study. It is filled with useful advice and tables that outline the procedures. The companion website offers data and program syntax files for most of the research examples, as well as links to CFA-related resources. New to This Edition *Updated throughout to incorporate important developments in latent variable modeling. *Chapter on Bayesian CFA and multilevel measurement models. *Addresses new topics (with examples): exploratory structural equation modeling, bifactor analysis, measurement invariance evaluation with categorical indicators, and a new method for scaling latent variables. *Utilizes the latest versions of major latent variable software packages--