Multilevel Confirmatory Factor Analysis Research Articles

The mediating role of science intrinsic motivation: effects of homework on science achievement in Western and Asian regions

ABSTRACT This study investigated cross-cultural differences in the relationships between science intrinsic motivation, teacher-perceived and student-perceived homework practices, and science achievement in Asian (Chinese Taipei, Japan, South Korean, and Singapore) and Western (Australia, US, England, and Ireland) top-performing regions according to ranking of science achievement in TIMSS. Multilevel confirmatory factor analysis and multilevel structural equation modelling were conducted using data from 45,448 eighth graders in the 2019 TIMSS study. In Western regions, the teacher control of homework was significantly negatively associated with whole-class science achievement and positively associated with whole-class intrinsic motivation. A higher student-perceived frequency of homework is negatively related to their individual intrinsic motivation and science achievement. However, in Asian regions, teacher-perceived frequency and time spent on homework positively predicted the whole-class intrinsic motivation and science achievement, while in contrast, a higher student-perceived frequency of homework was positively associated with their individual intrinsic motivation and a longer time spent on homework was negatively associated with their intrinsic motivation. Intrinsic motivation in science plays a significant mediating role between students’ and teachers’ homework practices and science achievement at individual and whole-class levels in Asian top performers. There, we discuss at the end that the cultural contexts might influence how teachers and students can use homework best to promote science intrinsic motivation and hence their science achievement.

Psychological safety in interdisciplinary virtual student project teams: A validation study

Psychological safety is an important variable for understanding teams, as it is positively associated with team performance, team learning, and creativity. A large body of research has been conducted on psychological safety in teams that work face-to-face. The aim of this study was to investigate the construct validity of Edmondson's Team Psychological Safety Scale in the context of virtual student teams working on a joint project. The sample used in this study consisted of 344 students who took part in an interdisciplinary project-based master's course at a Norwegian university. The data were analyzed with single-level confirmatory factor analysis (CFA) and multilevel CFA (MCFA). Reliability was investigated with intra-class correlation coefficients, in addition to Cronbach's alpha and McDonald's omega. Overall, the reliability estimates and goodness-of-fit estimates from the CFAs demonstrated an adequately good level of fit for the one-factor structure of psychological safety in the context of virtual student project teams. The Team Psychological Safety Scale was shown to have acceptable levels of construct validity for the virtual student project teams.

Multilevel Factor Mixture Modeling: A Tutorial for Multilevel Constructs

Multilevel factor mixture modeling (FMM) is a hybrid of multilevel confirmatory factor analysis (CFA) and multilevel latent class analysis (LCA). It allows researchers to examine population heterogeneity at the within level, between level, or both levels. This tutorial focuses on explicating the model specification of multilevel FMM that considers the conceptualization of multilevel constructs. Empirical data sets are used to demonstrate the applications of multilevel FMM for within-level constructs, between-level constructs, and within- and between-level constructs. Detailed model specifications of integrating latent classes into multilevel constructs are provided. For modeling the heterogeneity at the between level, parametric and nonparametric approaches are compared both conceptually and substantively using demonstration data. The interpretations of results using multilevel FMM are also provided. The tutorial is concluded with a discussion of some important aspects of applying multilevel FMM.

다층 구조방정식 모형 평가를 위한 수준 특수적 평가 방법의 적용

While a level-specific approach has been proposed for the valid evaluation of multilevel structural
 equation modeling (MSEM), the traditional simultaneous approach remains predominant among
 researchers. In fact, research on level-specific approaches is limited, and some studies have focused on
 multilevel confirmatory factor analysis models. This study conducted a Monte Carlo study to explore
 the performance of a LS approach for MSEM with a direct path between variables under five design
 factors: intraclass correlation (ICC), the number of groups (NG), group size (GS), the size of path
 coefficient (PCS), and misspecification type (MT). The results showed that the LS approach performed
 better than the SI approach even in MSEM at between-group level. This study identified three key
 findings. First, the performance of RMSEA was found to improve with an increase of ICC or GS.
 Second, the performance of LS fit indices (RMSEA, CFI, and TLI) was more promising for detecting
 the misspecified between-level model with a decrease of PCS. Third, the effect of PCS on CFI or TLI
 was strong when ICC was small. The study concludes with a summary and implications of the results.

Applying the Robust Chi-Square Goodness-of-Fit Test to Multilevel Multitrait-Multimethod Models: A Monte Carlo Simulation Study on Statistical Performance

As the robust maximum likelihood χ2 goodness-of-fit test had been found to yield inflated type-I error rates for certain two-level confirmatory factor analysis (CFA) models, a new correction for the test was implemented in Mplus version 8.7. In this simulation study, we inspected whether the corrected test statistics follow the expected χ2 distributions when applying more complex two-level models for multitrait-multimethod data with varying sample sizes and correlations within trait factors. Investigating rejection rates and probability-probability plots, we found that the new correction markedly and sufficiently reduced previously inflated rejection rates in conditions with within-trait correlations equal to 1, 100 between-level units, and 10 or 20 within-level units. In other conditions, rejection rates were hardly affected or not sufficiently reduced by the new correction. While in most conditions, 2 within-level units did not suffice, 5 within-level units and 250 between-level units were enough to yield correct rejection rates given within-trait correlations did not exceed 0.80. Correlations above 0.80 required larger sample sizes. In planning studies with multilevel CFA models, researchers should be aware that sample size requirements for likelihood-based model fit evaluations can depend on several different factors and might consider conducting Monte Carlo simulations tailored to their specific modeling conditions.

Evaluating the Classroom Environment: Multilevel Validation and Measurement Invariance of Classroom Behavioral Climate

The present study contributes to the development of effective measures to evaluate classroom climate, especially in elementary education where these remain limited. In addition, it addresses a typical “flaw” of several studies by approaching classroom climate as a group-level construct rather than an individual characteristic. Following a multilevel approach, we attempted to establish validity evidence for an adapted version of the Classroom Behavioral Climate (CBC) questionnaire and examine its equivalence across two contexts. The participants included 2963 students from 176 fourth and fifth grade classes. Multilevel Confirmatory Factor Analyses resulted in a shorter version of the CBC with three correlated factors in both countries. Two-level models performed better than the single-level models. Configural, metric, and partial scalar invariance was also achieved. The study highlights the need for researchers and practitioners to use appropriate methods and instruments to evaluate the classroom environment based on their research aims and intended use.

Measuring Student Engagement in Lessons Using an Experience Sampling Methodology: The Development and Validation of the Dynamic Engagement With Learning Questionnaire

Student engagement is a central construct in education research, as it is a strong predictor and/or mediator of both cognitive and non-cognitive student outcomes. However, there remain questions among researchers about how to properly measure student engagement during lessons. Our study contributes to this discussion by presenting the development and validation of a theoretically grounded questionnaire of students’ lesson engagement, which is based on an experience sampling methodology. To validate this questionnaire, we issued it to 410 fifth- and sixth-grade students (10–12 years old). The results of both an exploratory and multilevel confirmatory factor analysis approved construct validity of the measure. Adequate within-person variability across time points was also found, which makes the questionnaire suitable for capturing fluctuations over time. Furthermore, partial scalar measurement invariance for questionnaire administration across time points and between fifth- and sixth-grade students was found.

On the importance of being clear about the level of analysis of interest: An illustration using the case of self-compassion.

Theories about within-person (WP) variation are often tested using between-person (BP) research, despite the well-established fact that results may not generalize across levels of analysis. One possible explanation is vague theories that do not specify which level of analysis is of interest. We illustrate such a case using the construct of self-compassion. The factor structure at the BP level has been highly debated, although the theory is actually concerned with relationships at the WP level. Multilevel confirmatory factor analysis was applied to experience-sampling data of self-compassion (N = 213, with n = 4052 measurement occasions). At both levels of analysis, evidence for a two-factor model was found. However, the factors were moderately related at the WP level (r = 0.37, p < 0.001) but largely independent at the BP level (r = 0.04, p = 0.696). Exploratory analyses revealed considerable heterogeneity in the WP relationship among individuals. We discuss how our results provide new impulses to move the debate around self-compassion forward. Lastly, we outline how the WP level-which is of major interest for self-compassion and other constructs in psychology-can guide the conceptualization and assessment to promote advancements of the theory and resulting applications.

The factor structure of Big Five personality trait measures at the between- and within-person levels

The cross-sectional factor structure of Big Five personality trait measures is a weighted blend of the factor structure at the between-person level (i.e., the level of time-invariant interpersonal personality differences) and the factor structure at the within-person level (i.e., the level of intrapersonal personality fluctuations and changes). Thus, the Big Five might not represent the underlying factors at either of the two levels. To probe this issue, I fit multilevel exploratory and confirmatory factor analysis (EFA and CFA) models to longitudinal data from the Netherlands and Australia ( N = 15,085 and 22,582, respectively). In both data sets, the canonical Big Five emerged at the between- but not at the within-person level. At the within-person level, a two-factor structure appeared to be most suitable. Factor 1 indicated positive attitudes towards the self. Factor 2 indicated neuroticism or negative attitudes towards the self. The two factors might reflect short-term fluctuations in positive and negative affect rather than long-lasting personality changes. The findings are tentative due to methodological limitations. We need more research on the within-person factor structure of personality trait measures to clarify what fluctuations and changes they assess and whether we need to revise the way we measure and analyze trait changes.

Measuring safety perceptions of students with behavioural problems in special education: A validation study of the safe at school questionnaire

AbstractAssessment and monitoring of safety in special education schools is imperative to establish and maintain a safe environment in which students can develop academic and social–emotional skills. The present study describes the development of a student self‐report measure, the Safe at School questionnaire. Factorial validity, reliability and concurrent validity was examined in a construction sample of 280 students (68.6% male, age M = 13.29, SD = 2.52) and a validation sample of 1572 students (77.4% male; age M = 14.33, SD = 2.04), representing 233 classes from 20 schools for special education in the Netherlands. Multilevel confirmatory factor analysis was used to examine the factor structure of the Safe at School questionnaire. A model with three correlating factors at the within‐ and between‐group level (Rules and Social Norms, Perceived Own Safety and Perceived Safety of Other Students) showed a satisfactory fit to the data. Reliabilities of the scales were good. Also, measurement invariance for gender was demonstrated, indicating no differences in latent means between boys and girls. Implications for practice and future research are discussed.

Measuring grateful climate at the sports team

ABSTRACT Developing a measurement of grateful climate is an essential step to examining the function of a grateful climate in a team or group context. Accordingly, the purpose of this study was to develop a valid and reliable measure of the grateful climate of sports teams. We defined the grateful climate of sports teams as a culturally shaped perception that team members collectively exhibit values, beliefs and expected behaviors that fit with the script of gratitude. Exploratory factor analysis and multilevel confirmatory factor analysis were conducted to evaluate the validity and reliability of the Sports Team Grateful Climate Questionnaire. In addition, nomological validity and incremental validity were also examined. The results indicated that the 9-item Sports Team Grateful Climate Questionnaire has good convergent validity, nomological validity, and incremental validity. We concluded that it can be a useful tool for future studies aiming to better understand grateful climate in sports teams.

Collective School Leadership and Collective Teacher Efficacy through Turbulence

ABSTRACT Based on data collected in South Carolina from 2020–2022 through the Collective Leadership Initiative, we examined the collective leadership of six schools through cascading crises that resulted in extreme turbulence in many schools. Through an annual collective leadership survey, a collective teacher efficacy survey that we administered three times each year, and focus groups, we found remarkable consistency across all six schools. After conducting a multi-level confirmatory factor analysis and a categorical confirmatory factor analysis to determine that we could report collective leadership and collective teacher efficacy as single school-level factors, we found moderate to strong correlations between collective leadership and collective teacher efficacy. These schools used collective leadership processes that facilitated stability and improvement during a time of unprecedented, cascading crises.

Assessing affect in adolescents with e-diaries: multilevel confirmatory factor analyses of different factor models.

In the last two decades, e-diary studies have gained increasing interest, with a dominant focus on mood and affect. Although requested in current guidelines, psychometric properties are rarely reported, and methodological investigations of factor structure, model fit, and the reliability of mood and affect assessment are limited. We used a seven-day e-diary dataset of 189 adolescent participants (12-17 years). The e-diary affect assessments revealed a considerable portion of within-person variance. The six-factor model showed the best model fit compared to the less complex models. Factor loadings also improved with the complexity of the models. Accordingly, we recommend that future e-diary studies of adolescents use the six-factor model of affect as well as reporting psychometric properties and model fit. For future e-diary scale development, we recommend using a minimum of three items per scale to enable the use of confirmatory multilevel factor analyses.

Teachers’ and principals’ perceptions of school climate: the role of principals’ leadership style in organizational quality

In this paper, we report on teachers’ and principals’ shared perceptions regarding beliefs, rules, trust, and encouragement of new initiatives. Collectively, these are aspects of leadership for learning (LFL) describing an overall shared climate in schools. We demonstrate how these perceptions on school climate differ across teachers and principals within and across countries. Moreover, we report how different perceptions of school climate are associated with leadership style. We analyze data from 37 countries that participated in the last cycle of the Teaching and Learning International Survey (TALIS) in 2018. To build the measurement model, we employ multigroup multilevel confirmatory factor analysis, whereas multivariate linear regression is used to inspect associations. Overall, principals and teachers differ in their views of school climate. In the majority of the countries, principals report stronger school climate than teachers. We further confirm these perceptual differences between teachers and principals by separately studying the relationships between teacher perceived school climate and principal perceived school climate with relevant leadership variables. In the entire sample, we find that principals’ perceptions of school climate are more strongly and consistently associated with leadership in schools. This relationship is particularly stable for distributed leadership. In the entire sample, leadership styles are weakly positively correlated with teacher perceptions of school climate too; however, this association is less pronounced and less stable within individual countries. The analyses conducted within countries revealed that the distributed leadership rather than instructional leadership shapes teachers’ perceptions of school climate. More discussion is presented on the need for alignment between different perceptions of school climate and leadership styles in the overall organizational quality.

5613177 AN INTERNATIONAL, LONGITUDINAL, OBSERVATIONAL STUDY TO EVALUATE THE PSYCHOMETRIC PROPERTIES OF A DAILY DIARY FOR ADOLESCENTS AND ADULTS WITH SICKLE CELL DISEASE

Background: Vaso-occlusive crises (VOCs) are a serious complication of sickle cell disease (SCD). The Sickle Cell Pain Diary – Self Report (SCPD-S) was developed to capture the impact of VOCs, which occur sporadically and unpredictably. The daily diary evaluates the severity of SCD-related pain during and outside of a VOC and the impact of pain on patient-relevant concepts. Aims: A longitudinal observational study was conducted to evaluate the psychometric properties of the SCPD-S items that assess VOC impact. Methods: Participants were eligible for the study if they were ≥12 years old, had a verified diagnosis of SCD, experienced ≥1 VOC in the past 12 months, and lived in the United States, United Kingdom, or Kingdom of Saudi Arabia. Participants completed the SCPD-S daily for at least 3 months through a mobile application and at pre-specified intervals, completed additional surveys which served as criterion measures. Factor analyses were conducted to examine the conceptual framework of the SCPD-S VOC impact items. Psychometric properties of 3 different VOC impact scores were evaluated. Analyses included tests of internal consistency reliability, test-retest reliability, convergent/divergent validity, known-groups validity, and responsiveness. Results: The analytic sample included 299 participants (87% of enrolled participants) who had ≥1 VOC during the study period and missed ≤5 daily assessments in the month in which the VOC occurred. The median age was 30 (range: 12-67); participants were predominantly female (72%) and Black (83%). A unidimensional multi-level confirmatory factor analysis (CFA) yielded poor model fit, failing to support the hypothesized unidimensional construct of VOC impact (Table 1, Model 1). Multi-level exploratory factor analysis revealed a 2-factor model with sufficient model fit (Model 2), which was also supported by a CFA (Model 3). The 2 factors represent interference with general activities (Scale 1) and difficulty completing specific activities of daily living (Scale 2). Cronbach’s α was 0.87 for Scale 1 and 0.86 for Scale 2, indicating adequate internal consistency reliability. Tests of differential item function show that after controlling for VOC impact, participants’ responses to SCPD-S items do not depend on age or country of residence. Psychometric analyses were conducted on the Daily VOC Impact scores, which measure VOC impact over the past 24 hours. Test-retest reliability of Scale 1 Daily VOC Impact scores was satisfactory; results for Scale 2 were weaker (intraclass correlation coefficient=0.78 for Scale 1 and 0.66 for Scale 2). Scale 1 Daily VOC Impact scores were responsive to change; change scores were correlated with changes in symptom severity (ρ=0.41, P<0.001). Scale 2 was not responsive to change (ρ=0.27, P<0.001). Additional analyses provided support for scores that aggregate across days within a VOC, not for scores that aggregate across a fixed period.Conclusion: Findings support the use of the SCPD-S to characterize the daily impact of VOCs, confirming the need for a daily diary to capture variability in VOC impact across days. Compliance with the diary was high (87% completed ≥25 days in a month). While factor analyses supported a 2-factor model of VOC impact, Scale 1 performed consistently better than Scale 2 across analyses. As such, it is recommended that items comprising Scale 2 be dropped from the diary, leaving a single scale that is a reliable and valid measure of VOC impact. Reference 1. White et al, J Patient Rep Outcomes 2021; 5(1):1-12

Psychometric properties of an innovative smartphone application to investigate the daily impact of hypoglycemia in people with type 1 or type 2 diabetes: The Hypo-METRICS app.

The aim of this study was to determine the acceptability and psychometric properties of the Hypo-METRICS (Hypoglycemia MEasurement, ThResholds and ImpaCtS) application (app): a novel tool designed to assess the direct impact of symptomatic and asymptomatic hypoglycemia on daily functioning in people with insulin-treated diabetes. 100 adults with type 1 diabetes mellitus (T1DM, n = 64) or insulin-treated type 2 diabetes mellitus (T2DM, n = 36) completed three daily 'check-ins' (morning, afternoon and evening) via the Hypo-METRICs app across 10 weeks, to respond to 29 unique questions about their subjective daily functioning. Questions addressed sleep quality, energy level, mood, affect, cognitive functioning, fear of hypoglycemia and hyperglycemia, social functioning, and work/productivity. Completion rates, structural validity, internal consistency, and test-retest reliability were explored. App responses were correlated with validated person-reported outcome measures to investigate convergent (rs>±0.3) and divergent (rs<±0.3) validity. Participants' mean±SD age was 54±16 years, diabetes duration was 23±13 years, and most recent HbA1c was 56.6±9.8 mmol/mol. Participants submitted mean±SD 191±16 out of 210 possible 'check-ins' (91%). Structural validity was confirmed with multi-level confirmatory factor analysis showing good model fit on the adjusted model (Comparative Fit Index >0.95, Root-Mean-Square Error of Approximation <0.06, Standardized Root-Mean-square Residual<0.08). Scales had satisfactory internal consistency (all ω≥0.5), and high test-retest reliability (rs≥0.7). Convergent and divergent validity were demonstrated for most scales. High completion rates and satisfactory psychometric properties demonstrated that the Hypo-METRICS app is acceptable to adults with T1DM and T2DM, and a reliable and valid tool to explore the daily impact of hypoglycemia.

The Importance of Social Support in Recovery Populations: Toward a Multilevel Understanding

ABSTRACT Although social support is commonly investigated in the context of substance use recovery, researchers have widely neglected its multilevel nature, thus limiting what we know about its measurement across levels of observation. The current study used multilevel confirmatory factor analyses (MCFA) on 229 individuals living in 42 recovery homes to investigate the structure of single factor of social support at the individual and house-levels. Multilevel structural equation model (MSEM) was then conducted to examine whether the social support factor was associated with stress at the individual and house-levels. MCFA results showed that within individuals, all social support measures were significant and positive while at the house-level, there were a few discrepancies (e.g. network size was negative). Stress was significantly and negatively related to the social support factor at the individual-level, but this association was positive at the house-level. These findings suggest that on an individual-level, a person’s perception and source of social support is particularly important – even if the source of support comes from someone who is not abstinent. On a house-level, social support is more sensitive to outside influences than within individuals. Implications for future research and substance use interventions targeting social support are discussed.

Creating a brief form of the Supervisory Working Alliance Inventory in the Chinese cultural context using multilevel factor analyses.

The aim of this study was to examine the multilevel factor structure of the Supervisory Working Alliance Inventory (SWAI; Efstation et al., 1990) and shorten it for routine clinical use in the Chinese cultural context. Participants were 203 (Sample 1) and 97 (Sample 2) beginning counseling trainees from a master's level training program in China. They were given the SWAI after every supervision session, and a subset of Sample 1 trainees also completed measures of their counseling self-efficacy before every client session. With Sample 1, multilevel exploratory factory analysis (M-EFA) showed that for the original 19-item version of SWAI, two factors emerged on the within-trainee level (Rapport and Client Focus) as proposed in Efstation et al. (1990), whereas only one General Alliance factor was identifiable on the between-trainee level. After removing poorly or double-loaded items, a six-item brief version of SWAI (SWAI-Brief) was obtained. Using Sample 2, multilevel confirmatory factor analysis and multilevel bifactor models supported the factor structure found in M-EFA with Sample 1. Empirical evidence also supported the reliability of the SWAI-Brief, as well as its criterion-related concurrent and predictive validity given its significant associations with trainee's counseling self-efficacy at the same week and in the subsequent week. Results suggested that the six-item SWAI-Brief could be a reliable and valid measure for assessing supervisory working alliance in the Chinese cultural context and may be used to continuously track clinical supervision processes. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Validating a novel tool for coding body worn camera footage of police‐community member interactions

AbstractObjectivesThis study aimed to validate a tool for coding police body‐worn camera (BWC) footage to measure the dynamics of police–community encounters, including items related to community members, officer performance, and situational outcomes.MethodsOver 1000 BWC videos were scored by five coders who participated in an iterative interrater reliability exercise to improve the agreement. Krippendorf's alphas and a multilevel simultaneous component analysis were used to assess interrater reliability and the component structure of ratings, respectively. Bootstrapped and multilevel exploratory and confirmatory factor analyses were conducted to ascertain any underlying dimensions supporting officer and community member behavior, and cluster analyses were performed to examine whether interactions could be grouped into coherent categories.ResultsThe lower bound of unweighted means of bootstrapped alphas ranged from 0.67 to 0.96 depending upon item type, and 21.60% of the total variance in raters was between‐subjects. Factor analyses revealed only a single factor (“defensiveness”) underlying four items related to hostility directed toward officers by community members. Cluster analyses described seven categories that were highly overlapping but conceptually plausible with cluster silhouette means ranging from −0.06 to 0.13.ConclusionsWe validated a collection of interval‐level coding metrics on a random sample of over 1000 interactions between police officers and community members recorded through BWCs. In doing so, we provide a roadmap for researchers and practitioners to effectively measure the dynamics of police–community encounters and officer performance within those encounters and a framework for addressing the validity and reliability of items in future studies involving the measurement of BWC footage.

A multilevel perspective on the role of job demands, job resources, and need satisfaction for employees' outcomes

This study investigates the mediator role of psychological need satisfaction for the effects of job demands and resources on turnover intentions, psychological distress, and work-to-family conflict, simultaneously at the employee and work unit levels. In doing so, we consider how need satisfaction, when considered at the work unit level, creates a context likely to play an additional role in the prediction of these outcomes. These questions were investigated using a combination of doubly latent multilevel confirmatory factor analyses and structural equation models applied to responses provided by a large sample (N = 5716 employees nested within 50 work units) of Canadian Armed Forces/Department of National Defence personnel. The results supported the idea that work environment effects on the outcomes considered in this study were mediated by psychological need satisfaction at the individual and work unit levels and demonstrated that these associations were driven by global work environment perceptions and global need satisfaction. Furthermore, need satisfaction was found to create a context, at the work unit level, leading employees working in units including more highly satisfied co-workers to present higher levels of turnover intentions but lower levels of work-to-family conflict than would be expected based on their individual levels of need satisfaction.