Cross-classified Models Research Articles

The role of item format in the PISA 2018 mathematics literacy assessment: A cross-country study

When construct-irrelevant sources affect item difficulty, validity of the assessment is compromised. Using responses of 260000 students from 71 countries to the Programme for International Student Assessment (PISA) 2018 mathematics assessment and cross-classified mixed effects models, we examined three validity concerns associated with the construct-irrelevant factor, item format: whether the format influenced item difficulty, whether item format’s impact on difficulty varied across countries, undermining PISA’s foundational goal of meaningful country comparisons, and whether item format effects differed between genders, affecting assessment fairness. Item format contributed to a substantial average of 12 % of variance in item difficulties. The effect of item format was non-uniform across countries, with 30 % of the variance in item difficulties being due to format in lower-performing countries, and 10 % in higher-performing countries, challenging the comparability of educational outcomes. The impact of gender on item format differences was minor. Implications for secondary research and assessment design are discussed.

Where nonresponse is at its loudest: Cross-country and individual differences in item nonresponse across the PISA 2018 student questionnaire

The Programme for International Student Assessment (PISA) student questionnaire, despite being designed for low cognitive demand, may induce test burden due to its 306-item length, resulting in increased item nonresponse toward the questionnaire’s end. Using the PISA 2018 response data from 80 countries and a cross-classified mixed effects model, this study explored the average, cross-country, and individual nonresponse trends throughout the questionnaire. The average nonresponse probability increased from 2% at the questionnaire’s start to 11% at its end. East Asian countries displayed the flattest nonresponse trends, while South American countries showed the steepest. Individual differences were also prominent: students with a lower initial nonresponse propensity experienced a steeper increase in their nonresponse propensity as they progressed through the questionnaire, whereas those with a higher initial propensity remained at a relatively constant level. Our findings highlight the importance of the questionnaire design, the context in which it is administered, and its interaction with individual and cultural factors in shaping student response behavior. We discuss the implications for questionnaire design and researchers using PISA student questionnaire data for secondary analysis.

College Selectivity, Choice of Major, and Post-College Earnings

<p style="text-align:justify"><span style="font-family:Times New Roman,Times,serif"><span style="font-size:16px"><span dir="ltr" lang="EN-US"><span style="color:black">College choice and choice of major are the most important decisions for future earnings. It is still unclear, however, what makes a greater difference—college or major—or whether a choice of college matters more for some majors, but not the others. Using cross-classified models and College Scorecard data, I show that a discipline is more consequential for future earnings than a college. The effect of STEM is substantial but is less pronounced at institutions with higher overall median earnings. The effect of college selectivity on earnings is more pronounced for non-STEM disciplines. Institutional characteristics—such as tuition, shares of graduates receiving different forms of financial aid, institutional size and location, and type of college—correlate with earnings of graduates. Racial and gender composition of an educational program correlate with expected earnings of its graduates even after control for other institutional and disciplinary characteristics. Models presented here provide a better understanding of the effect of college and major choices on future earnings.</span></span></span></span></p>

Student characteristics and effort during test-taking

BackgroundAchievement tests are designed as measurement tools for student knowledge and learning, but also reflect student effort during the test. Understanding better what determines differences in (different dimensions of) effort can help in understanding what measured achievement differences reflect. AimWe analyze how test-taking effort relates to students’ demographic characteristics, past attainment and personality traits. Data13,791 9th grade students in the Netherlands, administered in 2012, 2014 and 2016, answering a total of 449,956 observations. MethodsWe distinguish between two measures of effort: solution behavior and response time given solution behavior. We estimate multi-level cross-classified models that include individual and test characteristics as predictors. We further include interaction terms between question position and individual characteristics, to identify how effort decline across the test differs by student type. ResultsGirls, high achievers, more agreeable, more conscientious and less extravert students exert more effort. Differences by past achievement are especially large and further increase along the test, while differences in other characteristics tend to be more stable. Effort differences by socioeconomic status are relatively small. ConclusionsSystematic differences between different types of students can partly reflect differences in test effort across these groups. Thus, test effort should be considered when analyzing achievement gaps and differences across learning outcomes. Tests with different test lengths imply different measures of educational inequality through differential effort declines.

Examining the effects of prekindergarten to second grade classroom quality on early elementary achievement

Previous research has investigated the influence of preschool interventions in early elementary environments with some results suggesting that high quality early preschool experiences may have meaningful effects on student academic outcomes not only immediately following the preschool years but also in the long term. This study aimed to investigate this topic further by using a novel cross-classified model to account for the complex hierarchical structure of a large-scale longitudinal and multilevel dataset. First, we examined the carryover classroom effects from prekindergarten (PK), kindergarten (K), and first grade on later grades’ reading and mathematics scores. Second, we explored the effect of PK to second grade classroom quality on students’ K, first, and second grade reading and mathematics scores. Our analyses yielded significant PK carryover classroom effects for reading; these results provide evidence that a student’s PK classroom continues to impact that student for at least three years after. Additionally, one domain of PK classroom quality —classroom organization—was found to be positively related to students’ K, first, and second grade reading outcomes.

Geographic and Physician-Level Variation in the Use of Hypofractionated Radiation Therapy for Breast Cancer in the U.S.: A Cross-Classified Multilevel Analysis

We aimed to assess geographic and physician-level variation for hypofractionated whole-breast irradiation (HF-WBI) use for early-stage breast cancer patients in the United States. We further evaluated the association between HF-WBI use and demographic factors after accounting for these variations. We performed a retrospective study of early-stage breast cancer patients using private employer-sponsored insurance claims from 2008 to 2017. Patients were clustered according to geographic level and by radiation oncologist. Bayesian cross-classified multilevel logistic models were used to examine the geographic heterogeneity and variation of radiation oncologists simultaneously. Intracluster correlation coefficient (ICC) and median odds ratios (MOR) were calculated to quantify the variation at different levels. We also used the cross-classified model to identify patient demographic factors associated with receiving HF-WBI. The study included 79,747 women (74.0%) who received conventionally fractionated whole-breast irradiation (CF-WBI) and 27,999 women (26.0%) who underwent HF-WBI. HF-WBI adoption increased significantly across time (2008-2017). The variation in HF-WBI utilization was attributed mostly to physician-level variability (MOR = 2.59). The variability of HF-WBI utilization across core-based statistical areas (CBSAs) (MOR = 1.55) was found to be the strongest among all geographic classifications. After accounting for variability in both CBSAs and radiation oncologists, age, receiving chemotherapy, and several community-level factors, including distance from home to facility, community education level, and racial composition, were found to be associated with HF-WBI utilization. This study demonstrated geographic and physician-level heterogeneity in the use of HF-WBI among early-stage breast cancer patients. HF-WBI utilization was also found to be associated with patient and community-level characteristics. Given observed physician-level variability, intervention through continuing medical education could help doctors to better understand the advantages of HF-WBI and promote the adoption of HF-WBI in the U.S. Influence of physician-level characteristics on HF-WBI utilization merits further study.

Interviewer Ratings of Physical Appearance in a Large-Scale Survey in China

Abstract Interviewer ratings of respondents’ physical appearance have been collected in several major social surveys. While researchers have made good use of such ratings data in substantive studies, empirical evidence on their measurement properties is rather limited. This study evaluates two potential threats to the quality of interviewer ratings of physical appearance: interviewer effects and halo effects. Using data from the China Family Panel Studies, we show large interviewer effects on interviewer ratings of respondents’ physical appearance based on cross-classified models. We also provide possible evidence for halo effects based on high correlations between physical appearance ratings and other theoretically distinct constructs, after controlling for interviewer effects. However, we find support for convergent and discriminant validity of physical appearance ratings when both interviewer effects and halo effects are controlled for. Empirical studies using interviewer observation data should take into account interviewer effects and halo effects when possible or at least discuss their potential impact on the substantive findings.

Exploring Disparities for Obesity in Korea Using Hierarchical Age-Period-Cohort Analysis With Cross-Classified Random Effect Models.

This research article investigates the age, period, and birth cohort effects on prevalence of obesity in the Korean population, with the goal of identifying key factors to inform effective public health strategies. We analyzed data from the Korea National Health and Nutrition Examination Survey, spanning 2007-2021, including 35,736 men and 46,756 women. Using the hierarchical age-period-cohort (APC) analysis with cross-classified random effects modeling, we applied multivariable mixed logistic regression to estimate the marginal prevalence of obesity across age, period, and birth cohort, while assessing the interaction between APC and lifestyle and socioeconomic factors. Our findings reveal an inverted U-shaped age effect on obesity, influenced by smoking history (P for interaction = 0.020) and physical activity (I for interaction < 0.001). The period effect was positive in 2020 and 2021, while negative in 2014 (P for period effect < 0.001). A declining trend in obesity prevalence was observed in birth cohorts from 1980s onward. Notably, disparities in obesity rates among recent birth cohorts have increased in relation to smoking history (P for interaction = 0.020), physical activity (P for interaction < 0.001), and residence (P for interaction = 0.005). Particularly, those born after 1960 were more likely to be obese if they were ex-smokers, physical inactive, or lived in rural areas. These findings highlight growing disparities in obesity within birth cohorts, underscoring the need for targeted health policies that promote smoking cessation and physical activity, especially in rural areas.

Approaches to Item-Level Data with Cross-Classified Structure: An Illustration with Student Evaluation of Teaching

Student evaluation of teaching (SET) questionnaires are ubiquitously applied in higher education institutions in North America for both formative and summative purposes. Data collected from SET questionnaires are usually item-level data with cross-classified structure, which are characterized by multivariate categorical outcomes (i.e., multiple Likert-type items in the questionnaires) and cross-classified structure (i.e., non-nested students and instructors). Recently, a new approach, namely the cross-classified IRT model, was proposed for appropriately handling SET data. To inform researchers in higher education, in this article, the cross-classified IRT model, along with three existing approaches applied in SET studies, including the cross-classified random effects model (CCREM), the multilevel item response theory (MLIRT) model, and a two-step integrated strategy, was reviewed. The strengths and weaknesses of each of the four approaches were also discussed. Additionally, the new and existing approaches were compared through an empirical data analysis and a preliminary simulation study. This article concluded by providing general suggestions to researchers for analyzing SET data and discussing limitations and future research directions.

Effects of family and neighborhood vulnerability on dual language learner and monolingual children’s preschool outcomes

IntroductionResearch has documented that home and neighborhood contexts of children from low-income families are associated with lower cognitive and social–emotional skills than their higher-income peers. Even though over a third of young children growing up in poverty are dual language learners (DLLs), little research has examined how contextual effects differ between DLL and monolingual children. The current study examines how these two contexts, neighborhood vulnerability and family socioeconomic risk, impact executive function (EF) and social–emotional skills in DLL and monolingual preschoolers.MethodsA secondary analysis was completed on data from two Head Start programs. A series of cross-classified models with interactions were conducted to examine the moderating role of DLL status on associations between neighborhood vulnerability and family risk and preschoolers’ EF and social-emotional skills.ResultsProficient bilingual children’s EF skills were not impacted by neighborhood risks, suggesting that proficient bilingual children may have more opportunities to grow their EF skills when switching between English and Spanish regardless of neighborhood context. An unexpected result occurred for emergent bilingual children who were reported to demonstrate fewer behavior problems regardless of family risk, highlighting the importance of ensuring all DLL families have access to resources to promote their children’s social–emotional skills; and teachers have the proper training to support the behaviors of children in their classroom with varying levels of English proficiency.DiscussionAlthough speaking two languages may be a protective factor for young DLLs growing up in poverty, little research has examined how contextual effects differ between DLL and monolingual children. The current study contributes by examining how DLL status, especially two different DLL statuses (i.e., Proficient Bilinguals and Emergent Bilinguals), may vary as a buffer in moderating the negative associations between collective neighborhood vulnerability, individual family risk, and children’s EF and social–emotional skills.

Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching

The cross-classified data structure is ubiquitous in education, psychology, and health outcome sciences. In these areas, assessment instruments that are made up of multiple items are frequently used to measure latent constructs. The presence of both the cross-classified structure and multivariate categorical outcomes leads to the so-called item-level data with cross-classified structure. An example of such data structure is the routinely collected student evaluation of teaching (SET) data. Motivated by the lack of research on multilevel IRT modeling with crossed random effects and the need of an approach that can properly handle SET data, this study proposed a cross-classified IRT model, which takes into account both the cross-classified data structure and properties of multiple items in an assessment instrument. A new variant of the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm was introduced to address the computational complexities in estimating the proposed model. A preliminary simulation study was conducted to evaluate the performance of the algorithm for fitting the proposed model to data. The results indicated that model parameters were well recovered. The proposed model was also applied to SET data collected at a large public university to answer empirical research questions. Limitations and future research directions were discussed.

Gender differences in item nonresponse in the PISA 2018 student questionnaire

Gender differences in item nonresponse are well-documented in high-stakes achievement tests, where female students are shown to omit more items than male students. These gender differences in item nonresponse are often linked to differential risk-taking strategies, with females being risk-averse and unwilling to guess on an item, even if it could gain them credits. In low-stakes settings, similar trends should not apply, as the students carry no consequence for their performance. Instead, test-taking motivation is argued to be the pivoting factor, with female students seen as more motivated and omitting fewer items than male students. In contrast to the high- and low-stakes achievement tests, less is known about gender differences in item nonresponse in student background questionnaires. Using cross-classified mixed effects models, we examined gender differences in item nonresponse on the Programme for International Student Assessment (PISA) 2018 student questionnaire across 80 countries and 71 scales. On average, the odds of male students omitting a questionnaire item were double the odds of female students, consistent with the expected trend in the low-stakes setting. However, we show that gender differences in item nonresponse are not merely a function of the stakes involved for individual students but a more complex phenomenon that is context-dependent and not necessarily stable across countries, scales’ formats, and contents. We argue that examining differences in item nonresponse patterns could serve as a source of additional information about the students’ test-taking behaviour and the quality of the questionnaire.

Factors Associated with the Digital Patient Experience of Virtual Care Across Specialties.

We aimed to characterize patient experience with virtual care across medical specialties using validated survey data. Primary objective: to determine whether experience varied by visit modality (virtual vs. in-person) and whether relationships persisted after adjusting for patient and provider characteristics. Secondarily, among physicians with sufficient data, we compared virtual versus in-person patient experience scores at the physician level and identified characteristics associated with better experience scores for virtual care. This was a retrospective analysis of administrative databases from a large New England health care system, including all ambulatory visits from October 1, 2020 to September 30, 2021 with patient experience scores recorded. We compared experience between virtual and in-person at the visit level (score: 0-10) and the physician level for likelihood of recommending the physician to friends or family. We used a series of cross-classified hierarchical models with visits grouped by patient and by physician to decompose sources of variation. Among physicians with sufficient data, we compared physicians with higher virtual versus higher in-person net promoter score (NPS). Of 378,472 visits performed by 3368 physicians, 86,878 (23%) were conducted virtually. Most scored ≥9 for either modality, with a small preference for virtual versus in-person care (9.6 vs. 9.5, p < 0.001). We found that more variation in scores was explained by patient than by physician (22.9% vs. 3%). Visit modality was of minimal explanatory value. Most physicians' virtual and in-person NPS were similar, and virtual visit volume was not associated. We found robust evidence for the parity of patient experience between virtual and in-person modalities across specialties.

Learning from Forest Trees: Improving Urban Tree Biomass Functions

Trees are one of the few carbon sinks in urban areas. Different methods are available to assess the biomass of urban trees, one of these being allometric biomass functions. Biomass functions are well investigated, reliable and easy to apply if the required information is available. Our goal is to use biomass functions to enhance urban forest management tools with information on stored biomass and carbon. In this study, we test several approaches to estimate new species-specific biomass functions. We include data from urban and traditional forest trees since both origins can be modeled by the allometric relationship solely giving different parameter estimates. The tested models include mixed allometric models for urban trees only, the adjustments of available forest tree biomass models and a cross-classified mixed model (CCMM) using both data from urban and forest trees. We then show by cross validation that the CCMM, statistically separating the data into different species and origins, shows greater improvement over the simpler models. Hence, we state that the inclusion of forest tree data improves the performance on biomass predictions for urban trees: the urban tree biomass functions “learned from forest trees”. The CCMM is also compared against the predictions of the above-ground biomass functions applied in the German National Forest Inventory. Comparable RMSE and slightly lower BIAS values are found, both for deciduous and coniferous tree species. With the approach of a cross-classified model, we also enable predictions for non-observed conifers in urban space, assuming comparable growth differences between deciduous and conifer species in forest stands and urban areas. A sample application using the CCMM model shows results for a small subset of data of an urban tree inventory, collected in a residential area in the city of Munich, Germany. It is applied to estimate carbon storage at two points in time and, hence, carbon fluxes in the period under consideration. Such information can help in the decision making and management of urban trees.

From Flirting to F*cking: Examining the Robustness of the Precarious Sexuality Effect.

Men, versus women, face more doubts about their heterosexuality based on a single same-gender sexual experience, a phenomenon known as the precarious sexuality effect. This phenomenon has thus far only been examined with sexually explicit same-gender acts (e.g., kissing, oral sex). Here, we conducted secondary data analyses of five large datasets of US adults (total N = 9770) to examine the replicability and robustness of the precarious sexuality effect across a range of same-gender acts varying in sexual explicitness. Using cross-classified hierarchical linear modeling, we replicated the precarious sexuality effect across all samples and demonstrated-in preregistered exploratory and confirmatory tests-that this effect was moderated by sexual explicitness. Consistent with our predictions, same-gender acts that were merely flirtatious (e.g., blowing a kiss) elicited stronger precarious sexuality (i.e., target gender) effects than same-gender acts that were explicitly sexual (e.g., oral sex), presumably because the former acts are more ambiguous and thereby allow more room for interpretation. Further, we found no consistent evidence that the precarious sexuality effect was moderated by perceiver characteristics including gender, sexual orientation, age, race, gender role beliefs, religiosity, or political orientation. Discussion considers possible explanations for the precarious sexuality effect and identifies important avenues for future research.

Workplace socioeconomic characteristics and coronary heart disease: a nationwide follow-up study

ObjectivesImportant gaps in previous research include a lack of studies on the association between socioeconomic characteristics of the workplace and coronary heart disease (CHD).We aimed to examine two contextual factors...

Individual Mobility across Clusters: The Impact of Ignoring Cross-Classified Data Structures in Discrete-Time Survival Analysis

A multilevel-discrete time survival model may be appropriate for purely hierarchical data, but when data are non-purely hierarchical due to individual mobility across clusters, a cross-classified discrete time survival model may be necessary. The purpose of this research was to investigate the performance of a cross-classified discrete-time survival model and assess the impact of ignoring a cross-classified data structure on the model parameters of a conventional discrete-time survival model and a multilevel discrete-time survival model. A Monte Carlo simulation was used to examine the performance of three discrete-time survival models when individuals are mobile across clusters. Simulation factors included the value of the between-clusters variance, number of clusters, within-cluster sample size, Weibull scale parameter, and mobility rate. The results suggest that substantial relative parameter bias, unacceptable coverage of the 95% confidence intervals, and severely biased standard errors are possible for all model parameters when a discrete-time survival model is used that ignores the cross-classified data structure. The findings presented in this study are useful for methodologists and practitioners in educational research, public health, and other social sciences where discrete-time survival analysis is a common methodological technique for analyzing event-history data.

Mirror, mirror on the wall, who is the fairest dancer of them all? A naturalistic lens model study on the judgment of dance performance

Success as a dancer is closely associated with positive dance judgments by perceivers. Although dancers’ physical appearance (attractiveness, style) might affect dance judgments beyond dance-specific attributes (technique, expression), they have largely been unconsidered in previous studies. To contribute to a comprehensive explanation of real-life dance judgments, we applied the lens model, an approach explicitly developed to explain the emergence of social judgments by multiple attributes. Therefore, video-records of 70 solo performances were (1) rated regarding dancers’ physical appearance, technique, and expression and (2) judged by 33 perceivers. Results of cross-classified mixed-effects models revealed that attributes of all domains were significantly related to dance judgements. Considered simultaneously, however, only dance-specific attributes contributed to the prediction of dance judgments. Additional moderation analyses underscored the importance of perceivers’ expertise in judging dance. We discuss the lens model as suitable framework for a naturalistic approach to the study of aesthetic experiences and sports performances.

Comparing random effects models, ordinary least squares, or fixed effects with cluster robust standard errors for cross-classified data.

Cross-classified random effects modeling (CCREM) is a common approach for analyzing cross-classified data in psychology, education research, and other fields. However, when the focus of a study is on the regression coefficients at Level 1 rather than on the random effects, ordinary least squares regression with cluster robust variance estimators (OLS-CRVE) or fixed effects regression with CRVE (FE-CRVE) could be appropriate approaches. These alternative methods are potentially advantageous because they rely on weaker assumptions than those required by CCREM. We conducted a Monte Carlo Simulation study to compare the performance of CCREM, OLS-CRVE, and FE-CRVE in models, including conditions where homoscedasticity assumptions and exogeneity assumptions held and conditions where they were violated, as well as conditions with unmodeled random slopes. We found that CCREM out-performed the alternative approaches when its assumptions are all met. However, when homoscedasticity assumptions are violated, OLS-CRVE and FE-CRVE provided similar or better performance than CCREM. When the exogeneity assumption is violated, only FE-CRVE provided adequate performance. Further, OLS-CRVE and FE-CRVE provided more accurate inferences than CCREM in the presence of unmodeled random slopes. Thus, we recommend two-way FE-CRVE as a good alternative to CCREM, particularly if the homoscedasticity or exogeneity assumptions of the CCREM might be in doubt. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Estimating promotion effects in email marketing using a large-scale cross-classified Bayesian joint model for nested imbalanced data

We consider a large-scale, cross-classified nested (CRON) joint model for modeling customer responses to opening, clicking, and purchasing from promotion emails. Our logistic regression-based joint model contains crossing of promotions and customer effects and allows estimation of the heterogeneous effects of different promotion emails, after adjusting for customer preferences, attributes, and historical behaviors. Using data from an email marketing campaign of an apparel retailer, we exhibit the varying effects of promotions not only based on the contents of the email but also across the three different stages, viz. open, click, and purchase of the conversion funnel. We conduct Bayesian estimation of the parameters in the joint model by using a block Metropolis–Hastings algorithm that not only incorporates nested subsampling to tackle the severe imbalance between conversions and no conversions but also uses additive transformation-based modifications of random walk Metropolis to scale estimation for large numbers of customers. We extend our approach to a segmented cross-classified nested (SCRON) joint model that encompasses the possibility of varying promotion effects across different customer segments. The resultant high-dimensional model is estimated using spike-and-slab priors on the promotion and customer segment interactions. Our nested joint model accounts for the correlations in customer preferences across the conversion funnel. Based on the promotion estimates from the model, we demonstrate how marketers can use different priced, nonpriced, and combination of price and nonprice promotions to increase brand awareness or increase purchases. Comparing estimates from CRON and SCRON models, we display the benefits of targeted marketing by using email promotion lists which are separately optimized for the different customer segments.