To Score or Not to Score? A Simulation Study on the Performance of Test Scores, Plausible Values, and SEM, in Regression With Socio-Emotional Skill or Personality Scales as Predictors.

Nivedita Bhaktha,Clemens M Lechner

doi:10.3389/fpsyg.2021.679481

Abstract

This article addresses a fundamental question in the study of socio-emotional skills, personality traits, and related constructs: “To score or not to score?” When researchers use test scores or scale scores (i.e., fallible point estimates of a skill or trait) as predictors in multiple regression, measurement error in these scores tends to attenuate regression coefficients for the skill and inflate those of the covariates. Unlike for cognitive assessments, it is not fully established how severe this bias can be in socio-emotional skill assessments, that is, how well test scores recover the true regression coefficients — compared with methods designed to account for measurement error: structural equation modeling (SEM) and plausible values (PV). The different types of scores considered in this study are standardized mean scores (SMS), regression factor scores (RFS), empirical Bayes modal (EBM) score, weighted maximum likelihood estimates (WLE), and expected a posteriori (EAP) estimates. We present a simulation study in which we compared these approaches under conditions typical of socio-emotional skill and personality assessments. We examined the performance of five types of test scores, PV, and SEM with regard to two outcomes: (1) percent bias in regression coefficient of the skill in predicting an outcome; and (2) percent bias in the regression coefficient of a covariate. We varied the number of items, factor loadings/item discriminations, sample size, and relative strength of the relationship of the skill with the outcome. Results revealed that whereas different types of test scores were highly correlated with each other, the ensuing bias in regression coefficients varied considerably. The magnitude of bias was highest for WLE with short scales of low reliability. Bias when using SMS or WLE test scores was sometimes large enough to lead to erroneous research conclusions with potentially adverse implications for policy and practice (up to 55% for the regression coefficient of the skill and 20% for that of the covariate). EAP, EBM, and RFS performed better, producing only small bias in some conditions. Additional analyses showed that the performance of test scores also depended on whether standardized or unstandardized scores were used. Only PV and SEM performed well in all scenarios and emerged as the clearly superior options. We recommend that researchers use SEM, and preferably PV, in studies on the (incremental) predictive power of socio-emotional skills.

Highlights

Assessing socio-emotional skills1 is becoming increasingly common in large-scale assessment surveys (LSAS) and beyond (Abrahams et al, 2019; Lechner et al, 2019)
We present a comprehensive simulation study in which we compare the performance of five different types of test scores, Structural Equation Modeling (SEM), and plausible values (PV) in scenarios where the focus is on the predictive power of socio-emotional skills in a regression
We present a comprehensive simulation study in which we assess the performance of the three principal approaches to analyzing skills as predictors in multiple regression under conditions that are typical for socio-emotional skill and personality assessments

Summary

Introduction

Assessing socio-emotional skills ( known as “non-cognitive skills,” “twenty-first century skills.” “character strengths,” or “soft skills”) is becoming increasingly common in large-scale assessment surveys (LSAS) and beyond (Abrahams et al, 2019; Lechner et al, 2019). Akin to many other research areas (Aiken and West, 1991; Westfall and Yarkoni, 2016; Sengewald et al, 2018), it is routinely of importance to examine whether socio-emotional skills incrementally predict an outcome above and beyond covariates such as cognitive skills, socioeconomic status, or other established predictors of that outcome (e.g., Roberts et al, 2007; Rammstedt et al, 2017; Bergner, 2020; Harzer, 2020; Wagner et al, 2020) That is, such studies are intent on demonstrating the (incremental) predictive validity of socio-emotional skills for consequential life outcomes, which is taken as evidence for the relevance of socioemotional skills

Methods

Results

Conclusion