Validity Of Test Score Interpretations Research Articles

Validity of test score interpretations and cross-cultural comparisons in the First and Second International Science Studies

AbstractIn international large-scale assessments, student performance comparisons across educational systems are frequently done to assess the state and development in different domains. These results often have a large impact on educational policy and on the perceptions of an educational system’s performance. Early assessments, such as the First and Second International Science Studies (FISS and SISS), have been used alongside recent studies to create unique scales for investigating changes in constructs. The implicit assumptions in system comparisons are that the measures are valid, reliable, and comparable. However, these assumptions have not always been investigated thoroughly. This study aims to investigate the validity and cross-system comparability of scores from the FISS and SISS, conducted by the International Association for the Evaluation of Educational Achievement in 1970–1971 and 1983–1984. Findings based on item response theory (IRT) modeling indicate that scores in most educational systems can be viewed as reliable measures of a single science construct, supporting the validity of test score interpretations in these educational systems individually. In a robust assessment of measurement invariance using standard IRT methods, an alignment-based method, and the root mean square difference (RMSD) fit statistic, we demonstrate that measurement invariance is violated across systems. The alignment-based method identified a well-fitting model with complex restrictions but no items exhibited invariance across all systems, a result supported by the RMSD statistics. These results question the appropriateness of score comparisons across systems in FISS and SISS. We discuss the implications of these results and outline consequences for score comparisons across time.

Using Item Scores and Distractors to Detect Item Compromise and Preknowledge

Any time examinees have had access to items and/or answers prior to taking a test, the fairness of the test and validity of test score interpretations are threatened. Therefore, there is a high demand for procedures to detect both compromised items (CI) and examinees with preknowledge (EWP). In this article, we develop a procedure that uses item scores and distractors to simultaneously detect CI and EWP. The false positive rate and true positive rate are evaluated for both items and examinees using detailed simulations. A real data example is also provided using data from an information technology certification exam.

Examination of differential effects of cognitive abilities on reading and mathematics achievement across race and ethnicity: Evidence with the WJ IV

There has been little research investigating the predictive validity of modern intelligence tests for racially and ethnically diverse students. The validity of test score interpretation within educational and psychological assessment assumes that test scores predict educationally relevant phenomena equally well for individuals, regardless of group membership (American Educational Research Association et al., 2014; Messick, 1995; Warne et al., 2014). We used multiple group latent variable structural equation modeling (SEM) to investigate Cattell-Horn-Carroll general (g) and broad cognitive abilities on reading and mathematics achievement and whether these differed between racial (African American, Asian, and Caucasian) and ethnic (Hispanic, non-Hispanic) children and adolescents within the Woodcock-Johnson IV norming sample (N = 3127). After establishing construct equivalence across racial and ethnic groups, supporting the consistent calculation of composite scores regardless of group membership, we then examined the predictive validity of intelligence on achievement. After controlling for parent education, findings suggested two instances of differential predictive relations: (a) general intelligence had larger influences on basic reading skills for Caucasians when compared to Asian peers, and (b) comprehension-knowledge had larger influences on basic reading skills for Asians when compared to Caucasian peers. The overall pattern of findings suggests there is little to no predictive bias with the WJ IV. However, the findings indicate that when latent mean differences exist (after establishing strong factorial invariance), then bias will be introduced into the estimation of regression parameters used to identify differential predictive validity. Thus, even when measurement invariance is supported, differential prediction bias is inevitable when there are mean differences in the scores used as predictors. Future test bias research should consider latent ability differences and how that may impact findings of bias, and possibly, socioeconomic status-related indicators when assessing for measurement or prediction bias in intelligence and achievement tests.

Application of Sampling Variance of Item Response Theory Parameter Estimates in Detecting Outliers in Common Item Equating.

In common item equating, the existence of item outliers may impact the accuracy of equating results and bring significant ramifications to the validity of test score interpretations. Therefore, common item equating should involve a screening process to flag outlying items and exclude them from the common item set before equating is conducted. The current simulation study demonstrated that the sampling variance associated with the item response theory (IRT) item parameter estimates can help detect outliers in the common items under the 2-PL and 3-PL IRT models. The results showed the proposed sampling variance statistic (SV) outperformed the traditional displacement method with cutoff values of 0.3 and 0.5 along a variety of evaluation criteria. Based on the favorable results, item outlier detection statistics based on estimated sampling variability warrant further consideration in both research and practice.

Development and initial validation of scales to assess Decisional Balance (NSSI-DB), Processes of Change (NSSI-POC), and Self-Efficacy (NSSI-SE) in a population of young adults engaging in nonsuicidal self-injury.

Tools for assessing an individual's readiness to change nonsuicidal self-injury (NSSI) are limited. The Transtheoretical model of change (TTM) has been useful in predicting behavior change and guiding interventions for a variety of health behaviors including eating disorders, substance use disorders, and exercise. The aim of the current study was to examine the validity of test score interpretations from 3 newly developed measures to assess NSSI behavior change. Following in-depth interviews, modifications to existing TTM scales were made and tested with a set of web-based measures delivered to people who self-injure. This resulted in 11 new scales that reflect 3 domains associated with behavior change: NSSI-Decisional Balance (NSSI-DB), NSSI-Processes of Change (NSSI-POC), and NSSI-Self-Efficacy (NSSI-SE). Overall, results provide support for the reliability of test scores and validity of test scores interpretations. Implications for research, clinical use, and next steps are discussed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Construct validity of the G-CPAQ and its mediating role in pain interference and adjustment.

The Chronic Pain Acceptance Questionnaire (CPAQ) is a measure of pain acceptance comprised of pain willingness (PW) and activity engagement (AE; McCracken et al., 2004). Concerns about the factorial structure of the CPAQ exist, as it is not yet clear whether PW and AE constitute 2 independent constructs or 1, pain acceptance. Concerns also exist about the internal and predictive validity of test score interpretations of this measure. This study also presents that the choice of predictor variables has contributed to theoretical confusion regarding the impact of pain acceptance on pain-related adjustment. The purpose of this study was: (a) to examine the psychometric properties of both the long (20 items) and short (8 items) versions of the Greek-Chronic Pain Acceptance Questionnaire (G-CPAQ); (b) to examine the utility of a 2-factor solution in predicting psychosocial adjustment to pain using confirmatory factor analysis; and (c) to explore the mediating effects of pain acceptance and cognitive defusion, comprising the "open" response style to pain, between pain interference and pain related outcomes. One hundred and sixty chronic pain patients completed a questionnaire packet including pain indexes, pain acceptance, cognitive fusion, avoidance, and emotional distress. Confirmatory factor analyses supported the 2-factor solution, though a general good model fit was achieved only for the short G-CPAQ version. Structural equation modeling showed that PW and AE coupled with cognitive defusion partially mediated the influence of pain interference on pain severity, emotional distress, and avoidance of pain. (PsycINFO Database Record

Differential Item Functioning: Beyond validity evidence based on internal structure.

In the latest release of the Standards for Educational and Psychological Testing, Differential Item Functioning (DIF) is considered as validity evidence based on internal structure. However, there are no indications of how to design a DIF study as a validation study. In this paper, we propose relating DIF to all sources of validity evidence, and provide a general conceptual framework for transforming “typical” DIF studies into validation studies. We perform a comprehensive review of the literature and make theoretical and practical proposals. The article provides arguments in favour of addressing DIF detection and interpretation as validation studies, and suggestions for conducting DIF validation studies. The combination of quantitative and qualitative data within a mixed methods research perspective, along with planning DIF studies as validation studies, can help improve the validity of test score interpretations.

Factor structure and validity of the State-Trait Inventory for Cognitive and Somatic Anxiety.

The State-Trait Inventory for Cognitive and Somatic Anxiety (STICSA; Ree, French, MacLeod, & Locke, 2008) is a relatively new measure of anxiety. The current research investigated the factor structure and reliability of scores on the STICSA and the validity of the interpretation of STICSA scores in a sample of undergraduate students. Participants completed a battery of self-report questionnaires online, including measures of anxiety, depression, affect, and social desirability. Scores on the 4 subscales of the STICSA-Trait Cognitive, Trait Somatic, State Cognitive, and State Somatic-exhibited good internal consistencies (αs ≥ .92). Results of a confirmatory factor analysis provided support for a hierarchical model of the STICSA including a global anxiety factor plus 4 specific factors corresponding to the STICSA subscales. Support was also found for a four-factor model, with factors corresponding to the STICSA subscales. Pearson product-moment correlations with other measures of anxiety provided evidence of the convergent validity of the interpretation of STICSA scores, and Pearson product-moment correlations with measures of depression and affect provided evidence of the divergent validity of the interpretation of STICSA scores. The STICSA is the only existing self-report anxiety measure that contains scales measuring state and trait anxiety as well as cognitive and somatic anxiety. Comparisons between the convergent and divergent validity of test score interpretations of the STICSA and the State-Trait Anxiety Inventory (STAI; Spielberger et al., 1983) revealed that the STICSA has better convergent validity with measures of somatic anxiety and better divergent validity with measures of depression and affect. (PsycINFO Database Record

What Are We Aligning Tests to When We Report Test Alignment to the CEFR?

The study reported here investigates the validity of judgments made when aligning tests to the Common European Framework of Reference (CEFR). Listening tests operationalizing pre-defined difficulty-determining characteristics were to be aligned to CEFR levels. We employed a modified version of the item-descriptor-matching-method. Ten judges stated the CEFR descriptors they thought each item operationalizes and assigned a global CEFR level per item. We compared agreement on CEFR level judgments and CEFR descriptors quoted. Analyzing the relationship between CEFR level judgments and item ratings of difficulty-determining characteristics shed light on further criteria the judges employed. Follow-up interviews helped to triangulate the findings by examining judges’ perceptions of the alignment procedure.We found that judges relied on different criteria and CEFR descriptors to a varying degree, interpreting CEFR levels differently. There seemed little comparability in what aspects judges used to form their global CEFR judgments. Therefore, if an alignment does not take into account the meaning of the CEFR levels as expressed by existing CEFR descriptors, this raises issues with alignment validity, and hence the validity of test-score interpretation and usage. Given the impact of using CEFR aligned tests for high-stakes purposes, this article aims to shed more light on what assigning a CEFR level to a test actually means.

Assessment of nonsuicidal self-injury: development and initial validation of the Non-Suicidal Self-Injury-Assessment Tool (NSSI-AT).

Research tools for assessing nonsuicidal self-injury (NSSI) epidemiology in community populations are few and are either limited in the scope of NSSI characteristics assessed or included as part of suicide assessment. Though these surveys have been immensely useful in establishing the presence of NSSI and in documenting basic epidemiological characteristics, they have been less useful in describing secondary NSSI features such as NSSI context, habituation, or perceived life impact. The aim of the current study was to examine the reliability of the test scores and validity of test score interpretations in a university population for the Non-Suicidal Self-Injury-Assessment Tool (NSSI-AT), a web-based measure of NSSI designed to assess primary (such as form, frequency, and function) and secondary (including but not limited to NSSI habituation; contexts in which NSSI is practiced; and NSSI perceived life interference, treatment, and impacts) NSSI characteristics for research purposes. Data for these analyses were drawn from 3 samples, all of which were originally part of a 2007 study of randomly selected students from 8 northeast and midwest public and private universities that participated in a web-based study entitled the Survey of Student Wellbeing. Overall, results provide support for the reliability of NSSI-AT test scores (as assessed by test-retest) and validity of NSSI-AT test score interpretations for the behavior and frequency modules (as assessed using concurrent, convergent, and discriminant evidence) in this population. Implications for research as well as next steps are discussed.

Test Directions as a Critical Component of Test Design: Best Practices and the Impact of Examinee Characteristics

The purpose of test directions is to familiarize examinees with a test so that they respond to items in the manner intended. However, changes in educational measurement as well as the U.S. student population present new challenges to test directions and increase the impact that differential familiarity could have on the validity of test score interpretations. This article reviews the literature on best practices for the development of test directions as well as documenting differences in test familiarity for culturally and linguistically diverse students that could be addressed with test directions and practice. The literature indicates that choice of practice items and feedback are critical in the design of test directions and that more extensive practice opportunities may be required to reduce group differences in test familiarity. As increasingly complex and rich item formats are introduced in next-generation assessments, test directions become a critical part of test design and validity.

Motivation Filtering on a Multi-Institution Assessment of General College Outcomes

Possible lack of motivation is a perpetual concern when tests have no stakes attached to performance. Specifically, the validity of test score interpretations may be compromised when examinees are unmotivated to exert their best efforts. Motivation filtering, a procedure that filters out apparently unmotivated examinees, was applied to the Collegiate Learning Assessment (CLA), an open-ended test of general college outcomes. Filtering based on self-reported effort was compared to a new approach based on CLA performance relative to expected. Both approaches had expected positive effects on aggregate CLA scores, but the new approach achieved these gains while filtering out fewer students and maintaining sample representativeness. Motivation filtering had negligible effects on the relative standings of schools based on their average CLA scores, although the impact of filtering was slightly greater on lower-performing schools. Motivation filtering increased longitudinal estimates of growth on the CLA because of apparently lower motivation among second-semester sophomores compared to entering freshmen.

Test Anxiety, Computer -Adaptive Testing, and the Common Core

This paper highlights the current findings and issues regarding the role of computer-adaptive testing in test anxiety. The computer-adaptive test (CAT) proposed by one of the Common Core consortia brings these issues to the forefront. Research has long indicated that test anxiety impairs student performance. More recent research indicates that taking a test in a CAT format can affect the ability estimates of students with test anxiety. Inaccurate measures of ability are disconcerting because of the threat they pose to the validity of test score interpretation. This paper raises concerns regarding how the implementation of a computer-adaptive test for a large-scale common core assessment system could differentially affect students with test anxiety. Issues of fairness and score comparability are raised, and the implications of these issues are discussed.

学内開発プレイスメントテスト得点解釈と使用の妥当性の評価について Evaluating Validity for In-House Placement Test Score Interpretations and Uses

Validity theory has evolved dramatically in the past few decades. The most prominent theory in recent years is an argument-based validity framework, proposed by Kane (1992, 2004, 2006). To evaluate test score interpretations and uses based on Kane’s framework, test developers first need to provide interpretive arguments and then validity arguments by proving sound warrants for the following four inferences: (a) scoring from observation to an observed score, (b) generalization from the observed score to the universe score, (c) extrapolation from the universe score to a target score, and (d) decision from the target score to use. In the field of language testing, a number of studies have been conducted to investigate the validity of test score interpretations and uses, especially for the ones considered to be high-stakes such as the TOEFL (Chapelle, 2008; Chapelle, Enright, & Jamieson, 2010). However, not many studies have been conducted to validate in-house placement test score interpretations and uses, and no study has evaluated the validity of such low-stakes tests using Kane’s validity framework. Regardless of whether the tests are high or low stakes, test developers need to be responsible for validating their test score interpretations and uses in order to attest to the validity. This study uses Kane’s (2006) argument-based validity framework to evaluate the validity of in-house placement test score interpretations and uses. The research questions are as follows: (a) to what extent do examinees get placement items correct and high-scoring examinees get more placement items correct; (b) to what extent are placement items consistently sampled from a domain sufficient in number so as to reduce measurement error; (c) to what extent do the difficulty of placement items match the objectives of a reading course; and (d) to what extent do placement decisions made to place examinees in their proper level of the course have an impact on washback in the course? An in-house placement test made up of 40-item grammar, 40-item vocabulary, and 10-item reading sections was developed and administered to 428 first-year private-university students in April 2010. The item format adopted was all multiple-choice so the answer sheets could be easily scored with a reader. Based on their test scores, about 60 high-scoring students and 50 low-scoring students were placed into one of two advanced or one of two basic reading classes. The remaining students were placed into one of several intermediate classes. A 55-item grammar achievement test was administered twice (once as a pretest and then again as a posttest) to the two basic and two intermediate classes. In addition, a 51-item class evaluation survey was administered to investigate students’ participation in the reading classes and to gauge students’ satisfaction with the classes and study support. Warrants for a validity argument of score inference were based on the results of the item analysis. A warrant for a validity argument of generalization inference was based on the composite generalizability coefficient of .92. A warrant for a validity argument of extrapolation inference was based on FACETS analysis, showing that difficulty estimates of learning levels were in an expected difficulty order. A warrant for validity arguments of decision inference was based on the basic-level students’ score gain on an achievement test and their positive reactions to a class evaluation survey. All the validity arguments presented in this study support the validity of the placement test score interpretations and uses. However, to further improve the validity of the test score interpretations and uses, it is necessary to investigate washback effects of the placement test in the reading classes and to revise the test to make grammar, vocabulary, and reading sections with 30 items each.

The Impact of Multidirectional Item Parameter Drift on IRT Scaling Coefficients and Proficiency Estimates

Item parameter drift (IPD) occurs when item parameter values change from their original value over time. IPD may pose a serious threat to the fairness and validity of test score interpretations, especially when the goal of the assessment is to measure growth or improvement. In this study, we examined the effect of multidirectional IPD (i.e., some items become harder while other items become easier) on the linking procedure and rescaled proficiency estimates. The impact of different combinations of linking items with various multidirectional IPD on the test equating procedure was investigated for three scaling methods (mean-mean, mean-sigma, and TCC method) via a series of simulation studies. It was observed that multidirectional IPD had a substantive effect on examinees' scores and achievement level classifications under some of the studied conditions. Choice of linking method had a direct effect on the results, as did the pattern of IPD.

From test validity to construct validity … and back?

Major changes in thinking about validity have occurred during the past century, shifting the focus in thinking from the validity of the test to the validity of test score interpretations. These changes have resulted from the 'new' thinking about validity in which construct validity has emerged as the central or unifying idea of validity today. Construct validity was introduced by Cronbach and Meehl in the mid-1950s in an attempt to address the validity of those many psychological concepts that have no clear referent in reality. To do this, construct validity theory required a nomological network--an elaborate theoretical network of constructs and observations connected by scientific laws--to validate the constructs. However, nomological networks are hard to come by and none that would do the job required by construct validity has been forthcoming to date. Thus, the current construct validity approach has retreated to one of simply 'interpretation and argument', but this seems to be too general to tie down the constructs in the way a nomological network would do to give credibility to the validity of the construct. As a result, the concept of validity seems to have been watered down and the credibility of validity claims weakened. The purpose of this paper is to encourage a discussion of the use of construct validity in medical education, and to suggest that test developers and users reconsider the use of abstract theoretical constructs that have no referent apart from theory. We present a critical review of these concerns about construct validity and provide for contrast a brief overview of a recently proposed view of measurement based on scientific realism and causality analysis.

Effects of diurnal variation on the Test of Variables of Attention performance in young adults with attention-deficit/hyperactivity disorder.

The Test of Variables of Attention (TOVA) is a continuous performance test that assesses attention, impulsivity, and processing speed. Continuous performance tests are used in the assessment of attention-deficit/hyperactivity disorder (ADHD) in children and adults. TOVA norms are based on a morning administration, and any TOVA administered after 1:00 p.m. is flagged as potentially invalid. Whereas the testing time recommendations make sense for pediatric samples, it is unclear whether they are appropriate for young adults, who typically show significant phase delay in their diurnal rhythms. The current study explores the impact of time of day on TOVA performance in young adults with ADHD. Participants were randomly assigned to either morning or afternoon administration. We found no significant diurnal variation in TOVA performance. We also found no interaction between diurnal preference and time of day of administration. Night owls endorsed more inattention symptoms on a self-report measure than more intermediate individuals but actually made significantly fewer omission (inattention) errors on the TOVA. Self-reported symptoms of inattention showed moderate, significant correlations with various TOVA performance indices. Self-reported symptoms of hyperactivity and impulsivity, however, showed no relationship to TOVA performance. These results suggest that the TOVA can be administered to adults with ADHD outside of the hours recommended in the manual without significantly compromising the interpretative validity of test score interpretation. Thus, a TOVA report that is consistent with ADHD should not be dismissed simply because it was administered in the late afternoon.

Effects of diurnal variation and caffeine consumption on Test of Variables of Attention (TOVA) performance in healthy young adults.

The Test of Variables of Attention (TOVA) is a continuous performance test (CPT) that assesses attention, impulsivity, and processing speed. CPTs are used in the assessment of attention-deficit/hyperactivity disorder (ADHD) in children, but more young adults are being assessed for ADHD as well. The TOVA norms are based on a standardization sample that was tested early in the day, and any TOVA administered after 1:00 p.m. will be flagged as potentially invalid. Whereas the testing time recommendations make sense for pediatric samples, it is unclear whether they are appropriate for young adults in college, who typically show significant phase delay in their diurnal rhythms. In addition, many college students consume large amounts of caffeine, and it is unclear how caffeine consumption affects TOVA performance. The current study examined the impact of time of day, self-reported diurnal preference, and caffeine consumption on TOVA performance in a double-blind, placebo-controlled experiment with healthy college students. There was evidence of diurnal variation on average response time and impulsivity but not on overall ADHD score, with participants tested in the afternoon responding faster but making more commission errors than did participants tested in the morning. Caffeine consumption led to significantly faster response times, but only for participants who typically consumed relatively little caffeine. We conclude that the TOVA can be administered to young adults outside the recommended time constraints without compromising the validity of test score interpretation but that the caffeine consumption of participants should be closely monitored.

Effect of first-encounter pretest on pass/fail rates of a clinical skills medical licensure examination

Due to the high-stakes nature of medical exams it is prudent for test agencies to critically evaluate test data and control for potential threats to validity. For the typical multiple station performance assessments used in medicine, it may take time for examinees to become comfortable with the test format and administrative protocol. Since each examinee in the rotational sequence starts with a different task (e.g., simulated clinical encounter), those who are administered non-scored pretest material on their first station may have an advantage compared to those who are not. The purpose of this study is to investigate whether pass/fail rates are different across the sequence of pretest encounters administered during the testing day. First-time takers were grouped by the sequential order in which they were administered the pretest encounter. No statistically significant difference in fail rates was found between examinees who started with the pretest encounter and those who encountered the pretest encounter later in the sequence. Results indicate that current examination administration protocols do not present a threat to the validity of test score interpretations.

Summative Assessment in Medicine: The Promise of Simulation for High‐stakes Evaluation

Throughout their careers, physicians are exposed to a wide array of assessments, including those aimed at evaluating knowledge, clinical skills, and clinical decision-making. While many of these assessments are used as part of formative evaluation activities, others are employed to establish competence and, as a byproduct, to promote patient safety. In the past 10 years, simulations have been successfully incorporated in a number of high-stakes physician certification and licensure exams. In developing these simulation-based assessments, testing organizations were able to promote novel test administration protocols, build enhanced assessment rubrics, advance sophisticated scoring and equating algorithms, and promote innovative standard-setting methods. Moreover, numerous studies have been conducted to identify potential threats to the validity of test score interpretations. As simulation technology expands and new simulators are invented, this groundbreaking work can serve as a basis for organizations to build or expand their summative assessment activities. Although there will continue to be logistical and psychometric problems, many of which will be specialty- or simulator-specific, past experience with performance-based assessments suggests that most challenges can be addressed through focused research. Simulation, whether it involves standardized patients (SPs), computerized case management scenarios, part-task trainers, electromechanical mannequins, or a combination of these methods, holds great promise for high-stakes assessment.