Interpretation Of Test Scores Research Articles

Using Corpus Analyses to Help Address the DIF Interpretation: Gender Differences in Standardized Writing Assessment.

Addressing differential item functioning (DIF) provides validity evidence to support the interpretation of test scores across groups. Conventional DIF methods flag DIF items statistically, but often fail to consolidate a substantive interpretation. The lack of interpretability of DIF results is particularly pronounced in writing assessment where the matching of test takers’ proficiency levels often relies on external variables and the reported DIF effect is frequently small in magnitude. Using responses to a prompt that showed small gender DIF favoring female test takers, we demonstrate a corpus-based approach that helps address DIF interpretation. To provide linguistic insights into the possible sources of the small DIF effect, this study compared a gender-balanced corpus of 826 writing samples matched by test takers’ performance on the reading and listening components of the test. Four groups of linguistic features that correspond to the rating dimensions, and thus partially represent the writing construct were analyzed. They include (1) sentiment and social cognition, (2) cohesion, (3) syntactic features, and (4) lexical features. After initial screening, 123 linguistic features, all of which were correlated with the writing scores, were retained for gender comparison. Among these selected features, female test takers’ writing samples scored higher on six of them with small effect sizes in the categories of cohesion and syntactic features. Three of the six features were positively correlated with higher writing scores, while the other three were negative. These results are largely consistent with previous findings of gender differences in language use. Additionally, the small differences in the language features of the writing samples (in terms of the small number of features that differ between genders and the small effect size of the observed differences) are consistent with the previous DIF results, both suggesting that the effect of gender differences on the writing scores is likely to be very small. In sum, the corpus-based findings provide linguistic insights into the gender-related language differences and their potential consequences in a testing context. These findings are meaningful for furthering our understanding of the small gender DIF effect identified through statistical analysis, which lends support to the validity of writing scores.

Group Differences in the Value of Subscores: A Fairness Issue

The aim of this article was to examine the quality of subscores from a test fairness perspective. We were using a method based on classical test theory to study group differences in the value that subscores from a typical college admission test add to the total test score. Significant group differences in such subscore value would raise questions about fairness for that testing program. In particular, differences in subscore value between subgroups based on certain accommodations could indicate problems with the appropriateness of the accommodation. The groups in this study were males and females, and additionally a typical time adaptation given to test takers in need of extended time was analyzed. We found that the value added ratio (VAR) for all subscores in the studied test were slightly larger for males than for females. There were some differences between subgroups with respect to the extended time accommodation, but they were not systematic in the same way as for males/females. While the differences in the subscores’ added value were small in general, they may in some instances be large enough to pose a threat to a valid and fair interpretation and use of test scores.

Validity Evidence in Science Achievement Assessments Found in a Sample of Published Research Articles on Science Teaching

The authors add to published empirical findings that an appropriate range of validity evidence for achievement tests, including science and mathematics, has either not been gathered, not reported, or is not accessible for independent review. The current study focuses on a sample of published, peer reviewed science intervention studies from a single journal using science achievement measures, over an 11-years period, with some discussion of validity, and finds results similar to previous studies that were conducted in other STEM areas and broader contexts. The consensus of the educational measurement profession is that validity is assessed by the extent to which evidence and theory support the proposed interpretation of test scores for proposed uses. What is troublesome is that a shortfall in validity evidence raises concerns about faulty, or insubstantial, test score interpretation to inform student short-term and long-term education and career trajectories, and to inform curricula intervention improvements. Discussion cautions readers that though the study findings report shortfalls in some kinds of validity evidence, this simply raises a flag for the test user to consider what kinds of validity evidence apply to her test use. It also raises a flag for the profession to explore the reasons for shortfalls.

The Redesigned TOEIC Bridge® Tests: Relations to Test‐Taker Perceptions of Proficiency in English

The redesigned four‐skills TOEIC Bridge® tests were designed to measure the listening, reading, speaking, and writing proficiency of beginning to low–intermediate English learners in the context of everyday life. In this paper, I describe two studies that were conducted to investigate claims about the meaningfulness of redesigned TOEIC Bridge test scores by comparing them to self‐assessments of listening, reading, speaking, and writing proficiency. In Study 1, test takers who participated in the redesigned TOEIC Bridge field test in Japan and Taiwan completed an online self‐assessment survey of their speaking and writing proficiency. In Study 2, test takers who participated in pretesting for the redesigned TOEIC Bridge Listening and Reading test in Japan and Taiwan completed a paper‐based self‐assessment survey of their listening and reading proficiency. Both studies were designed with the same research questions in mind in order to evaluate claims about the meaning of redesigned TOEIC Bridge test scores and to enhance the interpretation of test scores by relating them to test takers' self‐assessed ability to perform a variety of related tasks. The results indicated that self‐assessments had medium correlations with redesigned TOEIC Bridge Listening test scores (r = .55), Reading test scores (r = .54), Speaking test scores (r = .51), and Writing test scores (r = .46). These results compare favorably with the results of similar studies of the relationship between test scores and self‐assessments of language proficiency and provide evidence to support claims about the meaningfulness of TOEIC Bridge test scores. In addition, the pattern of results across TOEIC Bridge test proficiency levels and can‐do statements taken from language proficiency standards suggests that the tests meaningfully differentiate test‐takers' proficiency levels, an important consideration of test design.

ICT Engagement: a new construct and its assessment in PISA 2015

As a relevant cognitive-motivational aspect of ICT literacy, a new construct ICT Engagement is theoretically based on self-determination theory and involves the factors ICT interest, Perceived ICT competence, Perceived autonomy related to ICT use, and ICT as a topic in social interaction. In this manuscript, we present different sources of validity supporting the construct interpretation of test scores in the ICT Engagement scale, which was used in PISA 2015. Specifically, we investigated the internal structure by dimensional analyses and investigated the relation of ICT Engagement aspects to other variables. The analyses are based on public data from PISA 2015 main study from Switzerland (n = 5860) and Germany (n = 6504). First, we could confirm the four-dimensional structure of ICT Engagement for the Swiss sample using a structural equation modelling approach. Second, ICT Engagement scales explained the highest amount of variance in ICT Use for Entertainment, followed by Practical use. Third, we found significantly lower values for girls in all ICT Engagement scales except ICT Interest. Fourth, we found a small negative correlation between the scores in the subscale “ICT as a topic in social interaction” and reading performance in PISA 2015. We could replicate most results for the German sample. Overall, the obtained results support the construct interpretation of the four ICT Engagement subscales.

Interrater and Test-Retest Reliability of Performance-Based Clinical Tests Administered to Established Users of Lower Limb Prostheses.

A major barrier to reducing falls among users of lower limb prostheses (LLP) has been an absence of statistical indices required for clinicians to select and interpret scores from performance-based clinical tests. The study aimed to derive estimates of reliability, measurement error, and minimal detectable change values in performance-based clinical tests administered to unilateral LLP users. A total of 60 unilateral LLP users were administered the Narrowing Beam Walking Test, Timed ``Up and Go'' (TUG), Four Square Step Test (FSST), and 10-Meter Walk Test on 2 occasions, 3 to 9days apart. Intraclass correlation coefficients (ICCs) were calculated to assess interrater and test-retest reliability, while standard error of measurement (SEM) and minimal detectable change (MDC90) were derived to establish estimates of measurement error in individual scores or changes in score for each test. Interrater reliability ICCs (1,1) were high for all tests (ie, ≥0.98). Test-retest ICCs (2,1) varied by test, ranging from .88 for the TUG to .97 for the FSST. SEM and MDC90 varied between .39 and .96 and between .91seconds and 2.2seconds for the time-based tests (FSST, TUG, 10-Meter Walk Test). SEM and MDC90 for the Narrowing Beam Walking Test were .07 and .16, respectively. With the exception of the TUG, studied tests had test-retest ICCs (2,1) that exceeded the minimum required threshold to be considered suitable for group- and individual-level applications (ie, ICC≥0.70 and ≥0.90, respectively). Future research on individuals with dysvascular and transfemoral amputations or in specific age categories is required. Along with published validity indices, these reliability, error, and change indices can help clinicians select balance tests suitable for LLP users. They can also help clinicians interpret test scores to make informed, evidence-based clinical decisions.

Assessing subjective and objective information literacy at upper secondary schools - an empirical study in four German-speaking countries

Information literacy (IL) is a key 21st-century skill. This paper contributes to the body of knowledge by specifying the construct of IL. It defines IL and presents a seven-dimensional IL model. On this basis, two types of test instruments are evaluated: a performance test for objective IL (OIL) and a self-assessment instrument for subjective IL (SIL). Both instruments are validated by means of item response theory. We utilise a sample with N = 432 upper secondary students from German-speaking countries. Rasch scalability of the instruments is demonstrated. Reliability equals in the case of OIL for EAP/PV .84 and for WLE .88; in the case of SIL, .75 and .77, respectively. For OIL, we present a proficiency level model to ensure a criterion-based interpretation of test scores. In terms of instructional sensitivity, we provide evidence for a positive influence of students' grade and IL-related content in class on OIL. The correlation between OIL and SIL equals .52.

Assessment of Item Response Model-Data Fit Via Bayesian Limited Information Model Comparison Posterior Predictive Checks

Model-data fit is critical to ensure valid interpretations of test scores. Recently, emphasis has been placed on Bayesian psychometric models. Despite a surge in research on Bayesian psychometric m...

The effect of rolling walker use on interpretation of Timed Up and Go test scores: a preliminary study.

Studies have shown that the type of assistive device used during the Timed Up and Go test impacts the time it takes to complete the test. This study examines the impact of the use of a rolling walker within individuals on Timed Up and Go scores and assessment of fall risk. The Berg Balance Scale, Timed Up and Go, and linear gait speed were used to measure balance and fall risk on 13 older adults as part of their usual reassessment in preparation for discharge. Timed Up and Go and gait speed scores were compared assistive device conditions. The Berg Balance Scale was used as a benchmark fall risk assessment. Linear gait speed was similar in both conditions. Timed Up and Go times were slower for all patients using a rolling walker versus a cane or no assistive device. Consideration for the effect of assistive device use on Timed Up and Go scores has important implications as it may influence interpretation of fall risk.

Mapping the TOEFL iBT® Test Scores to China's Standards of English Language Ability: Implications for Score Interpretation and Use

The past decade has seen an emerging interest in mapping (aligning or linking) test scores to language proficiency levels of external performance scales or frameworks, such as the Common European Framework of Reference (CEFR), as well as locally developed frameworks, such as China's Standards of English Language Ability (CSE). Such alignment is ultimately a claim about the interpretation of test scores in relation to external levels of language proficiency. To support such a claim, established procedures should be carefully implemented and multiple sources of evidence should be collected. In this research report, we demonstrate the application of a series of steps in building an argument for aligning the scores of the TOEFL iBT® test, an international, large‐scale language proficiency test of English as a foreign language (EFL), to the levels of the CSE. The alignment process comprised the following steps: (a) establishing construct congruence between the TOEFL iBT test and the CSE; (b) establishing recommended minimum test scores (cut scores), set by local experts, to classify language learners into the local proficiency levels; (c) collection of scores by test takers (N = 1,326) and evaluations of the test takers' proficiency levels by their teachers, based on the local framework; and (d) consideration of the results of other alignment studies in the local context as well as the link between the CEFR and the CSE levels. We conclude with a discussion of the contextual issues that should be considered when interpreting test scores in relation to external proficiency levels. These contextual issues are important considerations because they have the potential to impact score‐based decisions on individuals and institutions. We also discuss the implications for similar alignment research.

Interpreting Test Scores for Compensatory Education Students

Interpreting Test Scores for Compensatory Education Students

Word-reading ability as a “hold test” in cognitively normal young adults with history of concussion and repetitive head impact exposure: A CARE Consortium Study

Objective: Neuropsychological evaluations include hold tests like word-reading ability as estimates of premorbid intellect thought to be resilient to the effects of neurologic insult. We tested the alternative hypothesis that exposure to concussion or repetitive subclinical head impacts throughout early life may stunt acquisition of word-reading skills.Method: Data were obtained from student–athletes within the CARE Consortium that completed the Wechsler Test of Adult Reading (WTAR). Measures of head trauma burden included self-reported concussion history and cumulative years of exposure to collision sports. We evaluated the effects of head trauma, sociodemographic (race, SES), and academic (SAT/ACT scores, learning disorder) variables on WTAR standard score using linear regression. Analyses were repeated in a football-only subsample estimating age of first exposure to football as a predictor.Results: We analyzed data from 6,598 participants (72.2% white, 39.6% female, mean ± SD age = 18.8 ± 1.2 years). Head trauma variables collectively explained 0.1% of the variance in WTAR standard scores, with years of collision sport exposure weakly predicting lower WTAR standard scores (β = .026–.035, very small effect). In contrast, sociodemographic and academic variables collectively explained 20.9–22.5% of WTAR standard score variance, with strongest effects noted for SAT/ACT scores (β = .313–.337, medium effect), LD diagnosis (β = –.115 to –.131, small effect), and SES (β = .101–.108, small effect). Age of first exposure to football did not affect WTAR scores in a football-only sample.Conclusion: Wechsler Test of Adult Reading performance appears unrelated to history of self-reported concussion(s) and/or repetitive subclinical head trauma exposure in current collegiate athletes. Sociodemographic and academic variables should be incorporated in test score interpretations for diverse populations like athletes.

A new revised Graded Naming Test and new normative data including older adults (80-97years).

The Graded Naming Test (GNT) is commonly used in clinical and research settings to assess nominal functions. However, normative data for the GNT is over 20years old and norms for the older adult population are rather limited. Hence, confounding factors such as generational familiarity due to cultural changes need to be considered when applying the currently available norms. Moreover, normative data for the older population (80-97) are virtually absent. Such factors can be powerful confounds that can lead to incorrect interpretation of test scores. We gathered data for the GNT from 326 healthy controls aged between 18 and 97years. Surprisingly, we found no decline in performance for older adults, even for the cohort aged 80-97years (N=40). In contrast, the youngest cohort (aged 18-29years) performed unexpectedly below the whole sample mean. An item-by-item analysis revealed that five test items were unfamiliar to the youngest age cohort. To account for this generational familiarity effect, we created a revised 25-item GNT. The performance of the youngest cohort on the 25-item GNT was no longer below the whole sample mean. The performance of the oldest cohort (80-97years old) on the revised GNT was characterized by a significant decline in performance for the oldest cohort with an average NART IQ. By contrast, those with a NART IQ above the average range did not show a decline. This suggests that factors such cognitive reserve arrested the decline in performance in the oldest age cohort with higher premorbid IQ.

Fitting MD analysis in an argument-based validity framework for writing assessment: Explanation and generalization inferences for the ECPE

The argument-based approach to validity (Kane, 2013) focuses on two steps: (1) making claims about the proposed interpretation and use of test scores as a coherent, interpretive argument; and (2) evaluating those claims based on theoretical and empirical evidence related to test performances and scores. This paper discusses the role of multidimensional (MD) analysis (Biber, 1988), a corpus analytic approach to examining linguistic and discourse characteristics of language performance, in an argument-based validation framework for writing assessment. We propose a set of warrants, assumptions and evidence for inferential steps pertaining to investigations of performance characteristics in language assessments. Then, we illustrate how MD analysis can provide evidence for generalization and explanation inferences through a case study that examines lexico-grammatical features of writing performances on the Examination for the Certification of Proficiency in English (ECPE).

Questionnaire validation practice: a protocol for a systematic descriptive literature review of health literacy assessments

IntroductionContemporary validity testing theory holds that validity lies in the extent to which a proposed interpretation and use of test scores is justified, the evidence for which is dependent on...

Reference values for psychoacoustic tests on Polish school children 7-10 years old.

BackgroundGood hearing is a fundamental skill that allows children to develop properly, both socially and intellectually. In contrast to defects in inner ear function, however, auditory processing disorders (APDs)–which can affect up to 2–3% of school-children–are not easily identified with basic screening programs and must be diagnosed using special tests. Although such psychoacoustic tests are available, the scores achieved depend highly on the social, cultural, and linguistic characteristics of the population, and norms must be established for each population separately. Reference values are still lacking for the Polish population, especially for children in school-age, so that practitioners must interpret test scores themselves, often intuitively or using potentially biased thresholds from other countries.Materials and methodsWe investigated a sample of 94 Polish schoolchildren with normal hearing, divided into four age groups: from 7 years-olds to 10 years-olds. All children had no speech or language development disorder, learning problem, or symptom of APD. Participants were volunteers who had previously taken part in a large screening study. The group consisted of 56 girls (60%) and 38 boys (40%) with an average age of 8.6 years (SD = 1.1). The test battery included the Duration Pattern Test (DPT), Frequency Pattern Test (FPT), Time-Compressed Speech Test (CST), and Dichotic Digit Test (DDT).ResultsThe scores on all tests increased consistently with age. The difference between each age-group for DPT, CST, and left- and right-ear DDT tests was significant (Kruskal–Wallis test, p-values = 0.002, 0.006, 0.005, 0.020, respectively), but the effect of age on the FPT test was not (p-value = 0.143). The analysis showed a clear and significant separation between a merged group of ages 7 and 8 and another of ages 9 and 10. We, therefore, propose, for each test, separate reference values for these two particular age-groups. Using thresholds based on a 10% quantile, we offer the following reference values for ages 7–8 and 9–10 respectively: DPT, 28.5% and 53.8%; FPT, 18.5% and 27.5%; CST, 68.6% and 77.2%; left-ear DDT, 34.3% and 52.5%; right-ear DDT, 56% and 72.5%.ConclusionThe scores on psychoacoustic tests to diagnose APD differ between cultures and linguistic backgrounds. Clinicians should, therefore, use norms that have been designed for the population most similar to their patients. Here, we report the use of a test battery designed for the Polish language that accounts for various aspects of APD when screening school children. Together with a full methodology of those tests, we provide norms that can be used as cut-offs in clinical diagnosis. Practitioners are invited to use them to obtain more accurate, evidence-based decisions.

Students’ perceptions of assessment quality related to their learning approaches and learning outcomes

This study aims to provide more insight into how students’ perceptions of assessment quality are related to their learning approaches and learning outcomes. Six variables associated with the construct of students’ perceptions of assessment quality are distinguished: 1) effects of assessment on learning, 2) fairness of assessment, 3) conditions of assessment, 4) interpretation of test scores, 5) authenticity of assessment, and 6) credibility of assessment. 204 higher education students completed the Students’ Perceptions of Assessment Quality Questionnaire (SPAQQ), and the Approaches to Learning and Studying Inventory (ALSI), and the students’ learning outcomes (grades) were obtained. Firstly, results indicate that the students’ perceptions of the effects of assessment on learning are positively related to the students’ deep learning approach and the strategic learning approach and negatively related to the surface learning approach. Secondly, the students’ perceptions of the conditions of assessment are positively related to their learning outcomes of the assessments.

Ethics and Fairness in Assessing Learning Outcomes in Higher Education

Fairness is one of the most important quality criteria of assessments and a necessary condition for valid test score interpretations. In this paper, we describe findings related to an assessment of N = 7664 beginning business and economics students at 46 universities across Germany using a domain-specific higher education entry test. From the perspective of test fairness as defined by the internationally established validation standards outlined by AERA et al. (Standards for Educational and Psychological Testing, AERA, Washington, DC, 2014), we identify which students had particular difficulty completing the test, taking into account gender- and language-related influence factors in the assessment of their test performance. Our results highlight the particular challenges in admissions testing in higher education. One of these challenges is finding a suitable way to address the disadvantages experienced by various groups of students to guarantee fairness and ethical integrity when developing and administering tests. Current migration developments and the overall internationalization of higher education have led to an increasing heterogeneity of student bodies. These challenges are therefore of particular relevance for education policy and practice when it comes to ensuring fairness when assessing learning outcomes for all student groups. This also applies in the context of commonly used standardized assessments and established examination practices in higher education.

Validating Test Score Interpretations Using Time Information.

A validity approach is proposed that uses processing times to collect validity evidence for the construct interpretation of test scores. The rationale of the approach is based on current research of processing times and on classical validity approaches, providing validity evidence based on relationships with other variables. Within the new approach, convergent validity evidence is obtained if a component skill, that is expected to underlie the task solution process in the target construct, positively moderates the relationship between effective speed and effective ability in the corresponding target construct. Discriminant validity evidence is provided if a component skill, that is not expected to underlie the task solution process in the target construct, does indeed not moderate the speed-ability relation in this target construct. Using data from a study that follows up the German PIAAC sample, this approach was applied to reading competence, assessed with PIAAC literacy items, and to quantitative reasoning, assessed with Number Series. As expected from theory, the effect of speed on ability in the target construct was only moderated by the respective underlying component skill, that is, word meaning activation skill as an underlying component skill of reading competence, and perceptual speed as an underlying component skill of reasoning. Accordingly, no positive interactions were found for the component skill that should not underlie the task solution process, that is, word meaning activation for reasoning and perceptual speed for reading. Furthermore, the study shows the suitability of the proposed validation approach. The use of time information in association with task results brings construct validation closer to the actual response process than widely used correlations of test scores.

Maximum effort may not be required for valid intelligence test score interpretations

Intelligence tests are assumed to require maximal effort on the part of the examinee. However, the degree to which undergraduate first-year psychology volunteers, a commonly used source of participants in low-stakes research, may be motivated to complete a battery of intelligence tests has not yet been tested. Furthermore, the assumption implies that the association between test-taking motivation and intelligence test performance is linear – an assumption untested, to date. Consequently, we administered a battery of five intelligence subtests to a sample of 219 undergraduate volunteers within the first 30 min of a low-stakes research setting. We also administered a reading comprehension test near the end of the testing session (55 min). Self-reported test-taking motivation was measured on three occasions: at the beginning (as a trait), immediately after the battery of five intelligence tests (state 1), and immediately after the reading comprehension test (state 2). Six percent of the sample was considered potentially insufficiently motivated to complete the intelligence testing, and 13% insufficiently motivated to complete the later administered reading comprehension test. Furthermore, test-taking motivation correlated positively with general intelligence test performance (r ≈ 0.20). However, the effect was non-linear such that the positive association resided entirely between the low to moderate levels of test-taking motivation. While simultaneously acknowledging the exploratory nature of this investigation, it was concluded that a moderate level of test-taking effort may be all that is necessary to produce intelligence test scores that are valid. However, in low-stakes research settings, cognitive ability testing that exceeds 25 to 30 min may be inadvisable, as test-taking motivational levels decrease to a degree that may be concerning for a non-negligible portion of the sample.