Interpretation Of Test Scores Research Articles

The development and validation of a measure of eating disorder-specific interpersonal problems: The Interpersonal Relationships in Eating Disorders (IR-ED) Scale.

Clinical reports suggest that interpersonal problems are associated with the onset and maintenance of eating pathology, but existing measures of such problems have limited links to eating pathology. Therefore, the aim of this study was to develop an eating-specific measure of interpersonal problems. The new measure, the Interpersonal Relationships in Eating Disorders scale (IR-ED), was administered to a large community sample, a nonclinical replication sample, and a clinical group of eating disorder patients. In Study 1, the psychometric properties of the IR-ED were established, and they were tested using confirmatory analyses in Study 2. Study 3 determined the validity of the test score interpretations in a clinical sample. The final 15-item version of the IR-ED demonstrated 3 distinct factors with reliability of test scores-Food-Related Isolation; Avoidance of Body Evaluation; and Food-Related Interpersonal Tension. Study 2 demonstrated that the IR-ED comprises a common Interpersonal Problems factor and a specific group factor-Avoidance of Body Evaluation. Study 3 showed that the clinical group had higher IR-ED scores than a nonclinical group. Across the studies, Avoidance of Body Evaluation was the strongest correlate of eating pathology in this group. The IR-ED has strong psychometric properties and its test scores appear to be more valid than those of a generic measure of interpersonal problems. Avoidance of Body Evaluation is the strongest facet of such interpersonal problems, and has meaningful links to models of eating psychopathology. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Assessing interactional skills in a paired speaking test

The operationalization of interactional competence (IC) within the paired speaking test format allows for a range of interactional skills to be tested. However, in terms of assessment, challenges are posed with regard to the co-constructed nature of IC, making investigations into raters’ perceptions of the construct essential to inform test score interpretation. This qualitative study explores features of IC that raters attended to as they evaluated performances in a paired speaking test, part of a Swedish national test of English as a Foreign Language (EFL). Two groups of raters, 17 EFL teachers from Sweden, using national standards based on the Common European Framework of Reference for Languages (CEFR), and 14 raters from Finland and Spain, using CEFR scales, rated six audio-recorded paired performances, and provided written comments to explain their scores and account for salient features. The findings of the content analysis indicate that raters attended to three main interactional resources: topic development moves, turn-taking management, and interactive listening strategies. As part of the decision-making process, raters also considered the impact of test-takers’ interactional roles and how candidates’ performances were interrelated. In the paper, interaction strategies that were perceived as more or less successful by raters are highlighted. The findings have implications for our understanding of raters’ operationalization of IC in the context of paired speaking tests, and for the development of rating scales and guidelines that reflect the social dimensions of the construct.

Comparison of Two Approaches to Interpretive Use Arguments

ABSTRACTThe Standards for Educational and Psychological Testing recommend an argument-based approach to validation that involves a clear statement of the intended interpretation and use of test scores, the identification of the underlying assumptions and inferences in that statement—termed the interpretation/use argument, and gathering of evidence to support or refute the assumptions and inferences. We present two approaches to articulating the interpretation/use argument. One approach uses the five sources of validity evidence in the Standards for Educational and Psychological Testing as a framework and the other approach uses Kane’s chain of assumptions/inferences as a framework. Through this process we identified aspects of these approaches that need to be further clarified for instrument developers to consistently implement either approach, identified important differences in the perspective each approach takes on validation, and highlight important questions for the measurement and mathematics education research fields to consider.

Domain Description: Validating the Interpretation of the TOEFL iBT® Speaking Scores for International Teaching Assistant Screening and Certification Purposes

In the past 2 decades, there has been an increasing tendency to use scores from the TOEFL iBT® Speaking test for decisions regarding the certification of international graduate students as teaching assistants at North American universities. To obtain validity evidence in support of the usefulness of the speaking scores for this secondary use of the test, this study adopted the argument‐based approach to validation. Focusing on the domain description inference in the TOEFL interpretive argument, the study investigated whether the language functions elicited by TOEFL iBT Speaking tasks can be identified in authentic discourse produced by international teaching assistants (ITAs) with different instructional roles. We compiled and analyzed 2 corpora—a TOEFL iBT speech corpus and an ITA speech corpus. The TOEFL corpus contained 2,738 responses to integrated and independent tasks. The ITA corpus of 119 spoken texts included multiple disciplines and 3 instructional genres: lab, recitation, and lecture. The 2 corpora were manually annotated using the knowledge framework (Mohan, 1986), which is a heuristic in systemic functional linguistics (SFL) used to identify knowledge structures (KSs) and language functions based on how linguistic choices function in the discourse. Then, the following types of data were quantitatively analyzed: discourse units annotated per KS category, discourse units annotated per language function, KS categories occurring in each text, language functions occurring in each text, and KSs and functions from each component of the spoken corpora. The corpus data revealed how the language functions were realized and how they varied. Overall, the results indicated that TOEFL iBT Speaking tasks elicit most of the language functions identified in ITA discourse, suggesting that this test accounts for the functional language ability necessary for effective instructional performance as a teaching assistant in the target domain of language use. The discrepancy detected in the use of some functions pertaining to 2 KSs warrants further examination of the extent to which it may impact secondary test use and score interpretation.

Sensitivity of test items to teaching quality

Instructional sensitivity is the psychometric capacity of tests or single items of capturing effects of classroom instruction. Yet, current item sensitivity measures’ relationship to (a) actual instruction and (b) overall test sensitivity is rather unclear. The present study aims at closing these gaps by investigating test and item sensitivity to teaching quality, reanalyzing data from a quasi-experimental intervention study in primary school science education (1026 students, 53 classes, Mage = 8.79 years, SDage = 0.49, 50% female). We examine (a) the correlation of item sensitivity measures and the potential for cognitive activation in class and (b) consequences for test score interpretation when assembling tests from items varying in their degree of sensitivity to cognitive activation. Our study (a) provides validity evidence that item sensitivity measures may be related to actual classroom instruction and (b) points out that inferences on teaching drawn from test scores may vary due to test composition.

Expert-Based Cognitive Model and Student-Based Cognitive Model in the Learning of “Time”: Match or Mismatch?

Cognitive model [CM] is the description of individual’s problem-solving method on selected tasks at a precise level to make meaningful inference on his/her strengths and weaknesses. CM serves as a foundation in developing a cognitive diagnostic assessment [CDA] that links the interpretation of test score to the attributes. This study employed the CDA involving “Time” which was developed under the framework of assessment triangle and Attribute Hierarchy Method [AHM]. In this study, the expert-based cognitive model was generated by 7 experienced primary mathematics teachers while the student-based cognitive model was generated based on 259 Year 6 pupils’ responses in the CDA. The consistency between these cognitive models was calculated by using Hierarchical Consistency Index [HCI] and triangulated by pupils’ interview transcript. Hence, this paper aims to discuss the consistency of structure between the expert-based cognitive models and the student-based cognitive models. The extent of match between the expert-based and the student-based cognitive models is important as it determines the validity of the expert-based cognitive model from students’ perspective. In general, all the 3 expert-based cognitive models were excellently fit with the student-based cognitive models. However, the result shows that the cognitive models of those pupils who scored 50% and below exhibited poor fit to the expert-based cognitive models. Results of the study implied that the proposed CMs are able to predict pupils’ responses and there is a need to construct CM which is suitable for low ability pupils in order to help them in studying the topic of “Time.”

The key-features approach to assess clinical decisions: validity evidence to date.

The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing's five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability. Of the 457 publications that referred to the KFs approach between 1984 and October 2017, 164 are cited here; the remaining 293 were either redundant or the authors simply mentioned the KFs concept in relation to their work. While one set of articles reported meeting the validity standards, another set examined KFs test development choices and score interpretation. The accumulated validity evidence for the KFs approach since its inception supports the decision-making construct measured and its use to assess clinical decision-making skills at all levels of training and practice and with various types of exam formats. Recognizing that gathering validity evidence is an ongoing process, areas with limited evidence, such as item factor analyses or consequences of testing, are identified as well as new topics needing further clarification, such as the use of the KFs approach for formative assessment and its place within a program of assessment.

Preliminary normative data on the BORB for children aged 3–8

ABSTRACTThe Birmingham Object Recognition Battery (BORB) is a theoretically based test battery that is used in adult cognitive neuropsychology in research and for clinical assessment. It allows a detailed analysis of underlying impairments in individuals with brain injury who have visual object recognition difficulties. The BORB’s usefulness in pediatrics is supported by numerous research studies. However, there is no published normative data for children, making clinical use of the test difficult. The aim of this brief report is to publish some preliminary normative data in 70 children aged between 3 and 8 years to assist both researchers and clinicians with interpretation of test scores. Results indicate that children’s performance on individual BORB subtests varies according to task demands and age. For some subtests there is improvement in performance with increasing age. However, very young children (age 3-4 years) perform at adult levels on some subtests, or alternatively on other subtests they perform at the level of chance. The current paper supports the need for pediatric data for the BORB due to large normal individual variation in performance and varying age-related performance on individual BORB subtests.

Confirmatory Factor Analysis of Neurocognitive Measures in Healthy Young Adults: The Relation of Executive Functions with Other Neurocognitive Functions.

The present study aimed to investigate the factor structure of a set of neurocognitive tests theoretically assessing executive functions (EF), verbal abilities (VA), and processing speed (PS). This study extended previous research by analyzing if each test is better explained by the specific factor to which it theoretically belongs or by a more general neurocognitive factor; and also by analyzing the relations between the neurocognitive factors. Using confirmatory factor analysis (CFA) we examined the factor structure of nine neurocognitive tests (EF: Working Memory, Tower, Divided Attention, Stroop, and Verbal Fluency tests; VA: Word List and Confrontation Naming tests; PS: Coding and Telephone Search tests) in a nonclinical sample (N = 90; 18-33 years old, 76 women). We tested five factor models of neurocognitive functioning: a one-factor model; two models with two-correlated factors; and two models with three-correlated factors. A three-correlated-factor model, with EF, VA, and PS factors, was the most suitable for our neuropsychological data. The Verbal Fluency test was better explained by the VA factor rather than by the EF factor. The EF factor was correlated with the PS factor, but not with the VA factor. Most of the neurocognitive measures used in the present study loaded in the expected factors (with the exception of the Verbal Fluency that was apparently more related to VA). EF and PS represent related but separable functions. Our results highlight the need for a careful interpretation of test scores since performance on one test usually requires multiple functions.

Patient-reported outcome measures in Huntington disease: Quality of life in neurological disorders (Neuro-QoL) social functioning measures.

Social functioning is an essential but poorly understood component of health-related quality of life (HRQOL) for people with Huntington disease (HD). We report on the psychometric properties of 2 Neuro-QoL patient-reported outcome measures to assess social functioning in HD. Persons with prodromal (n = 198) or manifest HD (n = 195 early and n = 117 late) completed Neuro-QoL Ability to Participate in Social Roles and Activities, and Satisfaction with Social Roles and Activities. Items from 2 generic HRQOL patient-reported outcome measures were used to create a social functioning composite score; items from the Unified Huntington's Disease Rating Scale and Problem Behaviors Assessment Scale were used to create a clinician-rated composite score of social function. Internal consistencies for the scores on the Neuro-QoL measures were excellent (> .88). Computer adaptive test administration had some advantages over computer-administered static Short Forms. Validity was supported by significant associations between the scores on the Neuro-QoL measures and other self- and clinician-reports of social function. Individuals with prodromal HD had better social functioning than the manifest HD groups; individuals with late-HD had less satisfaction and ability to participate in social roles and activities than the other 2 groups. Neuro-QoL provides brief, reliable scores of social functioning that measure ability to participate in, and satisfaction with, social roles and activities in persons with prodromal and manifest HD. In addition, test score interpretations of these measures support their validity in people with prodromal and manifest HD. These measurement tools add breadth to treatment outcome measures in HD and can increase understanding of the social implications of living with HD. (PsycINFO Database Record

Theoretically-Consistent Cognitive Ability Test Development and Score Interpretation

Clinical cognitive ability assessment—and its corollary, score interpretation—are in a state of disarray. Many current instruments are designed to provide a bevy of scores to appeal to a variety of school psychologists. These scores are not all grounded in the attribute’s theory or developed from sound measurement or psychometric theory. Thus, for a given instrument, there can be substantial variation between school psychologists when interpreting scores from the same instrument. This is contrary to the very purpose of psychological assessment. As a contrast, we provide a sketch of theoretically driven test development and score interpretation. In addition, we provide examples of how this could be implemented using two theories of intelligence (Spearman’s two-factor and Cattell and Horn’s Gf-Gc) and measurement theory about the nature of psychological test scores. While different from what is often implemented by school psychologists, it is consistent with the guiding principles of evidence-based psychological assessment.

Validation of sub-constructs in reading comprehension tests using teachers’ classification of cognitive targets

ABSTRACTReading comprehension is often treated as a multidimensional construct. In many reading tests, items are distributed over reading process categories to represent the subskills expected to constitute comprehension. This study explores (a) the extent to which specified subskills of reading comprehension tests are conceptually conceivable to teachers, who score and use national reading test results and (b) the extent to which teachers agree on how to locate and define item difficulty in terms of expected text comprehension. Eleven teachers of Swedish were asked to classify items from a national reading test in Sweden by process categories similar to the categories used in the PIRLS reading test. They were also asked to describe the type of comprehension necessary for solving the items. Findings of the study suggest that the reliability of item classification is limited and that teachers’ perception of item difficulty is diverse. Although the data set in the study is limited, the findings indicate, in line with recent validity theory, that the division of reading comprehension into subskills by cognitive process level will require further validity evidence and should be treated with caution. Implications for the interpretation of test scores and for test development are discussed.

Application of validity theory and methodology to patient-reported outcome measures (PROMs): building an argument for validity

BackgroundData from subjective patient-reported outcome measures (PROMs) are now being used in the health sector to make or support decisions about individuals, groups and populations. Contemporary validity theorists define validity not as a statistical property of the test but as the extent to which empirical evidence supports the interpretation of test scores for an intended use. However, validity testing theory and methodology are rarely evident in the PROM validation literature. Application of this theory and methodology would provide structure for comprehensive validation planning to support improved PROM development and sound arguments for the validity of PROM score interpretation and use in each new context.ObjectiveThis paper proposes the application of contemporary validity theory and methodology to PROM validity testing.Illustrative exampleThe validity testing principles will be applied to a hypothetical case study with a focus on the interpretation and use of scores from a translated PROM that measures health literacy (the Health Literacy Questionnaire or HLQ).DiscussionAlthough robust psychometric properties of a PROM are a pre-condition to its use, a PROM’s validity lies in the sound argument that a network of empirical evidence supports the intended interpretation and use of PROM scores for decision making in a particular context. The health sector is yet to apply contemporary theory and methodology to PROM development and validation. The theoretical and methodological processes in this paper are offered as an advancement of the theory and practice of PROM validity testing in the health sector.

Construct validity of the G-CPAQ and its mediating role in pain interference and adjustment.

The Chronic Pain Acceptance Questionnaire (CPAQ) is a measure of pain acceptance comprised of pain willingness (PW) and activity engagement (AE; McCracken et al., 2004). Concerns about the factorial structure of the CPAQ exist, as it is not yet clear whether PW and AE constitute 2 independent constructs or 1, pain acceptance. Concerns also exist about the internal and predictive validity of test score interpretations of this measure. This study also presents that the choice of predictor variables has contributed to theoretical confusion regarding the impact of pain acceptance on pain-related adjustment. The purpose of this study was: (a) to examine the psychometric properties of both the long (20 items) and short (8 items) versions of the Greek-Chronic Pain Acceptance Questionnaire (G-CPAQ); (b) to examine the utility of a 2-factor solution in predicting psychosocial adjustment to pain using confirmatory factor analysis; and (c) to explore the mediating effects of pain acceptance and cognitive defusion, comprising the "open" response style to pain, between pain interference and pain related outcomes. One hundred and sixty chronic pain patients completed a questionnaire packet including pain indexes, pain acceptance, cognitive fusion, avoidance, and emotional distress. Confirmatory factor analyses supported the 2-factor solution, though a general good model fit was achieved only for the short G-CPAQ version. Structural equation modeling showed that PW and AE coupled with cognitive defusion partially mediated the influence of pain interference on pain severity, emotional distress, and avoidance of pain. (PsycINFO Database Record

Differential Item Functioning: Beyond validity evidence based on internal structure.

In the latest release of the Standards for Educational and Psychological Testing, Differential Item Functioning (DIF) is considered as validity evidence based on internal structure. However, there are no indications of how to design a DIF study as a validation study. In this paper, we propose relating DIF to all sources of validity evidence, and provide a general conceptual framework for transforming “typical” DIF studies into validation studies. We perform a comprehensive review of the literature and make theoretical and practical proposals. The article provides arguments in favour of addressing DIF detection and interpretation as validation studies, and suggestions for conducting DIF validation studies. The combination of quantitative and qualitative data within a mixed methods research perspective, along with planning DIF studies as validation studies, can help improve the validity of test score interpretations.

Measurement of expressive vocabulary in school-age children: Development and application of the Kilifi Naming Test (KNT)

ABSTRACTThe dearth of locally developed measures of language makes it difficult to detect language and communication problems among school-age children in sub-Saharan African settings. We sought to describe variability in vocabulary acquisition as an important element of global cognitive functioning. Our primary aims were to establish the psychometric properties of an expressive vocabulary measure, examine sources of variability, and investigate the measure’s associations with non-verbal reasoning and educational achievement. The study included 308 boys and girls living in a predominantly rural district in Kenya. The developed measure, the Kilifi Naming Test (KNT), had excellent reliability and acceptable convergent validity. However, concurrent validity was not adequately demonstrated. In the final regression model, significant effects of schooling and area of residence were recorded. Contextual factors should be taken into account in the interpretation of test scores. There is need for future studies to explore the concurrent validity of the KNT further.

Evaluating Evidence Regarding Relationships With Criteria

ABSTRACTAn overview of standards related to demonstrating evidence regarding relationships with criteria as it pertains to instrument development was presented, along with heuristic examples. Additional measures and a comprehensive design are necessary to establish evidence related to the use and interpretation of test scores for the validation of a measure.

Validity and participation: implications for school comparison of Australia’s National Assessment Program

The National Assessment Program – Literacy and Numeracy (NAPLAN) in Australia is a series of literacy and numeracy tests that are used for purposes of school comparison. This paper argues that a key question for this use lies in whether or not this is a reasonable, or valid, use of the test data. Using Kane’s argumentative approach to validity, this paper argues that the comparisons of the quality of student achievement made available on the My School Website have low validity due to the lack of regard to rates of participation in schools. In bringing together the literature that addresses the ‘new governance’ of education through testing and an approach to validity that addresses the technical aspects of test score interpretation, with the ethics of how test scores are used and applied, this study identifies validity as an important consideration in comparative analyses of student achievement data. The identification of the need to consider participation in such comparisons through the application of the argumentative approach to validity highlights the contribution of this article not only to the testing field but also to critical policy literature.

Does Test Item Performance Increase with Test-to-Standards Alignment?

ABSTRACTVariation in test performance among examinees from different regions or national jurisdictions is often partially attributed to differences in the degree of content correspondence between local school or training program curricula, and the test of interest. This posited relationship between test-curriculum correspondence, or “alignment,” and test performance is usually inferred from highly distal evidence, rather than directly examined. Utilizing mathematics standards content analysis data and achievement test item data from ten U.S. states, we examine the relationship between topic-specific alignment and test item performance. When a particular item’s content type is emphasized by the standards, we find evidence of a positive relationship between the alignment measure and proportion-correct test item difficulty, although this effect is not consistent across samples. Implications of the results for curricular achievement test development and score interpretation are discussed.

Advanced clinical interpretation of the Delis-Kaplan Executive Function System: multivariate base rates of low scores

Objective: Multivariate base rates allow for the simultaneous statistical interpretation of multiple test scores, quantifying the normal frequency of low scores on a test battery. This study provides multivariate base rates for the Delis-Kaplan Executive Function System (D-KEFS). Method: The D-KEFS consists of 9 tests with 16 Total Achievement scores (i.e. primary indicators of executive function ability). Stratified by education and intelligence, multivariate base rates were derived for the full D-KEFS and an abbreviated four-test battery (i.e. Trail Making, Color-Word Interference, Verbal Fluency, and Tower Test) using the adult portion of the normative sample (ages 16–89). Results: Multivariate base rates are provided for the full and four-test D-KEFS batteries, calculated using five low score cutoffs (i.e. ≤25th, 16th, 9th, 5th, and 2nd percentiles). Low scores occurred commonly among the D-KEFS normative sample, with 82.6 and 71.8% of participants obtaining at least one score ≤16th percentile for the full and four-test batteries, respectively. Intelligence and education were inversely related to low score frequency. Conclusions: The base rates provided herein allow clinicians to interpret multiple D-KEFS scores simultaneously for the full D-KEFS and an abbreviated battery of commonly administered tests. The use of these base rates will support clinicians when differentiating between normal variations in cognitive performance and true executive function deficits.