Construct-irrelevant Variance Research Articles

Two Approaches for Identifying Low-Motivated Students in a Low-Stakes Assessment Context

Many universities rely on data gathered from tests that are low stakes for examinees but high stakes for the various programs being assessed. Given the lack of consequences associated with many collegiate assessments, the construct-irrelevant variance introduced by unmotivated students is potentially a serious threat to the validity of the inferences that institutions can make from their assessments. Two approaches to evaluating examinee motivation are discussed in this article: a global paper-and-pencil self-report measure of students' motivation across all tests completed during the course of a testing session, and a computer-based method that non-intrusively measures the amount of time students spend on each item in a test. This study presents evidence that the two motivation filtering methods provide similar filtered aggregate test scores, although more data was removed using the global paper-and-pencil self-report technique. Consequently, those interested in motivation filtering may not need to employ computer-based testing techniques but might instead effectively filter data from unmotivated students using self-report measures.

Consistency of report card grades and external assessments in a Canadian province

The study investigated how well report card grades communicate to students and parents that state educational standards are being met, standards that are objectively measured by infrequently administered mandated assessments. Data sources were report card grades and external assessment scores for 2006–09 for Ontario Canada. The information that parents and students received about student performance from report cards and external assessments were similar (rs = .47) to the r = .40–.60 range previously reported. Teachers assigned higher grades than external assessments warranted, even after a major source of construct irrelevant variance in report card grades (teacher ratings on multiple scales measuring student effort and school commitment) was controlled. The relationship of grades to assessment scores was robust across genders, school district types (Public versus Catholic) and language (English and French). Agreement of assessments was higher for grade 6 than for grade 3 and for Writing than for Reading or Mathematics. Report cards provided information about students’ future achievement that was accurate and delivered up to 2 years prior to the administration of external assessments. Seventy to 80% of students who reached the provincial achievement standard on one or both prior report cards were successful on the subsequent external assessment, compared to 30–50% of students who failed to meet the report card standard at least once.

The Second Time Around: Accounting for Retest Effects on Oral Examinations

Years of research with high-stakes written tests indicates that although repeat examinees typically experience score gains between their first and subsequent attempts, their pass rates remain considerably lower than pass rates for first-time examinees. This outcome is consistent with expectations. Comparable studies of the performance of repeat examinees on oral examinations are lacking. The current research evaluated pass rates for more than 50,000 examinees on written and oral exams administered by six medical specialty boards for several recent years. Pass rates for first-time examinees were similar for both written and oral exams, averaging about 84% across all boards. Pass rates for repeat examinees on written exams were expectedly lower, ranging from 22% to 51%, with an average of 36%. However, pass rates for repeat examinees on oral exams were markedly higher than for written exams, ranging from 53% to 77%, with an average of 65%. Four explanations for the elevated repeat pass rates on oral exams are proposed, including an increase in examinee proficiency, construct-irrelevant variance, measurement error (score unreliability), and memorization of test content. Simulated data are used to demonstrate that roughly one third of the score increase can be explained by measurement error alone. The authors suggest that a substantial portion of the score increase can also likely be attributed to construct-irrelevant variance. Results are discussed in terms of their implications for making pass-fail decisions when retesting is allowed. The article concludes by identifying areas for future research.

Aspects of performance on line graph description tasks: influenced by graph familiarity and different task features

Motivated by cognitive theories of graph comprehension, this study systematically manipulated characteristics of a line graph description task in a speaking test in ways to mitigate the influence of graph familiarity, a potential source of construct-irrelevant variance. It extends Xi (2005), which found that the differences in holistic scores on graph tasks with varying characteristics, although significant, were small. Using an analytic scoring approach, it re-examined how visual chunks in line graphs and planning time influenced some specific components of examinees’ performance on line graph description tasks. The analytic dimensions examined were determined based on results of previous studies and hypotheses about the relationships among visual chunks, planning, graph familiarity, and features of elicited discourse. It was found that participants less familiar with graphs described the line graphs in a less organized fashion and that their descriptions were weaker in content. Graph familiarity thus introduced construct-irrelevant variance. However, providing planning time and using less complex graphical displays improved the fluency, organization and content of the elicited oral discourse and helped lessen the influence of graph familiarity, thus enhancing the validity of this task. The theoretical and practical implications of the findings are discussed.

Methods for Evaluating the Validity of Test Scores for English Language Learners

In the United States, when English language learners (ELLs) are tested, they are usually tested in English and their limited English proficiency is a potential cause of construct-irrelevant variance. When such irrelevancies affect test scores, inaccurate interpretations of ELLs' knowledge, skills, and abilities may occur. In this article, we review validity issues relevant to the educational assessment of ELLs and discuss methods that can be used to evaluate the degree to which interpretations of their test scores are valid. Our discussion is organized using the five sources of validity evidence promulgated by the Standards for Educational and Psychological Testing. Technical details for some validation methods are provided. When evaluating the validity of a test for ELLs, the evaluation methods should be selected so that the evidence gathered specifically addresses appropriate test use. Such evaluations should be comprehensive and based on multiple sources of validity evidence.

Writing Differences in Teacher Performance Assessments: An Investigation of African American Language and Edited American English

Differential performance results occur when a specific population subgroup achieves a passing rate which is significantly lower than that of the normative reference group. African Americans do less well, in general, on all types of assessments, including constructed-response tests. The present study examined the writing styles of African American test takers in a National Board for Professional Teaching Standards (NBPTS) portfolio entry from the Middle Childhood/Generalist Certificate, as compared with the writing styles of European American test takers. The use of features other than Edited American English was examined as a possible source of construct-irrelevant variance in assessors’ scoring judgments. Thirty-two written commentaries, 18 from African American candidates and 14 from European American candidates, were coded for grammatical, lexical, and discourse features. The coding frame identified features of African American Language (AAL) and Speech Code Errors (SCE). Instances of AAL were fewer than instances of SCE and clustered according to potential users of AAL.

Objective structured clinical examinations as an assessment method in residency training: practical considerations.

Objective structured clinical examinations as an assessment method in residency training: practical considerations.

Speaking of fairness in testing.

In an article based on her presentation to the 114th Annual Convention of the American Psychological Association, Janet Helms described a concept of test fairness. Helms's approach to fairness appears to be based on the premise that socialization consists of experiences received from one's environment, which are simply internalized as personal attributes. From an ecological perspective, this is incorrect, misleading, and oddly evocative of social-cognitive and psychodynamic notions. Individuals are not passive recipients of culture, and they do not necessarily simply reflect their experiences. Although Helms intended her model to apply to racial and cultural groups, it seems that she has overlooked the potential of applying her concept of fairness to another important level of social organization: the family. R. J. Griffore suggests that it is unfair to avoid using valid tests that are said to be unfair because of racially or culturally based construct-irrelevant variance if those tests can help individuals make decisions that lead to success outcomes. The use of race or culture as a variable that in any way jeopardizes the prediction of success for individuals or groups is unfair for all groups for whom the test is used.

Investigating the Validity of Math Word Problems for English Language Learners with DIF

When testing English language learners (ELLs) in subject matter areas, construct irrelevant variance could result from English, the language in which the test is presented. Differential item functioning (DIF) techniques have been used to determine if items are operating differently for population subgroups and might therefore be appropriate for such an investigation. In the study presented here, ELL and non-ELL 8th-grade students' scores on National Assessment of Educational Progress (NAEP) math word problems were compared. After the non-ELLs were found to outperform the ELLs, two independent techniques for detecting DIF—an item response theory approach and the Mantel-Haenszel approach—were employed to determine if a plausible explanation for the score difference could be attributed to DIF against ELLs. The results showed that only 1 of 10 original NAEP items and none of 10 linguistically simplified items were found to exhibit DIF against ELLs, suggesting that the score difference between the groups could not be attributed to DIF against ELLs.

Validation of a test screen design

Other than Fulcher (2003), there is little published literature in language testing on valid interface design of computerized language tests. This study describes how a web-based speaking test is validated in terms of good interface design. Although the validation process regarding computerized tests has recently become an important emerging issue in language testing (Chapelle, 2001), it has not been dealt with in mainstream validity arguments. Test interface design is an important issue for language testers who attempt to avoid any construct-irrelevant variance in language tests. When developing a web-based speaking test at an American university, a test administration, post-survey and, interview were carried out with 43 graduate ESL students to validate the test interface design. Factors regarding good interface design were defined and evaluated through analyzing the survey and interview data. Then, correlation coefficients between the factors and the test results were produced and the relationships were examined. Significant correlations were not found between certain factors studies dealing with test results, while other factors reveal significant association among sub-factors.

Assessment of English- and Spanish-speaking Students With the WISC-III and Leiter-R

The Leiter International Performance Scale-Revised (Leiter-R) and Wechsler Intelligence Scale for Children-Third Edition (WISC-III) scores of 47 English- and 47 Spanish-speaking students were analyzed, and the effects of English language ability on these scores were examined. Leiter-R validity was supported for both language groups. WISC-III validity was supported for English speakers but contained substantial construct irrelevant variance for Spanish speakers as a function of English proficiency as measured by the Woodcock Language Proficiency Battery-Revised Oral Language Scale (OLS). The WISC-III appeared to measure intelligence validly for Spanish speakers scoring ≥ 90 on the OLS but seemed to reflect linguistic ability for Spanish speakers scoring < 90. Because of similarities between the WISCIII and the newer WISC-IV index scores, the OLS criterion of ≥ 90 is suggested as a reasonable guideline for valid administration of the WISC-IV to Spanish speakers.

Fairness is not validity or cultural bias in racial-group assessment: A quantitative perspective.

When test scores that differ by racial groups are used for assessment purposes, resulting decisions regarding members of the lower scoring group are potentially unfair. Fairness is defined as the removal from test scores of systematic variance attributable to experiences of racial or cultural socialization, and it is differentiated from test-score validity and cultural bias. Two fairness models for identifying, quantifying, and removing from test scores construct-irrelevant variance attributable to racial or cultural psychological attributes are presented. ((c) 2006 APA, all rights reserved).

Psychometric Issues in the ELL Assessment and Special Education Eligibility

Assessments in English that are constructed for native English speakers may not provide valid inferences about the achievement of English language learners (ELLs). The linguistic complexity of the test items that are not related to the content of the assessment may increase the measurement error, thus reducing the reliability of the assessment. Language factors that are not relevant to the content being assessed may also be a source of construct-irrelevant variance and negatively impact the validity of the assessment. More important, the results of these tests used as the criteria for identification and classification of ELL students, particularly those at the lower end of the English proficiency spectrum, may be misleading. Caution must be exercised when the results of these tests are used for special education eligibility, particularly in placing ELL students with lower English language proficiency in the learning/reading disability category. This article discusses psychometric issues in the assessment of English language learners and examines the validity of classifying ELL students, with a focus on the possibility of misclassifying ELL students as students with learning disabilities.

Psychometric Issues in the ELL Assessment and Special Education Eligibility

Assessments in English that are constructed for native English speakers may not provide valid inferences about the achievement of English language learners (ELLs). The linguistic complexity of the test items that are not related to the content of the assessment may increase the measurement error, thus reducing the reliability of the assessment. Language factors that are not relevant to the content being assessed may also be a source of construct-irrelevant variance and negatively impact the validity of the assessment. More important, the results of these tests used as the criteria for identification and classification of ELL students, particularly those at the lower end of the English proficiency spectrum, may be misleading. Caution must be exercised when the results of these tests are used for special education eligibility, particularly in placing ELL students with lower English language proficiency in the learning/reading disability category. This article discusses psychometric issues in the assessment of English language learners and examines the validity of classifying ELL students, with a focus on the possibility of misclassifying ELL students as students with learning disabilities.

Assessing the Dimensionality and Factor Structure of Multiple-Choice Exams

Examination of a test's internal structure can be used to identify what domains or dimensions are being measured, identify relationships between the dimensions, provide evidence for hypothesized multidimensionality and test score interpretations, and identify construct-irrelevant variance. The purpose of this research is to provide a methodological comparison of three methods for assessing the dimensionality and factor structure for binary scored exams in which guessing may be operating. The factor analytic methods in Mplus, NOHARM, and TESTFACT are compared in the context of analyzing the Multistate Bar Examination, a multiple-choice exam reflecting a number of complex integrated content domains. Results indicate that the methods support similar factor solutions. However, support for solutions with more factors and more indicators per factor is only obtained when guessing is modeled. The implications for evaluating the internal structure of multiple-choice tests are discussed.

USING DIF DISSECTION METHOD TO ASSESS EFFECTS OF ITEM DELETION

Statistical procedures for detecting differential item functioning (DIF) are often used as an initial step to screen items for construct irrelevant variance. This research applies a DIF dissection method and a two-way classification scheme to SAT Reasoning Test™ verbal section data and explores the effects of deleting sizable DIF items on reported scores after re-equating. The DIF dissection approach and the two-way classification scheme may yield new and detailed insight into item functioning at the subgroup level. Two hypotheses are studied: (1) whether or not the deletion of a sizeable DIF item that is the most disadvantageous to a particular subgroup will affect the scores for that subgroup the most; and (2) whether or not the effects of item deletion on scores can be predicted by the standardization method. Both hypotheses are supported by the results of this research.

Do visual chunks and planning impact performance on the graph description task in the SPEAK exam?

This study examines how task characteristics (the number of visual chunks and the amount of planning time) and test-taker characteristics (graph familiarity) influence the perceptual and cognitive processes involved in graph comprehension, the strategies used in describing graphs, and the scores obtained on the graph description task in a semi-direct oral test. Specifically, it investigates whether providing planning time and reducing the number of visual chunks in bar and line graphs help mitigate the influence of graph familiarity on language performance on graph description tasks. These research questions were explored using a structural equation modeling approach in a framework of the influence of the interaction between task characteristics and test-taker characteristics on task performance. It was found that graph familiarity affected the overall communicative quality of participants’ performance on these graph description tasks. Graph familiarity thus represented a potential source of construct-irrelevant variance. However, reducing the number of visual chunks in a graph and providing planning time positively impacted the perceptual and cognitive processes involved in graph comprehension and helped mitigate the influence of graph familiarity. Practical and theoretical implications of this study are discussed.

The Effects of Violating Standard Item Writing Principles on Tests and Students: The Consequences of Using Flawed Test Items on Achievement Examinations in Medical Education

The purpose of this research was to study the effects of violations of standard multiple-choice item writing principles on test characteristics, student scores, and pass-fail outcomes. Four basic science examinations, administered to year-one and year-two medical students, were randomly selected for study. Test items were classified as either standard or flawed by three independent raters, blinded to all item performance data. Flawed test questions violated one or more standard principles of effective item writing. Thirty-six to sixty-five percent of the items on the four tests were flawed. Flawed items were 0-15 percentage points more difficult than standard items measuring the same construct. Over all four examinations, 646 (53%) students passed the standard items while 575 (47%) passed the flawed items. The median passing rate difference between flawed and standard items was 3.5 percentage points, but ranged from -1 to 35 percentage points. Item flaws had little effect on test score reliability or other psychometric quality indices. Results showed that flawed multiple-choice test items, which violate well established and evidence-based principles of effective item writing, disadvantage some medical students. Item flaws introduce the systematic error of construct-irrelevant variance to assessments, thereby reducing the validity evidence for examinations and penalizing some examinees.

Assessing the interpersonal competence of beginning teachers: The quality of the judgement process

A procedure to assess the interpersonal competence of teachers was developed and validated. The procedure includes the use of the Questionnaire on Teacher Interaction (QTI), an observation instrument and a self-evaluation instrument. The concepts of construct-irrelevant variance and construct under-representation were used to explore the validity of the assessment procedure with regard to the noting of evidence and combining of evidence to attain an overall judgement by four separate assessors for the same beginning teacher. The validity of the assessment procedure was found to be satisfactory and the use of multiple assessment instruments to clearly have added value.

Construct‐Irrelevant Variance in High‐Stakes Testing

There are many threats to validity in high‐stakes achievement testing. One major threat is construct‐irrelevant variance (CIV). This article defines CIV in the context of the contemporary, unitary view of validity and presents logical arguments, hypotheses, and documentation for a variety of CIV sources that commonly threaten interpretations of test scores. A more thorough study of CIV is recommended.