Source Of Construct-irrelevant Variance Research Articles

The Impact of Non-Effortful Responding on Item and Person Parameters in Item-Pool Scaling Linking

ABSTRACT Low test-taking effort of examinees has been considered a source of construct-irrelevant variance in item response modeling, leading to serious consequences on parameter estimation. This study aims to investigate how non-effortful response (NER) influences the estimation of item and person parameters in item-pool scale linking (IPSL) and whether differential NER for two groups has a different effect on parameter estimation under the equivalent and non-equivalent group designs. Three simulation studies and an empirical study were conducted to answer these research questions. Results show that NER on items in a certain item set (e.g. common new items, exclusive new items) have a dominant negative effect on parameter estimation of items from the same item set, while NER on items from the item pool would seriously impact the accuracy of ability parameters. Discussions and recommendations for item set orders are provided.

Examining successful and unsuccessful time management through process data: two novel indicators of test-taking behaviors

Test-taking behavior is a potential source of construct irrelevant variance for test scores in international large-scale assessments where test-taking effort, motivation, and behaviors in general tend to be confounded with test scores. In an attempt to disentangle this relationship and gain further insight into examinees’ test-taking processes, researchers can now utilize process and timing data to obtain a more comprehensive view of test-taking behaviors, such as test-taking effort. The purpose of this study is to propose and evaluate two novel response-based, standardized indicators of test-taking behaviors that utilize a combination of examinee response and process (timing) data to better understand and describe test-taking effort in ILSAs. These indices were empirically estimated with USA data from two booklets from e-TIMSS 2019 in mathematics for grade 4. In addition, their predictive validity was examined with respect to achievement estimates. Their network of associations with other relevant variables such as motivation, interest in the subject, as well as across subjects were also examined to test their intra-individual stability in e-TIMSS.

Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

The linguistic complexity of many text-based tests can be a source of construct-irrelevant variance, as test-takers’ performance may be affected by factors that are beyond the focus of the assessment itself, such as reading comprehension skills. This experimental study examined the extent to which the use of animated videos, as opposed to written text, could (i) reduce construct-irrelevant variance attributed to language and reading skills and (ii) impact test-takers’ reactions to a situational judgment test. The results indicated that the variance attributed to construct-irrelevant factors was lower by 9.5% in the animated version of the test. In addition, those who took the animated test perceived it to be more valid, fair, and enjoyable, than those who took the text-based test. They also rated the language used as less difficult to understand. The implications of these findings are discussed.

What interpretations can we make from scores on graphic-prompt writing (GPW) tasks? An argument-based approach to test validation

This argument-based validation research examines the validity of score interpretations on computer-based graphic-prompt writing (GPW) tasks, centering on the explanation inference. The GPW tasks, designed for English placement testing, measure examinees’ ability to incorporate visual graphic information into their writing. Over 100 ESL students, studying at a public university in the United States, completed GPW tasks and two online questionnaires on graph familiarity (Xi, 2005) and computer mode preference (Lee, 2004) and submitted their standardized English writing test scores. A Pearson product-moment correlation, corrected for attenuation, revealed scores on the GPW tasks and the standardized writing tests had a moderately strong positive relationship (rT1T2 = .50). Multiple linear regression and follow-up correlation analyses showed that GPW task scores were attributed to examinees’ academic writing ability and carried weak, but significant, positive relations to three predictors of graph familiarity (examinees’ experience with and preference for graphs, and graph reading ability). The findings suggest the GPW tasks and the standardized English writing tests assessed different dimensions of the same underlying construct (academic writing ability), and the three predictors of graph familiarity served as sources of construct-irrelevant variance. Theoretical and practical implications of the findings, as well as methodological limitations, are discussed.

Internationally Educated Nurses and the Canadian English Language Benchmark Assessment for Nurses: A Qualitative Test Validation Study of Test-Taker Accounts

This qualitative validation study examines sixteen Internationally Educated Nurses’ (IENs’) accounts of the Canadian English Language Benchmark Assessment for Nurses (CELBAN) at two testing centres (Toronto and Hamilton). This study adopts both focus groups and one-on-one interviews to investigate the inferences drawn from the test, and its consequences. Focus groups and interviews were conducted using an adapted interview guide utilized in the TOEFL iBT investigation of test-taker accounts of construct representation and construct irrelevant variance (DeLuca et al., 2013). While construct representation describes the degree of authenticity in the presentation of Canadian English language nursing tasks, construct irrelevant variance refers to potential factors impacting the test-taking experience which might contribute to a score variance that was not reflective of test-taker knowledge of the testing constructs (Messick, 1989, 1991, 1996). In this study, test-taker accounts of construct representation and construct irrelevant variance constituted the data which were coded and analyzed abductively via the sensitizing concepts derived from DeLuca et al., and Cheng and DeLuca (2011) on examining test-takers’ experience and their contribution to validity. Seven themes emerged, answering four research questions: How do IENs characterize their test experience? How do IENs describe the assessment constructs? What, if any, sources of Construct Irrelevant Variance (CIV) do IENs describe? Do IENs feel the language tasks are authentic? Overall, participants reported positive experiences with the CELBAN, while identifying some possible sources of CIV. Given the CELBAN’s widespread use for high-stakes decisions (a component of nursing certification and licensure), further research of IEN-test-taker responses to construct representation and construct irrelevant variance will remain critical to our understanding of the role of language competency testing for IENs.

Investigating the Psychometric Impact of Negative Worded Items in Reading Comprehension Passages with a 3PL Cross-Classified Testlet Model

Negative worded (NW) items used in psychological instruments have been studied with the bifactor model to investigate whether the NW items form a secondary factor due to negative wording orthogonal to the measured latent construct, a validation procedure which checks whether NW items form a source of construct irrelevant variance (CIV) and hence constitute a validity threat. In the context of educational testing, however, no such validation attempts have been made. In this study, we studied the psychometric impact of NW items in an English proficiency reading comprehension test using a modeling approach similar to the bifactor model, namely the three-parameter logistic cross-classified testlet response theory (3PL CCTRT) model, to account for both guessing and possible local item dependence due to passage effect in the data set. The findings indicate that modeling the NW items with a separate factor leads to noticeable improvement in model fit, and the factor variance is marginal but nonzero. However, item and ability parameter estimates are highly similar between the 3PL CCTRT model and other models that do not model the NW items. It is concluded that the NW items introduce CIV into the data, but its magnitude is too small to change item and person ability parameter estimates to an extent of practical significance.

An investigation of construct relevant and irrelevant features of mathematics problem-solving questions using comparative judgement and Kelly’s Repertory Grid

ABSTRACTThe relationship between the characteristics of 33 mathematical problem-solving questions answered by 16-year-old students in England and the quality of problem-solving elicited was investigated in two studies. The first study used comparative judgement (CJ) to estimate the quality of the problem-solving elicited by each question, involving 33 mathematics teachers judging pairs of journal-style responses to the questions and the application of the Bradley–Terry model. In the second study a variant of Kelly’s Repertory Grid was used with five mathematics teachers to identify 23 dimensions along which the problem-solving questions varied. Significant relationships between ratings on some dimensions and the problem-solving quality estimated in the first study were found. This suggests that the Kelly’s Repertory Grid approach could be an effective way to identify features of questions that are relevant to the construct being assessed and features that could be potential sources of construct-irrelevant variance in test scores.

Construct Relevant or Irrelevant? The Role of Linguistic Complexity in the Assessment of English Language Learners’ Science Knowledge

This article addresses the issue of language-related construct-irrelevant variance on content area tests from the perspective of systemic functional linguistics. We propose that the construct relevance of language used in content area assessments, and consequent claims of construct-irrelevant variance and bias, should be determined according to the degree of correspondence between language use in the assessment and language use in the educational contexts in which the content is learned and used. This can be accomplished by matching the linguistic features of an assessment and the linguistic features of the domain in which the assessment is measuring achievement. This represents a departure from previous work on the assessment of English language learners’ content knowledge that has assumed complex linguistic features are a source of construct irrelevant variance by virtue of their complexity.

Validity Issues in Assessing Linguistic Minorities

Throughout the world, tests are administered to some examinees who are not fully proficient in the language in which they are being tested. It has long been acknowledged that proficiency in the language in which a test is administered often affects examinees’ performance on a test. Depending on the context and intended uses for a particular assessment, linguistic proficiency may be relevant to the tested construct and subsequent interpretations, or may be a source of construct-irrelevant variance that undermines accurate interpretation of the test performance of linguistic minorities who are not proficient in the language of the assessment. In this article, we highlight key validity issues to be considered when testing linguistic minorities, regardless of whether language is central or construct-irrelevant. We discuss examples of the different types of studies test users and developers could conduct to evaluate the validity of scores of linguistic minorities. These issues span test development and validation activities. We conclude with a list of critical factors to consider in test development and evaluation whenever linguistic minorities are tested.

Examiners and content and site: Oh My! A national organization's investigation of score variation in large-scale performance assessments.

Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada's National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.

A Substantive Process Analysis of Responses to Items from the Multistate Bar Examination

We investigated examinees' cognitive processes while they solved selected items from the Multistate Bar Exam (MBE), a high-stakes professional certification examination. We focused on ascertaining those mental processes most frequently used by examinees, and the most common types of errors in their thinking. We compared the relationships between different problem-solving processes, errors, and performance, and assessed the degree to which performance was related to correspondence between examinees' and model problem-solving approaches. Participants most commonly solved problems with reference to legal principles and cues presented in problem stems, but also resorted to other mental processes. We found that performance on the MBE is strongly related to domain-specific skill in using legal principles, as well as to domain-general metacognitive skill in organizing thinking. Evidence that would suggest that testwiseness strategies or common-sense reasoning was a source of construct-irrelevant variance on the MBE was not found. Our study demonstrated a systematic method for in-depth analysis of examinee substantive processes on examinations such as the MBE that assess complex reasoning within a specific domain.

Consistency of report card grades and external assessments in a Canadian province

The study investigated how well report card grades communicate to students and parents that state educational standards are being met, standards that are objectively measured by infrequently administered mandated assessments. Data sources were report card grades and external assessment scores for 2006–09 for Ontario Canada. The information that parents and students received about student performance from report cards and external assessments were similar (rs = .47) to the r = .40–.60 range previously reported. Teachers assigned higher grades than external assessments warranted, even after a major source of construct irrelevant variance in report card grades (teacher ratings on multiple scales measuring student effort and school commitment) was controlled. The relationship of grades to assessment scores was robust across genders, school district types (Public versus Catholic) and language (English and French). Agreement of assessments was higher for grade 6 than for grade 3 and for Writing than for Reading or Mathematics. Report cards provided information about students’ future achievement that was accurate and delivered up to 2 years prior to the administration of external assessments. Seventy to 80% of students who reached the provincial achievement standard on one or both prior report cards were successful on the subsequent external assessment, compared to 30–50% of students who failed to meet the report card standard at least once.

Writing Differences in Teacher Performance Assessments: An Investigation of African American Language and Edited American English

Differential performance results occur when a specific population subgroup achieves a passing rate which is significantly lower than that of the normative reference group. African Americans do less well, in general, on all types of assessments, including constructed-response tests. The present study examined the writing styles of African American test takers in a National Board for Professional Teaching Standards (NBPTS) portfolio entry from the Middle Childhood/Generalist Certificate, as compared with the writing styles of European American test takers. The use of features other than Edited American English was examined as a possible source of construct-irrelevant variance in assessors’ scoring judgments. Thirty-two written commentaries, 18 from African American candidates and 14 from European American candidates, were coded for grammatical, lexical, and discourse features. The coding frame identified features of African American Language (AAL) and Speech Code Errors (SCE). Instances of AAL were fewer than instances of SCE and clustered according to potential users of AAL.

Psychometric Issues in the ELL Assessment and Special Education Eligibility

Assessments in English that are constructed for native English speakers may not provide valid inferences about the achievement of English language learners (ELLs). The linguistic complexity of the test items that are not related to the content of the assessment may increase the measurement error, thus reducing the reliability of the assessment. Language factors that are not relevant to the content being assessed may also be a source of construct-irrelevant variance and negatively impact the validity of the assessment. More important, the results of these tests used as the criteria for identification and classification of ELL students, particularly those at the lower end of the English proficiency spectrum, may be misleading. Caution must be exercised when the results of these tests are used for special education eligibility, particularly in placing ELL students with lower English language proficiency in the learning/reading disability category. This article discusses psychometric issues in the assessment of English language learners and examines the validity of classifying ELL students, with a focus on the possibility of misclassifying ELL students as students with learning disabilities.

Psychometric Issues in the ELL Assessment and Special Education Eligibility

Assessments in English that are constructed for native English speakers may not provide valid inferences about the achievement of English language learners (ELLs). The linguistic complexity of the test items that are not related to the content of the assessment may increase the measurement error, thus reducing the reliability of the assessment. Language factors that are not relevant to the content being assessed may also be a source of construct-irrelevant variance and negatively impact the validity of the assessment. More important, the results of these tests used as the criteria for identification and classification of ELL students, particularly those at the lower end of the English proficiency spectrum, may be misleading. Caution must be exercised when the results of these tests are used for special education eligibility, particularly in placing ELL students with lower English language proficiency in the learning/reading disability category. This article discusses psychometric issues in the assessment of English language learners and examines the validity of classifying ELL students, with a focus on the possibility of misclassifying ELL students as students with learning disabilities.

WRITING DIFFERENCES IN TEACHER PERFORMANCE ASSESSMENTS: USE OF AFRICAN AMERICAN LANGUAGE

ABSTRACTThis study investigated potential writing differences between African American and European American candidates in a National Board for Professional Teaching Standards (NBPTS) portfolio entry from the Middle Childhood/Generalist Certificate. The use of features other than Edited American English was examined as a possible source of construct‐irrelevant variance in assessors' scoring judgments. Thirty‐two written commentaries, 18 from African American candidates and 14 from European American candidates, were coded for grammatical, lexical, and discourse features. The coding frame identified features of African American Language (AAL) and Speech Code Errors (SCE). Instances of AAL were fewer than instances of SCE and clustered according to potential users of AAL.

Threats to the validity of locally developed multiple-choice tests in medical education: construct-irrelevant variance and construct underrepresentation.

Construct-irrelevant variance (CIV) - the erroneous inflation or deflation of test scores due to certain types of uncontrolled or systematic measurement error - and construct underrepresentation (CUR) - the under-sampling of the achievement domain - are discussed as threats to the meaningful interpretation of scores from objective tests developed for local medical education use. Several sources of CIV and CUR are discussed and remedies are suggested. Test score inflation or deflation, due to the systematic measurement error introduced by CIV, may result from poorly crafted test questions, insecure test questions and other types of test irregularities, testwiseness, guessing, and test item bias. Using indefensible passing standards can interact with test scores to produce CIV. Sources of content underrepresentation are associated with tests that are too short to support legitimate inferences to the domain and which are composed of trivial questions written at low-levels of the cognitive domain. "Teaching to the test" is another frequent contributor to CUR in examinations used in medical education. Most sources of CIV and CUR can be controlled or eliminated from the tests used at all levels of medical education, given proper training and support of the faculty who create these important examinations.