Scores For Examinees Research Articles

When Should Individual Ability Estimates Be Reported if Rapid Guessing Is Present?

ABSTRACT Testing programs are confronted with the decision of whether to report individual scores for examinees that have engaged in rapid guessing (RG). As noted by the Standards for Educational and Psychological Testing, this decision should be based on a documented criterion that determines score exclusion. To this end, a number of heuristic criteria (e.g., exclude all examinees with RG rates of 10%) have been adopted in the literature. Given that these criteria lack strong methodological support, the objective of this simulation study was to evaluate their appropriateness in terms of individual ability estimate and classification accuracy when manipulating both assessment and RG characteristics. The findings provide evidence that employing a common criterion for all examinees may be an ineffective strategy because a given RG percentage may have differing degrees of biasing effects based on test difficulty, examinee ability, and RG pattern. These results suggest that practitioners may benefit from establishing context-specific exclusion criteria that consider test purpose, score use, and targeted examinee trait levels.

Demographic factors that influence performance on the Praxis Earth and Space Science: Content Knowledge Test

What teachers know about their content area plays a role in how they effectively enhance the scientific literacy of their students. This leads to questions about teacher content knowledge as they enter the field. The Praxis Earth and Space Science: Content Knowledge Test is the most widely used licensure assessment for aspiring Earth science teachers in the United States. There are few previous reports on the personal and professional characteristics of those test-takers which would provide information about Earth science teacher candidates. This study presents findings based on analysis of trends in reported demographic variables and scaled scores for examinees assessed by the licensure exam from June 2006 to May 2016 in order to identify which have the highest correlation with their performance. Three major themes emerged as a result of the study: there is currently a large proportion of out of field teaching, a lack of diversity among test-takers, and underperformance of Black/African American test-takers when compared to their White counterparts as well as in females compared to males. Our analysis uncovered more subtle differences in performance based on undergraduate GPA and age group as well. Findings from this study suggest recommendations for improving focused recruitment and retention efforts to support identified at-risk populations enrolled in Earth science teacher preparation programs.

Metric Stability in Item Response Models

In item response theory, uncertainty associated with estimated item parameters can lead to greater uncertainty in subsequent analyses, such as estimating trait scores for individual examinees. Most existing methods to characterize or correct for item parameter uncertainty implicitly assume that the latent trait continuum is fixed across the posterior distribution of item parameters. However, the latent trait continuum can also be understood as an artifact of the fitted model, such that the location of this continuum is determined with error. In other words, item parameter estimation error implies uncertainty about the location of the metric. This article uses Ramsay’s (1996) geometry of the latent trait metric to develop a quantitative measure of metric stability, that is, the sampling variability of the latent trait continuum implied by errors in item parameter estimation. Through a series of illustrations, it is clarified how metric stability is related to other item response model evaluation outcomes (e.g., test information, model fit), and how metric stability can be useful in identifying well-determined regions of the latent trait continuum, making sample size recommendations, and selecting a model. Overall, the proposed measure of metric stability provides meaningful and highly interpretable information to aid in item response model evaluation.

Comparing the Performance of Eight Item Preknowledge Detection Statistics.

Item preknowledge describes a situation in which a group of examinees (called aberrant examinees) have had access to some items (called compromised items) from an administered test prior to the exam. Item preknowledge negatively affects both the corresponding testing program and its users (e.g., universities, companies, government organizations) because scores for aberrant examinees are invalid. In general, item preknowledge is hard to detect due to multiple unknowns: unknown groups of aberrant examinees (at unknown test centers or schools) accessing unknown subsets of items prior to the exam. Recently, multiple statistical methods were developed to detect compromised items. However, the detected subset of items (called the suspicious subset) naturally has an uncertainty due to false positives and false negatives. The uncertainty increases when different groups of aberrant examinees had access to different subsets of items; thus, compromised items for one group are uncompromised for another group and vice versa. The impact of uncertainty on the performance of eight statistics (each relying on the suspicious subset) was studied. The measure of performance was based on the receiver operating characteristic curve. Computer simulations demonstrated how uncertainty combined with various independent variables (e.g., type of test, distribution of aberrant examinees) affected the performance of each statistic.

Race, Poverty and SAT Scores: Modeling the Influences of Family Income on Black and White High School Students’ SAT Performance

BackgroundEducational policy makers and test critics often assert that standardized test scores are strongly influenced by factors beyond individual differences in academic achievement such as family income and wealth. Unfortunately, few empirical studies consider the simultaneous and related influences of family income, parental education, and high school achievement on college admissions test scores.Focus Of StudyThis research was animated by the nagging question of the association of family income with SAT performance. For example, is the relationship between family income and SAT performance non-linear? Does the relationship differ markedly by race? More importantly, how strong are the effects of poverty on SAT performance?Research DesignThis study is a secondary analysis of a large national sample of Black and White college-bound high school students who took the SAT in 2003 (N = 781,437).Data Collection and AnalysisEmploying data from the College Board's Student Descriptive Questionnaire, this study used structural equation modeling (SEM) to estimate the effects of family income on SAT scores for Black and White examinees accounting for the simultaneous effects of parental education and high school achievement.Findings/ResultsResults suggest the effects of family income on SAT scores, though relatively modest in contrasts to high school achievement, are substantial, non-linear, and nearly twice as large for Black students. Moreover, the unstandardized direct effect of high school achievement on SAT performance is not enough to address the substantial effects of poverty for Black students.Conclusions/RecommendationsThe findings are discussed with respect to social inequality and educational opportunity in college admissions.

Comment on 3PL IRT Adjustment for Guessing

Guessing behavior is an issue discussed widely with regard to multiple choice tests. Its primary effect is on number-correct scores for examinees at lower levels of proficiency. This is a systematic error or bias, which increases observed test scores. Guessing also can inflate random error variance. Correction or adjustment for guessing formulas has been applied to address some of these issues. The purpose of this research comment is to draw attention to the adjustment for guessing implicit in the three-parameter logistic item response theory model. Potential equity issues also arise with respect to this adjustment.

Delay in taking the American Board of Surgery qualifying examination affects examination performance

The American Board of Surgery (ABS) Qualifying Examination (QE) represents an important step along the pathway to board certification. We investigated whether candidates who delayed taking the QE had worse performance on the examination. QE pass rates and equated scaled scores for all first-time examinees from 2006 to 2010 (n = 5,193) were reviewed. After eliminating examinees who could not be matched to final ABS In-Training Examination (ABSITE) scores, the remaining cohort (n = 4,909) was analyzed by comparing those who took the exam immediately after residency (Immediate, n = 4,488) to those who delayed for 1 or more years (Delay, n = 421). The Immediate group had a mean first-time QE pass rate of 87% compared to 57% for those who delayed 1 year and 48% for those who delayed 2 or more years (P < .001). Regression analysis demonstrated that delay in taking the QE remained a significant determinant of exam failure after controlling for ABSITE scores (odds ratio = 0.35; 95% CI, 0.29-0.43; P = .001). Undergraduate medical education and postresidency training did not affect the results. The Delay group had lower equated scaled scores, a greater ultimate failure rate on the QE, and was more likely to fail the ABS Certifying Examination on the first attempt. These results demonstrate that candidates who delayed taking the QE immediately are at extremely high risk for exam failure and failure to achieve board certification. These findings presumably are due to deterioration of knowledge over time, but they also may represent characteristics of the Delay group that are currently undefined.

Evaluating Construct Equivalence and Criterion-Related Validity for Repeat Examinees on a Standardized Patient Examination

Prior studies report large score gains for examinees who fail and later repeat standardized patient (SP) assessments. Although research indicates that score gains on SP exams cannot be attributed to memorizing previous cases, no studies have investigated the empirical validity of scores for repeat examinees. This report compares single-take and repeat examinees in terms of both internal (construct) validity and external (criterion-related) validity. Data consisted of test scores for examinees who took the United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam between July 16, 2007, and September 12, 2009. The sample included 12,090 examinees who completed Step 2 CS on one occasion and another 4,030 examinees who completed the exam on two occasions. The internal measures included four separately scored performance domains of the Step 2 CS examination, whereas the external measures consisted of scores on three written assessments of medical knowledge (Step 1, Step 2 clinical knowledge, and Step 3). The authors subjected the four Step 2 CS domains to confirmatory factor analysis and evaluated correlations between Step 2 CS scores and the three written assessments for single-take and repeat examinees. The factor structure for repeat examinees on their first attempt was markedly different from the factor structure for single-take examinees, but it became more similar to that for single-take examinees by their second attempt. Scores on the second attempt correlated more highly with all three external measures. The findings support the validity of scores for repeat examinees on their second attempt.

The Effect of Task Exposure on Repeat Candidate Scores in a High-Stakes Standardized Patient Assessment

Background: Failing candidates often make multiple attempts on licensure and certification examinations. For performance-based assessments, where available test material is often limited, overlap in examination content is frequently inevitable. Purpose: The purpose of this study was to investigate the performance of repeat candidates, both on new and exposed material, on a standardized patient clinical skills assessment. Methods: Analysis of variance techniques were used to investigate the longitudinal performance of repeat candidates. Score differences for the second test session were analyzed as function of prior exposure to assessment materials. Results: Although the performance of candidates improved between the first and second assessment, score increases could not be attributed to the random exposure of examination material. Conclusions: The exposure of case content on clinical skills examinations does not appear to provide any advantage, or disadvantage, to repeat test takers. This finding, although based on average scores for repeat examinees, suggests that valid assessment scores can be obtained even if there is some overlap in assessment content.

The Validity and Comparability of Entrance Examination Scores After Accommodations Are Made for Students with LD

Every year, thousands of college and university applicants with learning disabilities (LD) present scores from standardized examinations as part of the admissions process for postsecondary education. Many of these scores are from tests administered with nonstandard procedures due to the examinees' learning disabilities. Using a sample of college students with LD and a control sample, this study investigated the criterion validity and comparability of scores on the Miller Analogies Test when accommodations for the examinees with LD were in place. Scores for examinees with LD from test administrations with accommodations were similar to those of examinees without LD on standard administrations, but less well associated with grade point averages. The results of this study provide evidence that although scores for examinees with LD from nonstandard test administrations are comparable to scores for examinees without LD, they have less criterion validity and are less meaningful for their intended purpose.

COMPARABILITY OF PAPER‐AND‐PENCIL AND COMPUTER ADAPTIVE TEST SCORES ON THE GRE® GENERAL TEST

ABSTRACTThis report summarizes a study conducted to assess the comparability of paper‐and‐pencil (P&amp;P) and computer adaptive test (CAT) scores on the GRE® General Test. Volunteer examinees from around the country were randomly assigned to take the test in either CAT mode or P&amp;P mode. Results indicated that for each measure (verbal, quantitative, analytical), mean scores on the CAT were higher than mean scores on P&amp;P. It was hypothesized that CAT examinee test‐taking behavior with regard to the CAT scoring method that was in place at the time of the study (allowing student to receive a score by answering only 80% or more of the questions) may have affected CAT scores. Investigation of this hypothesis indicated that CAT examinees who did not complete their CATs obtained higher mean scores than would be predicted. However, mean scores for examinees who completed their CATs were similar to mean scores for P&amp;P examinees. A new psychometrically‐defensible CAT scoring method, called proportional scoring, was developed in which it is to the examinees' advantage, in terms of maximizing their scores, to carefully consider and answer as many items as they can. The proportional scoring method has been implemented in the operational GRE CAT program, and it is believed that this scoring method will result in CAT scores that are comparable to P&amp;P scores. GRE CAT scores will continue to be closely monitored.

Formula Scoring of Multiple‐Choice Tests (Correction for Guessing)

Formula scoring is a procedure designed to reduce multiple‐choice test score irregularities due to guessing. Typically, a formula score is obtained by subtracting a proportion of the number of wrong responses from the number correct. Examinees are instructed to omit items when their answers would be sheer guesses among all choices but otherwise to guess when unsure of an answer. Thus, formula scoring is not intended to discourage guessing when an examinee can rule out one or more of the options within a multiple‐choice item. Examinees who, contrary to the instructions, do guess blindly among all choices are not penalized by formula scoring on the average; depending on luck, they may obtain better or worse scores than if they had refrained from this guessing. In contrast, examinees with partial information who refrain from answering tend to obtain lower formula scores than if they had guessed among the remaining choices. (Examinees with misinformation may be exceptions.) Formula scoring is viewed as inappropriate for most classroom testing but may be desirable for speeded tests and for difficult tests with low passing scores. Formula scores do not approximate scores from comparable fill‐in‐the‐blank tests, nor can formula scoring preclude unrealistically high scores for examinees who are very lucky.

Internal Construct Validity of a College Admissions Test Across Handicapped and Nonhandicapped Groups

This study investigated the comparability of SAT Verbal and Mathematical scores for nonhandicapped examinees taking standard administrations and handicapped populations taking extended-time, special administrations. A simple two-factor confirmatory model based on Verbal and Mathematical item parcels was posed and tested for invariance across two sets of handicapped and nonhandicapped samples, with each set taking a different form of the SAT. Results showed the two common-factor model to provide a reasonable fit in all populations with the mathematical factor generally fitting somewhat better than the verbal one. Compared with the nonhandicapped population, the factors tended to be less correlated in most of the handicapped groups. In addition, some evidence of verbal method factors defined by the antonym and reading comprehension item types was evident. Finally, there was some suggestion that one of the two Mathematical item types led to different observed score scale units for learning disabled students taking a cassette administration.

TESTING HANDICAPPED PEOPLE—THE VALIDITY ISSUE

ABSTRACTAdmissions testing programs provide modified tests and test procedures for handicapped individuals in order to reduce the likelihood that a sensory or physical disability may spuriously lower test scores. Regulations under Section 504 of the Rehabilitation Act require that the validity of such tests be demonstrated. The meaning of validity in this instance is examined in consideration of federal legislation, current test theory and practice, and unusual circumstances encountered in the testing of handicapped people. It is concluded that test validity for handicapped subgroups means primarily test fairness and the measure of test fairness is comparability. Five measures of comparability between test scores for handicapped and nonhandicapped examinees are distinguished. For considerations of equity as well as the inherent difficulty in establishing score comparability, it is important to give careful attention also to the comparability of the task presented by the assessment procedure.

A STUDY OF HYPOTHESES BASIC TO THE USE OF RIGHTS AND FORMULA SCORES

The hypothesis that some students, when tested under formula directions, omit items about which they have useful partial knowledge implies that such directions are not as fair as rights directions, especially to those students who are less inclined to guess. This hypothesis may be called the differential effects hypothesis. An alternative hypothesis states that examinees would perform no better than chance expectation on items that they would omit under formula directions but would answer under rights directions. This may be called the invariance hypothesis. Experimental data on this question were obtained by conducting special test administrations of College Board SAT‐verbal and Chemistry tests and by including experimental tests in a Graduate Management Admission Test administration. The data provide a basis for evaluating the two hypotheses and for assessing the effects of directions on the reliability and parallelism of scores for sophisticated examinees taking professionally developed tests. Results support the invariance hypothesis rather than the differential effects hypothesis.

LATENT TRAIT MODELS AND THEIR USE IN THE ANALYSIS OF EDUCATIONAL TEST DATA1,2,3

A theory of latent traits supposes that in testing situations, examinee performance on a test can be predicted (or explained) by defining characteristics of examinees, referred to as traits, estimating scores for examinees on these traits, and using the scores to predict or test performance (Lord and Novick, 1968). Since the traits are not directly measurable and therefore unobservable, they are often referred to as latent traits or abilities. A latent trait model specifies a relationship between examinee test performance and the traits or abilities assumed to underlie performance on the test. The relationship between the observable and the unobservable quantities is described by a mathematical function. For this reason, latent trait models are mathematical models. Also, latent trait models are based on assumptions about the test data. When selecting a particular latent trait model to apply to one's test data, it is necessary to consider whether the test data satisfy the assumptions of the model. If they do not, different test models should be considered. Alternately, some psychometricians (for example, Wright, 1968) have recommended that test developers design their tests so as to satisfy the assumptions of the particular latent trait model they are interested in using. Recent work by Lord (1968, 1974a), Lord and Novick (1968), Wright (1968), Wright and Panchapakesan (1969), Samejima (1969, 1972), Bock and Wood (1971), and Whitely and Dawis (1974) has been helpful in introducing educational measurement specialists to the topic of latent trait models. Also, the work of these and other individuals has contributed substantially to the current interest among test practitioners in applying the models to a wide variety of educational and psychological testing problems. Latent trait models are now being used to explain examinee test performance as well as to provide a framework for solving test design problems and other important testing questions that have, to date, gone unresolved (Lord, 1977; Wright, 1977a, 1977b). Why has the use of latent trait models in practical testing situations been low? There are at least five reasons. For one, the topic of latent trait theory represents a complex branch of the field of test theory. The advanced mathematical skills required to study many of the papers published on the topic have probably discouraged many potential