Multiple Test Forms Research Articles

Investigating the Impact of Compromised Anchor Items on IRT Equating Under the Nonequivalent Anchor Test Design

The prevalence of high-stakes test scores as a basis for significant decisions necessitates the dissemination of accurate and fair scores. However, the magnitude of these decisions has created an environment in which examinees may be prone to resort to cheating. To reduce the risk of cheating, multiple test forms are commonly administered. When multiple forms are employed, the forms must be equated to account for potential differences in form difficulty. If cheating occurs on one of the forms, the equating procedure may produce inaccurate results. A simulation study was conducted to examine the impact of cheating on item response theory (IRT) true score equating. Recovery of equated scores and scaling constants was assessed for the Stocking–Lord IRT scaling method under various conditions. Results indicated that cheating artificially increased the equated scores of the entire examinee group that was administered the compromised form. Future research should focus on the identification and removal of compromised items.

Item Selection for the Development of Parallel Forms From an IRT-Based Seed Test Using a Sampling and Classification Approach

Two sampling-and-classification–based procedures were developed for automated test assembly: the Cell Only and the Cell and Cube methods. A simulation study based on a 540-item bank was conducted to compare the performance of the procedures with the performance of a mixed-integer programming (MIP) method for assembling multiple parallel test forms. The study investigated the statistical equivalence of the forms generated by the three test assembly methods (Cell Only, Cell and Cube, and MIP) in terms of test information functions, test characteristic curves, mean square deviations, and practical constraints, such as content balancing and nonoverlap among forms. The results indicated that the 13-point MIP method outperformed the other two methods in terms of the “closeness” test information functions between the reference form and the assembled parallel tests. Regarding test characteristic curves, the Cell Only and Cell and Cube methods yielded more similar test characteristic curves than the MIP method. Constraining test information functions apparently does not guarantee that the assembled forms will yield similar test characteristic curves. Overall, the Cell Only and Cell and Cube methods have the potential to provide results similar to the optimization approach.

Bees Algorithm for Construction of Multiple Test Forms in E-Testing

The purpose of this research is to automatically construct multiple equivalent test forms that have equivalent qualities indicated by test information functions based on item response theory. There has been a trade-off in previous studies between the computational costs and the equivalent qualities of test forms. To alleviate this problem, we propose an automated system of test construction based on the Bees Algorithm in parallel computing. We demonstrate the effectiveness of the proposed system through various experiments.

Exploring Domain-General and Domain-Specific Linguistic Knowledge in the Assessment of Academic English Language Proficiency

This study examined the role of domain-general and domain-specific linguistic knowledge in the assessment of academic English language proficiency using a latent variable modeling approach. The goal of the study was to examine if modeling of domain-specific variance results in improved model fit and well-defined latent factors. Analyses were carried out on data from the ACCESS for ELLs® test battery, which comprises multiple test forms targeting different grade and proficiency levels. The results of the study provide empirical evidence in support of the conceptual distinction of domain-specific and domain-general linguistic knowledge. Domain-specific factors tended to become more salient with increasing language proficiency, whereas the salience of domain-general factors tended to decrease. However, overall domain-general factors remained stronger than the domain-specific factors. In one test form targeting high levels of proficiency, this factor pattern was reversed, suggesting some degree of fluidity in the relationship between domain-general and domain-specific linguistic knowledge.

Automated Test Assembly Using lp_Solve Version 5.5 in R

This article reviews the use of the software program lp_solve version 5.5 for solving mixed-integer automated test assembly (ATA) problems. The program is freely available under Lesser General Public License 2 (LGPL2). It can be called from the statistical language R using the lpSolveAPI interface. Three empirical problems are presented to demonstrate how to use the program and interface to (a) simultaneously assemble multiple test forms with absolute targets for their test information functions, (b) assemble shadow tests for computerized adaptive testing, and (c) assemble multistage tests using relative targets for their test information functions, all subject to various quantitative and categorical constraints. The results of this study indicate that it is now possible for researchers and testing organizations to implement ATA for small to moderately sized test assembly problems using free software.

Item Difficulty Predictors of a Multiple-choice Reading Test

The researchers’ experiential knowledge demonstrates that the task of predicting and controlling the difficulty level of the multiple-choice items of the College Scholastic Ability Test (CSAT) for English is substantially left to the subjective judgment of experienced item writers. The present study accordingly recognizes a need to identify item difficulty predictors and build an item difficulty prediction model to handle this pertinent issue. While taking separate interest in constructing a model for the multiplechoice reading subset of the CSAT, the study was conducted by identifying item difficulty predictor variables from previous research, and by validating the candidate predictors via questionnaires by highly experienced teacher-raters when asked to analyze reading items from the English reading subset of the preliminary CSAT (i.e., yun-hap-hak-ryuk-pyung-ga) administered in March 2009. Using multiple regression technique and maximum likelihood estimation, an item difficulty prediction model was generated. In order to check validity and applicability of the prediction model, the hypothetical model was finally tested on a subsequent version of the test administered in September 2009. This type of model building is expected to guide test developers design an item pool in accordance with special needs, such as to construct multiple test forms, which have similar mean difficulties.

An NCME Instructional Module on Booklet Designs in Large‐Scale Assessments of Student Achievement: Theory and Practice

In most large‐scale assessments of student achievement, several broad content domains are tested. Because more items are needed to cover the content domains than can be presented in the limited testing time to each individual student, multiple test forms or booklets are utilized to distribute the items to the students. The construction of an appropriate booklet design is a complex and challenging endeavor that has far‐reaching implications for data calibration and score reporting. This module describes the construction of booklet designs as the task of allocating items to booklets under context‐specific constraints. Several types of experimental designs are presented that can be used as booklet designs. The theoretical properties and construction principles for each type of design are discussed and illustrated with examples. Finally, the evaluation of booklet designs is described and future directions for researching, teaching, and reporting on booklet designs for large‐scale assessments of student achievement are identified.

The Bookmark Procedure for Setting Cut-Scores and Finalizing Performance Standards: Strengths and Weaknesses

The Bookmark standard-setting procedure was developed to address the perceived problems with the most popular method for setting cut-scores: the Angoff procedure (Angoff, 1971). The purposes of this article are to review the Bookmark procedure and evaluate it in terms of Berk’s (1986) criteria for evaluating cut-score setting methods. The strengths and weaknesses of the Bookmark are critically examined and discussed. In general, the strengths of the Bookmark method are that it (a) accommodates constructed-response as well as selected-response test items; (b) efficiently accommodates multiple cut-scores and multiple test forms; and (c) reduces cognitive complexity for panelists. Despite unresolved issues like the choice and understanding of the response probability, the Bookmark method remains a promising procedure for setting cut-scores and finalizing performance standards.

A Comparison of Using the Fixed Common-Precalibrated Parameter Method and the Matched Characteristic Curve Method for Linking Multiple-Test Items

A linking design typically consists of a data collection procedure together with an item linking procedure that places item parameters calibrated from multiple test forms onto a common scale. This study considered 2 potentially useful item response theory linking designs. The first one is characterized by selecting a single set of common items across all multiple test forms, the precalibrated item parameters of which are kept fixed while the unknown parameters of the other items are being estimated. This linking design will be referred to as the fixed common-precalibrated item parameter design. However, data collected under this design could also be analyzed by the characteristic curve method, which constituted an alternative linking procedure. In this study, the relative merits of the 2 linking designs were examined with respect to their robustness against 3 manipulated conditions-namely, when the common items have imprecise estimates, when there is a noticeable difference in the average item difficulty between the common and the noncommon items, and when the examinees are heterogeneous in terms of their abilities. A parameter recovery study was conducted to achieve this purpose. The results indicated that both linking designs were capable of producing accurate linking of items and equivalent estimation of ability parameters under the 3 conditions. When the 2 designs were actually utilized in the development of an item bank, it was found that both linking designs produced quite consistent solutions despite minor differences on some item and ability estimates. Condition under which a linking design is preferred over the other is also provided in the Discussion section of this article.

Detecting score drift in a high-stakes performance-based assessment.

Although studies have been conducted to examine the effects of a variety of factors on the comparability of scores obtained from standardized patient examinations (SPE), little research has been conducted to specifically investigate the challenge of detecting drift in case difficulty estimates over time, particularly for large-scale, performance-based, assessments. The purpose of the current study was to investigate the use of a procedure to detect drift in the difficulty estimates for a large-scale, high stakes SPE. The results of this investigation suggest that, for particular performance tasks, there was some variation in mean scores over time. These findings indicate that, although it is feasible to create a bank of case-SP means and link scores back to these fixed estimates, special attention must be paid to the standardization of exam materials over time. This is essential to ensure comparability of scores and pass-fail decisions for candidates who are assessed on multiple test forms throughout the year.

Assessment criteria in a large-scale writing test: what do they really mean to the raters?

The process of rating written language performance is still not well understood, despite a body of work investigating this issue over the last decade or so (e.g., Cumming, 1990; Huot, 1990; Vaughan, 1991; Weigle, 1994a; Milanovic et al., 1996). The purpose of this study is to investigate the process by which raters of texts written by ESL learners make their scoring decisions using an analytic rating scale designed for multiple test forms. The context is the Special Test of English Proficiency ( step), which is used by the Australian government to assist in immigration decisions. Four trained, experienced and reliable step raters took part in the study, providing scores for two sets of 24 texts. The first set was scored as in an operational rating session. Raters then provided think-aloud protocols describing the rating process as they rated the second set. A coding scheme developed to describe the think-aloud data allowed analysis of the sequence of rating, the interpretations the raters made of the scoring categories in the analytic rating scale, and the difficulties raters faced in rating. Data show that although raters follow a fundamentally similar rating process in three stages, the relationship between scale contents and text quality remains obscure. The study demonstrates that the task raters face is to reconcile their impression of the text, the specific features of the text, and the wordings of the rating scale, thereby producing a set of scores. The rules and the scale do not cover all eventualities, forcing the raters to develop various strategies to help them cope with problematic aspects of the rating process. In doing this they try to remain close to the scale, but are also heavily influenced by the complex intuitive impression of the text obtained when they first read it. This sets up a tension between the rules and the intuitive impression, which raters resolve by what is ultimately a somewhat indeterminate process. In spite of this tension and indeterminacy, rating can succeed in yielding consistent scores provided raters are supported by adequate training, with additional guidelines to assist them in dealing with problems. Rating requires such constraining procedures to produce reliable measurement.

Optimal Assembly of Psychological and Educational Tests

The advent of computers in psychological and educational measurement has led to the need for algorithms for optimal assembly of tests from item banks. This paper reviews optimal test assembly literature and introduces the contributions to this Special Issue. Four approaches to computerized test assembly are discussed: heuristic-based test assembly, 0-1 linear programming, network-flow programming, and an optimal design approach. Ap-plications of these methods to a variety of problems are examined, including IRT-based test assembly, classical test assembly, assembling multiple test forms, item matching, observed-score equating, constrained adaptive testing, assembling tests with item sets, item bank design, and assembling tests with multiple traits. A bibliography on optimal test assembly is provided.

Simultaneous Assembly of Multiple Test Forms

An algorithm for the assembly of multiple test forms is proposed in which the multiple‐form problem is reduced to a series of computationally less intensive two‐form problems. At each step, one form is assembled to its true specifications; the other form is a dummy assembled only to maintain a balance between the quality of the current form and the remaining forms. It is shown how the method can be implemented using the technique of O‐1 linear programming. Two empirical examples using a former item pool from the LSAT are given—one in which a set of parallel forms is assembled and another in which the targets for the information functions of the forms are shifted systematically.

The "Unbiased" Anchor: Bridging the Gap Between DIF and Item Bias

This research uses item response theory for the investigation of differential item functioning (DlF) and examines the advantages of using a multivariate external criterion to match examinees from the various subpopulations. An attempt was made to select a set of unbiased items to comprise the matching criterion. Expected course grades provided by the students' teachers were examined and found to function similarly across sex and race subgroups and were incorporated into the matching criterion. The inclusion of teachers' grades provides additional information in the extreme proficiency ranges and has the potential to enhance the validity of subgroup comparisons and ihferences of possible item bias. The use of an external matching criterion allows test developers to embed the same short, designated anchor in multiple test forms, thereby avoiding the circularity inherent to conventional DIF analyses and permitting comparison of DIF statistics across test forms.

Test-retest reliability of computerized, everyday memory measures and traditional memory tests

Abstract Test-retest reliabilities and practice effect magnitudes are considered for nine computer-simulated tasks of everyday cognition and five traditional neuropsychological tests. Most measures showed significant practice effects upon reevaluation. Test-retest reliabilities of the computerized recall measures were equal to or superior to the traditional memory tests, but traditional measures of attention and concentration had superior reliabilities to the everyday tests. Test-retest reliabilities of everyday recognition memory tests were disappointing. Advantages of computerization of psychometric assessment and the need for multiple test forms are discussed.

Multiple equivalent test forms in a computerized, everyday memory battery

Eight parallel forms for six computerized, everyday memory tests were examined for equivalence of difficulty level. Six equivalent forms were found for Telephone Dialing, Name-Face Association, First-Last Name memory, and Grocery List Learning, white eight equivalent forms were found for Misplaced Objects and Recognition of Faces. The clinical and research utility of multiple equivalent forms of everyday memory tests is discussed.

Multiple equivalent test forms in a computerized, everyday memory battery

Multiple equivalent test forms in a computerized, everyday memory battery

Effect of Examinee Group on Equating Relationships

Many educational tests make use of multiple test forms, which are then horizontally equated to establish interchangeability among forms. To have confidence in this interchangeability, the equating relationships should be robust to the particular group of examinees on which the equating is conducted. This study inves tigated the effects of ability of the examinee group used to establish the equating relationship on linear, equipercentile, and three-parameter logistic IRT esti mated true score equating methods. The results show all of the methods to be reasonably independent of ex aminee group, and suggest that population independ ence is not a good reason for selecting one method over another.

Best linear prediction of composite universe scores

The problem of predicting universe scores for samples of examinees based on their responses to samples of items is treated. A general measurement procedure is described in which multiple test forms are developed from a table of specifications and each form is administered to a different sample of examinees. The measurement model categorizes items according to the cells of such a table, and the linear function derived for minimizing error variance in prediction uses responses to these categories. In addition, some distinctions are drawn between aspects of the approach taken here and the familiar regressed score estimates.

SOME EXPLORATORY INDICES FOR SELECTION OF A TEST EQUATING METHOD

Most major testing programs require the creation of multiple test forms to preserve test security and to sustain the integrity of claims to generalizability. Regardless of the care with which they are constructed, multiple test forms invariably differ in difficulty level and in variability. Some method must therefore be used to translate scores on one form of the test to the scale of another. When the test forms are essentially parallel, such translating is called test equating. The methodological literature on test equating has been abundant since the early 1950's (Angoff, 1961, 1968; Flanagan, 1951; Lennon, 1964; Lord, 1950, 1955; Maxey & Lenning, 1974). Angoff (1971) has summarized this literature. More recent articles have concentrated on the application of latent trait models to problems which include test equating, through so-called sample-free specification of test item parameters (Marco, Petersen, & Stewart, 1980). This work is, however, beyond the scope of this paper, since we are not concerned with the effectiveness and utility of latent trait equating methods.