Non-equivalent Groups Anchor Test Research Articles

Analysis of Linear Equating of Senior School Certificate Multiple-Choice Examination Papers in Economics

Background: Test equating methods are statistical tools used to produce exchangeable scores across different test forms. However, people believe that it is impossible to develop multiple forms of tests that have the same psychometric properties. Contribution: The study therefore analyzed the linear equating of senior school certificate multiple-choice examination papers in Economics. Method: A Non-equivalent Groups Anchor Test (NEAT) design was adopted for the study. The sample for the study consisted of 1,119 senior secondary three students. The 2009 WAEC, NECO, and NABTEB SSCE multiple-choice Economics papers were adapted and used for data collection with 0.67, 0.64, and 0.60 coefficients of content validity as well as 0.79, 0.76, and 0.70 reliability coefficients. Data collected were analyzed using descriptive statistics, T-scores, and Analysis of Variance. Results: Findings revealed that there was no difference in examinees’ ability, thus equating process could take place. The finding also revealed an F-value of 1.953, which was not significant at the 0.05 alpha level since the p-value (0.142) was greater than 0.05. It showed a score of 41 in WAEC and NECO was equivalent to 42 in NABTEB. It also showed a score of 45 in WAEC was equivalent to 46 and 50 in NECO and NABTEB respectively. Conclusion: The researchers therefore concluded that the 2009 WAEC and NECO SSCE Economics multiple-choice items tended to be equivalent while that of NABTEB was different. It was recommended among others that a regulatory body could be established to monitor and standardize these examinations conducted in Nigeria

Different Methods of Adjusting for Form Difficulty Under the Rasch Model: Impact on Consistency of Assessment Results

AbstractWhen using the Rasch model, equating with a nonequivalent groups anchor test design is commonly achieved by adjustment of new form item difficulty using an additive equating constant. Using simulated 5‐year data, this report compares 4 approaches to calculating the equating constants and the subsequent impact on equating results. The 4 approaches are mean difference, mean difference with outlier removal using the 0.3 logit rule, mean difference with robust z statistic, and the information‐weighted mean difference. Factors studied included sample size, anchor test length, percentage of anchor items displaying outlier behavior, and the distribution of test item difficulty relative to examine ability. The results indicated that the mean difference and information‐weighted mean difference methods performed similarly across all conditions. In addition, with larger sample sizes, the mean difference with 0.3 logit method performed similarly to these 2 methods. The mean difference with robust z method performed most differently from the other three methods of calculating the equating constant. This method removed a large percentage of the anchor items compared to the mean difference with 0.3 logit method but seemed to produce the most stable trend in performance classification across the 5 years, particularly when sample sizes were large.

Equating in small-scale language testing programs

Language programs need multiple test forms for secure administrations and effective placement decisions, but can they have confidence that scores on alternate test forms have the same meaning? In large-scale testing programs, various equating methods are available to ensure the comparability of forms. The choice of equating method is informed by estimates of quality, namely the method with the least error as defined by random error, systematic error, and total error. This study compared seven different equating methods to no equating – mean, linear Levine, linear Tucker, chained equipercentile, circle-arc, nominal weights mean, and synthetic. A non-equivalent groups anchor test (NEAT) design was used to compare two listening and reading test forms based on small samples (one with 173 test takers the other, 88) at a university’s English for Academic Purposes (EAP) program. The equating methods were evaluated based on the amount of error they introduced and their practical effects on placement decisions. It was found that two types of error (systematic and total) could not be reliably computed owing to the lack of an adequate criterion; consequently, only random error was compared. Among the seven methods, the circle-arc method introduced the least random error as estimated by the standard error of equating (SEE). Classification decisions made using the seven methods differed from no equating; all methods indicated that fewer students were ready for university placement. Although interpretations regarding the best equating method could not be made, circle-arc equating reduced the amount of random error in scores, had reportedly low bias in other studies, accounted for form and person differences, and was relatively easy to compute. It was chosen as the method to pilot in an operational setting.

Comparison of Several Small Sample Equating Methods under the NEAT Design

The aim of this study is to compare the performances of Identity, Nominal Weights Mean, and Circle-Arc equating methods under the Non-Equivalent Groups Anchor-Test (NEAT) design. Synthetic equating functions (SFs) of the chosen equating methods (NWS and CAS) were also created using an equal weighting system ( w = 0.5). Different sizes of small examinee samples ( n = 10, 20, 50, 100) were used to equate new test forms to base test forms. Chained Equipercentile (CE) with bivariate log-linear presmoothing was used as population criterion equating function to compare the performances of the equating methods. Overall, the identity (ID) equating was the most favorable, but the NWS method produced less equating error than the ID and TL equating methods under specific simulation conditions. In the light of the findings, recommendation is given to practitioners. Limitations and directions for further research are also provided at the end of this study.

A Comparison of Raw‐to‐Scale Conversion Consistency Between Single‐ and Multiple‐Linking Using a Nonequivalent Groups Anchor Test Design

Maintaining score interchangeability and scale consistency is crucial for any testing programs that administer multiple forms across years. The use of a multiple linking design, which involves equating a new form to multiple old forms and averaging the conversions, has been proposed to control scale drift. However, the use of multiple linking often conflicts with the need for minimizing old item/form exposure and the need for pretesting. This study tried to find a balance point where the needs for equating, item/form exposure and controlling, and pretesting can be satisfied. Three equating scenarios were examined using real data: equating to one old form, equating to two old forms, or equating to three old forms. The finding is that the equating based on one old form produced persistent score drift and also showed increased variability in score means and standard deviations over time. In contrast, equating back to two or three old forms produced much more stable conversions and had less variation. Overall, equating based on multiple linking designs shows the promise of producing more consistent results and preventing scale drift. We recommend that testing programs and practitioners consider the use of multiple linking whenever possible.

Nominal Weights Mean Equating

The authors introduced nominal weights mean equating, a simplified version of Tucker equating, as an alternative for dealing with very small samples. The authors then conducted three simulation studies to compare nominal weights mean equating to six other equating methods under the nonequivalent groups anchor test design with sample sizes of 20, 50, and 80 examinees. Results showed that nominal weights mean equating was generally the most effective. Nominal weights mean equating was, furthermore, never among the least effective methods in any condition, indicating its utility across a wide variety of contexts. Circle-arc equating, another recently developed method, also showed a great deal of promise. The identity function (i.e., no equating) was adequate only when test forms were nearly equivalent in difficulty.

Self-Monitoring Assessments for Educational Accountability Systems

Test-based accountability is now the cornerstone of U.S. education policy, and it is becoming more important in many other nations as well. Educators sometimes respond to test-based accountability in ways that produce score inflation. In the past, score inflation has usually been evaluated by comparing trends in scores on a high-stakes test to trends on a lower-stakes audit test. However, separate audit tests are often unavailable, and their use has several important drawbacks, such as potential bias from motivational differences. As an alternative, we propose self-monitoring assessments (SMAs) that incorporate audit components into operational high-stakes assessments. This paper provides a framework for designing SMAs. It describes 5 specific SMA designs that could be incorporated into the non-equivalent groups anchor test linking approaches used by most large-scale assessments and discusses analytical issues that would arise in their use.

An Evaluation of Five Linear Equating Methods for the NEAT Design

This study uses the results of two previous papers (Kane, Mroch, Suh, & Ripkey, this issue; Suh, Mroch, Kane, & Ripkey, this issue) and the literature on linear equating to evaluate five linear equating methods along several dimensions, including the plausibility of their assumptions and their levels of bias and root mean squared difference (RMSD). The methods all employ non-equivalent groups anchor test (NEAT) design, but make different assumptions about the empirical relationship to be generalized across groups. The analyses indicate that the assumptions employed in Levine Observed-score and Levine True-score methods are more plausible than those for a Tucker, Tucker-like, and Chained Linear method, and that the Levine methods generally have lower levels of bias and RMSD than the other three methods. Furthermore, the methods that employed a chained linear relationship (CLR) approach, in which observed relationships between total test scores and anchor test scores are generalized across groups taking various tests, are found to be more consistent with programs in which a series of test forms administered over a period of years are equated to each other, than a parameter substitution (PS) approach, which estimates results for specific synthetic populations. It is argued that the Levine Observed-score and Levine True-score methods have strong advantages over the other methods studied, unless the groups taking the tests to be equated are known to be very similar.

Small‐Sample Equating Using a Synthetic Linking Function

This study addressed the sampling error and linking bias that occur with small samples in a nonequivalent groups anchor test design. We proposed a linking method called the synthetic function, which is a weighted average of the identity function and a traditional equating function (in this case, the chained linear equating function). Specifically, we compared the synthetic, identity, and chained linear functions for various‐sized samples from two types of national assessments. One design used a highly reliable test and an external anchor, and the other used a relatively low‐reliability test and an internal anchor. The results from each of these methods were compared to the criterion equating function derived from the total samples with respect to linking bias and error. The study indicated that the synthetic functions might be a better choice than the chained linear equating method when samples are not large and, as a result, unrepresentative.

New Results on the Linear Equating Methods for the Non-Equivalent-Groups Design

The two most common observed-score equating functions are the linear and equipercentile functions. These are often seen as different methods, but von Davier, Holland, and Thayer showed that any equipercentile equating function can be decomposed into linear and nonlinear parts. They emphasized the dominant role of the linear part of the nonlinear equating function and gave conditions under which the equipercentile methods in the non-equivalent-groups anchor test (NEAT) design give identical results. Consequently, this article focuses on linear equating methods in a NEAT design—the Tucker, chained, and Levine observed-score functions—and describes the theoretical conditions under which these methods produce the same equating function. Constructed examples illustrate the theoretical results.

Investigating the Population Sensitivity Assumption of Item Response Theory True-Score Equating Across Two Subgroups of Examinees and Two Test Formats

Dorans and Holland (2000) and von Davier, Holland, and Thayer (2003) introduced measures of the degree to which an observed-score equating function is sensitive to the population on which it is computed. This article extends the findings of Dorans and Holland and of von Davier et al. to item response theory (IRT) true-score equating methods that are commonly used in the nonequivalent-groups anchor test (NEAT) design. Using data from the Advanced Placement Program Calculus AB exam, which contain multiple-choice (MC) and free-response (FR) sections, the authors investigate the population sensitivity of the IRT equating functions computed for the MC section only and for the MC and FR sections together. The degree of population sensitivity is also compared across three equating methods: the IRT true-score equating method and two observed-score equating methods, chained equipercentile and Tucker linear equating.

THE IMPACT OF ANCHOR TEST LENGTH ON EQUATING RESULTS IN A NONEQUIVALENT GROUPS DESIGN

ABSTRACTThis study explored the effects of external anchor test length on final equating results of several equating methods, including equipercentile (frequency estimation), chained equipercentile, kernel equating (KE) poststratification PSE with optimal bandwidths, and KE PSE linear (large bandwidths) when using the nonequivalent groups anchor test (NEAT) design. This study used pseudotests constructed of item responses from a real operational test. The equating methods were evaluated using an equating criterion. Conditional differences between the criterion scores and equated scores, and root mean square error of the differences (RMSE) were used as measures to compare the methods to the criterion equating, which in this study is an equivalent groups (EG) equipercentile equating function. The results indicate that bias tended to increase in the conversions as the anchor test length decreased, but the KE PSE with optimal bandwidths and equipercentile (frequency estimation) methods were less sensitive to this change than the other methods. The KE PSE linear method with large bandwidths performed poorly compared to the criterion across all anchor test lengths.

AN ALTERNATIVE TO EQUATING WITH SMALL SAMPLES IN THE NON-EQUIVALENT GROUPS ANCHOR TEST DESIGN

This study addresses the sample error and linking bias that occur with small and unrepresentative samples in a non-equivalent groups anchor test (NEAT) design. We propose a linking method called the synthetic function, which is a weighted average of the identity function (the trivial equating function for forms that are known to be completely parallel) and a traditional equating function (in this case, the chained linear equating function) used in the normal case in which forms are not completely parallel. Specifically, we compare the synthetic, identity, and chained linear functions for various-sized samples from two types of national assessments. One design uses a high reliability test and an external anchor, and the other uses a relatively low reliability test and an internal anchor. The chained linear equating functions derived from the total sample are used as the criterion equating function in both assessments. The results from each of these methods were compared to the criterion equating function with respect to linking bias and error. The study indicates that the synthetic functions might be a better choice than the chained linear equating method when samples are neither large nor representative.

A Unified Approach to Linear Equating for the Nonequivalent Groups Design

This article describes a new, unified framework for linear equating in a non-equivalent groups anchor test (NEAT) design. The authors focus on three methods for linear equating in the NEAT design—Tucker, Levine observed-score, and chain—and develop a common parameterization that shows that each particular equating method is a special case of the linear equating function in the NEAT design. A new concept, the method function, is used to distinguish among the linear equating functions, in general, and among the three equating methods, in particular. This approach leads to a general formula for the standard error of equating for all linear equating functions in the NEAT design. A new tool, the standard error of equating difference , is presented to investigate if the observed difference in the equating functions is statistically significant.

The Chain and Post‐Stratification Methods for Observed‐Score Equating: Their Relationship to Population Invariance

The Non‐Equivalent‐groups Anchor Test (NEAT) design has been in wide use since at least the early 1940s. It involves two populations of test takers, P and Q, and makes use of an anchor test to link them. Two linking methods used for NEAT designs are those (a) based on chain equating and (b) that use the anchor test to post‐stratify the distributions of the two operational test scores to a common population (i.e., Tucker equating and frequency estimation). We show that, under different sets of assumptions, both methods are observed score equating methods and we give conditions under which the methods give identical results. In addition, we develop analogues of the Dorans and Holland (2000) RMSD measures of population invariance of equating methods for the NEAT design for both chain and post‐stratification equating methods.

A UNIFIED APPROACH TO LINEAR EQUATING FOR THE NON‐EQUIVALENT GROUPS DESIGN

ABSTRACTThis paper describes a new, unified framework for linear equating in a Non‐Equivalent‐groups Anchor Test (NEAT) design. We focus on three methods for linear equating in the NEAT design—Tucker, Levine observed‐score, and chain—and develop a common parameterization that allows us to show that each particular equating method is a special case of the linear equating function in the NEAT design. We use a new concept, the Method Function, to distinguish among the linear equating functions, in general, and among the three equating methods, in particular. This approach leads to a general formula for the standard error of equating for all equating functions in the NEAT design. We also present a new tool, the standard error of equating difference, to investigate if the observed difference in the equating functions is statistically significant.

NOTES ON LINEAR EQUATING METHODS FOR THE NON‐EQUIVALENT‐GROUPS DESIGN

ABSTRACTThis paper focuses on three observed‐score methods for linear equating in a Non‐Equivalent‐groups Anchor Test (NEAT) design—Tucker, Levine observed‐score, and chain. The methods and their assumptions are described, and theoretical conditions under which the three methods give identical results are presented.