Construct Responsiveness Research Articles

Investigating the Effectiveness of Equating Designs for Constructed‐Response Tests in Large‐Scale Assessments

Using data from a large‐scale exam, in this study we compared various designs for equating constructed‐response (CR) tests to determine which design was most effective in producing equivalent scores across the two tests to be equated. In the context of classical equating methods, four linking designs were examined: (a) an anchor set containing common CR items, (b) an anchor set incorporating common CR items rescored, (c) an external multiple‐choice (MC) anchor test, and (d) an equivalent groups design incorporating rescored CR items (no anchor test). The use of CR items without rescoring resulted in much larger bias than the other designs. The use of an external MC anchor resulted in the next largest bias. The use of a rescored CR anchor and the equivalent groups design led to similar levels of equating error.

STUDIES OF A LATENT CLASS SIGNAL DETECTION MODEL FOR CONSTRUCTED RESPONSE SCORING II: INCOMPLETE AND HIERARCHICAL DESIGNS

ABSTRACTA basic consideration in large‐scale assessments that use constructed response (CR) items, such as essays, is how to allocate the essays to the raters that score them. Designs that are used in practice are incomplete, in that each essay is scored by only a subset of the raters, and also unbalanced, in that the number of essays scored by each rater differs across the raters. In addition, all of the possible rater pairs may not be used. The present study examines the effects of these factors on parameter recovery and classification accuracy using simulations of a latent class model based on signal detection theory (SDT). Many tests also include more than one CR item, which introduces a nested or hierarchical structure into the design, in that raters are nested within essays (i.e., there are multiple raters per essay) and essays are nested within examinees (i.e., each examinee provides two or more essays). A hierarchical rater model (HRM) has previously been developed to recognize the nested structure. A version of the HRM that incorporates a latent class signal detection model in the first level, referred to as the HRM‐SDT model, is presented. Parameter recovery in the HRM‐SDT model is examined in simulations. The model is applied to data from several ETS tests.

SINGLE‐ VERSUS DOUBLE‐SCORING OF TREND RESPONSES IN TREND SCORE EQUATING WITH CONSTRUCTED‐RESPONSE TESTS

ABSTRACTThis study examines the differences in equating outcomes between two trend score equating designs resulting from two different scoring strategies for trend scoring when operational constructed‐response (CR) items are double‐scored—the single group (SG) design, where each trend CR item is double‐scored, and the nonequivalent groups with anchor test (NEAT) design, where each trend CR item is single‐scored during trend score equating—for varying sample sizes (n =150, 200, 250, 300, 400). Overall results suggest larger equating errors with smaller sample sizes, though errors were small regardless of sample size. The NEAT design performed about as well as the SG design with respect to conditional and summative standard errors of equating, though it did tend to produce larger bias and root mean‐squared differences (RMSDs). When accounting for the total number of trend scores required to do analyses, the NEAT design performed as well or better than the SG design (e.g., when the NEAT n =150 and the SG n = 300). This result might be partially attributable to a larger operational sample size (n = 792) and a good correlation between anchor and total score for the trend sample (r = 0.73). These results suggest that under these testing conditions, the NEAT design performed about as well as the SG design, but further research is required to assess the generalizability of the results.

Comparisons among Designs for Equating Mixed‐Format Tests in Large‐Scale Assessments

In this study we examined variations of the nonequivalent groups equating design for tests containing both multiple‐choice (MC) and constructed‐response (CR) items to determine which design was most effective in producing equivalent scores across the two tests to be equated. Using data from a large‐scale exam, this study investigated the use of anchor CR item rescoring (known as trend scoring) in the context of classical equating methods. Four linking designs were examined: an anchor with only MC items, a mixed‐format anchor test containing both MC and CR items; a mixed‐format anchor test incorporating common CR item rescoring; and an equivalent groups (EG) design with CR item rescoring, thereby avoiding the need for an anchor test. Designs using either MC items alone or a mixed anchor without CR item rescoring resulted in much larger bias than the other two designs. The EG design with trend scoring resulted in the smallest bias, leading to the smallest root mean squared error value.

EVALUATING SUBPOPULATION INVARIANCE OF LINKING FUNCTIONS TO DETERMINE THE ANCHOR COMPOSITION FOR A MIXED‐FORMAT TEST

ABSTRACTWe examined the appropriateness of the anchor composition in a mixed‐format test, which includes both multiple‐choice (MC) and constructed‐response (CR) items, using subpopulation invariance indices. We derived linking functions in the nonequivalent groups with anchor test (NEAT) design using two types of anchor sets: (a) MC only and (b) a mix of MC and CR. In each anchor condition, we derived the linking functions separately for males and females and then compared those subpopulation functions to the total group function. In the MC‐only condition, the difference between the subpopulation functions and the total group function was not trivial in a score region that included cut scores, leading to inconsistent pass/fail decisions for low‐performing examinees in particular. Overall, the mixed anchor was a better choice than the MC‐only anchor to achieve subpopulation invariance between males and females. The research reinforces subpopulation invariance indices as a means of determining the adequacy of the anchor.

Multidimensional Linking for Tests with Mixed Item Types

Numerous assessments contain a mixture of multiple choice (MC) and constructed response (CR) item types and many have been found to measure more than one trait. Thus, there is a need for multidimensional dichotomous and polytomous item response theory (IRT) modeling solutions, including multidimensional linking software. For example, multidimensional item response theory (MIRT) may have a promising future in subscale score proficiency estimation, leading toward a more diagnostic orientation, which requires the linking of these subscale scores across different forms and populations. Several multidimensional linking studies can be found in the literature; however, none have used a combination of MC and CR item types. Thus, this research explores multidimensional linking accuracy for tests composed of both MC and CR items using a matching test characteristic/response function approach. The two‐dimensional simulation study presented here used real data‐derived parameters from a large‐scale statewide assessment with two subscale scores for diagnostic profiling purposes, under varying conditions of anchor set lengths (6, 8, 16, 32, 60), across 10 population distributions, with a mixture of simple versus complex structured items, using a sample size of 3,000. It was found that for a well chosen anchor set, the parameters recovered well after equating across all populations, even for anchor sets composed of as few as six items.

Do Constructed-Response and Multiple-Choice Questions Measure the Same Thing?

Our study empirically investigates the relationship between constructed- response (CR) and multiple-choice (MC) questions using a unique data set compiled from several years of university introductory economics classes. We conclude that CR and MC questions do not measure the same thing. Our main contribution is that we show that CR questions contain independent information that is related to student learning. Specifically, we find that the component of CR scores that cannot be explained by MC responses is positively and significantly related to (i) performance on a subsequent exam in the same economics course, and (ii) academic performance in other courses. Further, we present evidence that CR questions provide information that could not be obtained by expanding the set of MC questions. A final contribution of our study is that we demonstrate that empirical approaches that rely on factor analyses or Walstad-Becker (1994)-type regressions are unreliable in the following sense: It is possible for these empirical procedures to lead to the conclusion that CR and MC questions measure the same thing, even when the underlying data contain strong, contrary evidence.

COMPARISONS AMONG DESIGNS FOR EQUATING CONSTRUCTED‐RESPONSE TESTS

ABSTRACTThis study examined variations of a nonequivalent groups equating design used with constructed‐response (CR) tests to determine which design was most effective in producing equivalent scores across the two tests to be equated. Using data from a large‐scale exam, the study investigated the use of anchor CR item rescoring in the context of classical equating methods. Four linking designs were examined: (a) an anchor set containing common CR items, (b) an anchor set incorporating common CR items rescored, (c) an external multiple‐choice (MC) anchor test, and (d) an equivalent groups design incorporating CR items rescored (no anchor test). The use of CR items without rescoring or the use of an external MC anchor resulted in much larger bias than the other two designs. The use of a rescored CR anchor and the equivalent groups design led to similar levels of equating error.

STUDIES OF A LATENT‐CLASS SIGNAL‐DETECTION MODEL FOR CONSTRUCTED‐RESPONSE SCORING

ABSTRACTRater behavior in essay grading can be viewed as a signal‐detection task, in that raters attempt to discriminate between latent classes of essays, with the latent classes being defined by a scoring rubric. The present report examines basic aspects of an approach to constructed‐response (CR) scoring via a latent‐class signal‐detection model. The model provides a psychological framework for CR scoring and includes rater parameters with a clear cognitive basis. Simulations are used to examine how well rater parameters and latent‐class sizes are recovered as well as the accuracy of classification. The relation of rater parameters to agreement statistics and classification accuracy is examined. The effects of using a balanced, incomplete block design are compared to those for a fully crossed design. The model is applied to several ETS datasets.

A Two-Stage Scoring Method to Enhance Accuracy of Performance Level Classification

Many tests classify each examinee into one of multiple performance levels on the basis of a combination of multiple-choice (MC) and constructed-response (CR) items. This study introduces a two-stage scoring method that identifies examinees whose MC scores place them near a cut point, advising scorers on which examinees will be most affected by score discrepancies. The idea that additional attention be given to such examinees' CR responses is explored from both psychometric and practical perspectives. Because more precision is achieved for examinees near the cut points, the method has potential to enhance the percentage of examinees classified correctly. In an empirical investigation of two operational assessments, the two-stage scoring approach successfully predicted which examinees would ultimately attain scores near the border between two performance levels.

EQUATING OF MIXED‐FORMAT TESTS IN LARGE‐SCALE ASSESSMENTS

ABSTRACTThis study examined variations of the nonequivalent‐groups equating design for mixed‐format tests—tests containing both multiple‐choice (MC) and constructed‐response (CR) items—to determine which design was most effective in producing equivalent scores across the two tests to be equated. Four linking designs were examined: (a) an anchor with only MC items; (b) a mixed‐format anchor containing both MC and CR items; (c) a mixed‐format anchor incorporating CR item rescoring; and (d) a hybrid combining single‐group and equivalent‐groups designs, thereby avoiding the need for an anchor test. Designs using MC items alone or those using a mixed anchor without CR item rescoring resulted in much larger bias than the other two design approaches. The hybrid design yielded the smallest root mean squared error value.

Differential effects of global modifications to large‐scale high stakes examination programmes

Driven largely by calls for accountability, the use of large‐scale testing is expanding in terms of the number and purposes of testing programmes. At the same time, financial constraints have resulted in attempts to reduce the lengths of such examinations. An examination of the 1994/1995 and 1995/1996 British Columbia Scholarship programme illustrates that differential and unanticipated differences can occur when such changes to the testing programme are made. The removal of a portion of the constructed‐response (CR) and written tasks (WT) items used to identify scholarship recipients resulted in differences in scholarship scores and the identification of scholarship recipients. Further, the differences were found to affect subgroups of students differentially. While there were no differences attributed to gender, higher difference rates were associated with course area (humanities vs. science) or examination session (January vs. June). The results illustrate the complex and contextual impact of changes to examination programmes and the potential consequences of such changes. Test developers and users must make more of an effort to examine the consequences of examination programmes and planned changes upon the students and others who may be affected by the results.

Differences between multiple choice items and constructed response items in the IEA timss surveys

In international large-scale surveys, constructed response (CR) items are increasingly being used and multiple-choice (MC) items are being used less frequently. In this article the two item types will be compared in terms of any differences they have on national mean scores. TIMSS 1995 and TIMSS 1999 data have been used. Are there different effects of the question types for mathematics and science? Does the introduction of open-ended items into the math and science tests affect the math and science achievement results?

PERFORMANCE OF MEN AND WOMEN ON MULTIPLE‐CHOICE AND CONSTRUCTED‐RESPONSE TESTS FOR BEGINNING TEACHERS

ABSTRACTSome previous research results imply that women tend to perform better, relative to men, on constructed‐response (CR) tests than on multiple‐choice (MC) tests in the same subjects. An analysis of data from several tests used in the licensing of beginning teachers supported this hypothesis, to varying degrees, in most of the tests investigated. The hypothesis was strongly supported in Praxis™ Principles of Learning and Teaching tests for secondary school teachers and in subject‐knowledge tests for social studies teachers, science teachers, and middle school mathematics teachers. Evidence for the hypothesis was weak in subject‐knowledge tests for middle school English teachers and for secondary school mathematics teachers. Subject‐knowledge tests for secondary school English teachers did not show the hypothesized relationship. The analysis was based on plots showing the cumulative percentages of men and women attaining each possible score on the MC and CR tests.

POPULATION INVARIANCE OF SCORE LINKING: THEORY AND APPLICATIONS TO ADVANCED PLACEMENT PROGRAM® EXAMINATIONS

ABSTRACTTest equating methods are intended to produce interchangeable scores and should not be strongly influenced by the group of examinees on which they are computed. Advanced Placement Program® (AP®) exams, which include both multiple‐choice (MC) items and constructed‐response (CR) items, provide an environment for studying the effects of subpopulations on the estimation of equating functions. The exams are equated to past forms using an internal anchor‐test design in which the linking items are restricted to MC items due to the immediate disclosure of the CR sections of the tests (precluding their reuse as linking items). This collection of closely related papers uses the variety of cases provided by AP exams to assess the effects of subpopulations on various important aspects of test equating. Earlier versions of these papers were prepared for a symposium at the 2002 Annual Meetings of the National Council on Measurement in Education.

Weighting Constructed-Response Items in IRT-Based Exams

Weighting responses to Constructed-Response (CR) items has been proposed as a way to increase the contribution these items make to the test score when there is insufficient testing time to administer additional CR items. The effect of various types of weighting items of an IRT-based mixed-format writing examination was investigated. Constructed-response items were weighted by increasing their representation according to the test blueprint, by increasing their contribution to the test characteristic curve, by summing the ratings of multiple raters, and by applying optimal weights utilized in IRT pattern scoring. Total score and standard errors of the weighted composite forms of CR and Multiple-Choice (MC) items were compared against each other and against a form containing additional rather than weighted items. Weighting resulted in a slight reduction of test reliability but reduced standard error in portions of the ability scale.

Construct Equivalence of Multiple‐Choice and Constructed‐Response Items: A Random Effects Synthesis of Correlations

A thorough search of the literature was conducted to locate empirical studies investigating the trait or construct equivalence of multiple‐choice (MC) and constructed‐response (CR) items. Of the 67 studies identified, 29 studies included 56 correlations between items in both formats. These 56 correlations were corrected for attenuation and synthesized to establish evidence for a common estimate of correlation (true‐score correlations). The 56 disattenuated correlations were highly heterogeneous. A search for moderators to explain this variation uncovered the role of the design characteristics of test items used in the studies. When items are constructed in both formats using the same stem (stem equivalent), the mean correlation between the two formats approaches unity and is significantly higher than when using non‐stem‐equivalent items (particularly when using essay‐type items). Construct equivalence, in part, appears to be a function of the item design method or the item writer's intent.

The Effectiveness of Constructed-Response and Multiple-Choice Study Tasks in Computer Aided Learning

This study compares the effectiveness of multiple-choice (MC) and constructed-response (CR) study tasks both with feedback. Graduate students ( n = 46) completed a computer-delivered vocabulary lesson on instructional design terms followed by cued recall and recognition posttests. The CR study task was significantly more effective than the MC study task on the recognition posttest, but not on the cued recall posttest. Designers should consider expanding the use of CR study tasks in similar instructional cases.

A Comparison of Three Scoring Methods for Tests With Selected-Response and Constructed-Response Items

A number of methods for scoring tests with selected-response (SR) and constructed-response (CR) items are available. The selection of a method depends on the requirements of the program, the particular psychometric model and assumptions employed in the analysis of item and score data, and how scores are to be used. This article compares 3 methods: unweighted raw scores, Item Response Theory pattern scores, and weighted raw scores. Student score data from large-scale end-of-course high school tests in Biology and English were used in the comparisons. In the weighted raw score method evaluated in this study, the CR items were weighted so that SR and CR items contributed the same number of points toward the total score. The scoring methods were compared for the total group and for subgroups of students in terms of the resultant scaled score distributions, standard errors of measurement, and proficiency-level classifications. For most of the student ability distribution, the three scoring methods yielded similar results. Some differences in results are noted. Issues to be considered when selecting a scoring method are discussed.

Are Multiple‐Choice Exams Easier for Economics Students? A Comparison of Multiple‐Choice and “Equivalent” Constructed‐Response Exam Questions

Many economics students believe that multiple‐choice (MC) exams are easier than constructed‐response (CR) exams. If true understanding of economics is reflected by performance on CR exams, then easier MC exams would give rise to misleading measures of student understanding of economics, creating a false sense of complacency among both instructors and students. This article investigates this issue by comparing student scores on MC and “equivalent” CR questions. Our results show that, for certain types of MC questions, students do indeed score better, even after correcting for guessing, and on other types of questions, they produce similar scores. These results are unaffected by comparing males versus females or “good” versus “poor” students.